Unmanned aerial vehicle and RIS cooperation edge calculation method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle (UAV) and reconfigurable intelligent surface (RIS) collaborative edge calculation method based on deep reinforcement learning, which is used for data collection service in an urban scene. The method comprises the following steps: firstly, designing a deep reinforcement learning driven UAV trajectory planning method, and converting an optimization target into a shortest flight trajectory design problem by introducing a penalty matrix to dynamically balance constraints such as data collection efficiency, risk avoidance and flight energy consumption; and secondly, deducing an optimal closed-form solution of RIS passive phase shift, and adjusting the phase in real time after each step of action of the unmanned aerial vehicle to enhance channel gain and improve air-ground link transmission quality. According to the method, the accumulated risk can be effectively reduced, and the equivalent service efficiency (EoS) and the quality of service (QoS) are improved at the same time.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing and unmanned aerial vehicle (UAV) collaborative communication technology, and in particular to a UAV and RIS collaborative edge computing method based on deep reinforcement learning for complex urban scenarios. Background Art

[0002] In recent years, with the surge in the number of IoT terminals and the deepening of smart city applications, terrestrial wireless communication networks are facing the dual challenges of exponential growth in data traffic and stringent service quality requirements. According to industry forecasts, global monthly mobile data traffic will exceed 1 zettabyte by 2028, and the bottlenecks of traditional infrastructure in coverage, computing elasticity, and energy efficiency are becoming increasingly prominent. In this context, Mobile Edge Computing (MEC) provides innovative solutions for real-time data processing and low-latency services by sinking computing power to the edge of the network. However, in urban scenarios, especially in emergency situations, fixed edge nodes have high deployment costs and poor environmental adaptability, making it difficult to cope with dynamically changing service demands and communication blockages caused by complex geographical barriers.

[0003] To overcome the above limitations, Unmanned Aerial Vehicles (UAVs) have become an important extension carrier of mobile edge computing due to their flexible maneuverability and rapid deployment capabilities. UAVs can dynamically adjust their flight trajectories to achieve efficient coverage of hotspots and provide near-field computing services to ground terminals by carrying edge servers. However, the dense high-rise buildings in actual urban scenarios cause air-to-ground links to frequently encounter non-line-of-sight (NLoS) transmission problems, which seriously restricts communication quality and data collection efficiency. To this end, Reconfigurable Intelligent Surface (RIS) technology has been introduced into air-to-ground collaborative networks. By regulating the electromagnetic wave propagation environment and reconstructing the phase of the reflected signal, it significantly enhances the channel gain between the UAV and the terminal, forming a collaborative enhancement mechanism of "dynamic trajectory + intelligent reflection".

[0004] Initial progress has been made in existing research on RIS-assisted UAV communication systems. For example, some work uses alternating optimization algorithms to jointly design UAV trajectories and RIS phase shifts, achieving energy efficiency improvements under specific energy constraints. Other research explores autonomous decision-making strategies in dynamic environments based on deep reinforcement learning. However, these solutions still have three major limitations: First, most studies are limited to simplified scenarios involving a single UAV and a single RIS, failing to consider resource competition and interference avoidance issues when multiple UAVs collaborate. Second, existing phase control often uses a uniform linear array model, which exhibits beamforming differences from the actually deployed uniform planar array (UPA), leading to a disconnect between theoretical performance and engineering practice. Third, due to the lack of quantitative modeling of the risks of dense obstacles in complex urban scenarios, UAV trajectory planning often falls into local optimality, making it difficult to balance data collection efficiency and flight safety. Summary of the Invention

[0005] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a UAV and RIS collaborative edge computing method based on deep reinforcement learning, comprising the following steps:

[0007] Based on a mobile edge system architecture with multiple UAVs and RIS in collaboration, a joint optimization problem model is proposed, with UAV trajectory planning decisions and RIS phase alignment decisions as optimization variables, under the constraints of flight energy consumption, data collection efficiency, and obstacle risk in three-dimensional urban scenes, and the optimization objectives are to maximize the equivalent service efficiency (Efficiency of Service, EoS) per unit energy consumption and minimize the cumulative risk. The optimization problem model is converted into a Markov decision process model, and two types of intelligent agents are constructed for UAVs and RIS. Based on a reinforcement learning-driven penalty matrix iterative update mechanism and the derivation of a closed-form optimal solution for RIS phase shift, offline trajectory planning and online dynamic phase alignment are performed.

[0008] Furthermore, the equivalent efficiency of service (EoS) is defined as the amount of data collected per unit path length of the UAV; the cumulative risk is quantified using a probabilistic model of the horizontal distance between the obstacle and the UAV; the UAV trajectory planning decision is represented by a displacement vector in a three-dimensional Cartesian coordinate system, subject to fixed flight altitude and step size constraints; the RIS phase alignment decision is achieved by optimizing the phase shift matrix of the reflective element, and its closed-form solution is derived by maximizing the channel gain of the GT-RIS-UAV reflective link.

[0009] Furthermore, the global state space of the Markov decision process model includes the current position of the UAV, the distribution of obstacles within the observation radius, the amount of ground terminal (GT) data within the service radius, and the RIS phase state; the action space includes the real-time adjustment strategy of the UAV's directional displacement vector and RIS phase shift matrix in a gridded environment.

[0010] Furthermore, the observation space of the UAV agent includes its current position, the coordinates of obstacles within the detection radius, the distribution of GTs data within the service radius, and historical trajectory information. The action space is a displacement vector with a fixed step size. The observation space of the RIS agent includes the real-time position of the UAV, the position of GTs, and channel state information. The action space is the phase adjustment strategy of the passive phase shift matrix.

[0011] Furthermore, this deep reinforcement learning-driven iterative update mechanism is built on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework. A centralized critic network comprehensively evaluates data collection benefits, Gaussian risk aversion, and flight energy consumption factors, dynamically generates corrections, and combines a priority experience replay strategy to accelerate matrix convergence. Finally, the policy gradient is analyzed through a distributed actor network to generate an optimal trajectory sequence that satisfies kinematic constraints.

[0012] Furthermore, the closed-form optimal solution for the RIS phase shift is derived based on the channel gain maximization criterion of the GT-RIS-UAV reflection link. By adjusting the phase of the reflection elements to compensate for the differences in signal propagation paths, coherent superposition of the reflection link is achieved. After each step of the UAV's movement, the RIS recalculates the phase shift matrix based on the real-time position information to ensure maximum channel gain.

[0013] Furthermore, the offline trajectory planning adopts rasterized environment modeling, divides the study area into uniform grids, and generates collision-free and risk-minimizing trajectory sequences through penalty matrix iteration; the online dynamic phase alignment triggers RIS phase reconfiguration by real-time monitoring of the UAV position and GTs distribution to ensure the stability of the data transmission rate.

[0014] In a second aspect, the present invention provides a UAV and RIS collaborative edge computing system based on deep reinforcement learning, comprising:

[0015] The problem modeling module is used to build a joint optimization model for UAV trajectory planning, RIS phase alignment, and data collection in a 3D urban scenario, defining a multi-objective function to maximize the equivalent efficiency of service (EoS) and minimize the cumulative risk;

[0016] A deep reinforcement learning module generates optimal trajectories based on a distributed actor network. A centralized critic network comprehensively evaluates data collection benefits, Gaussian risk aversion, and flight energy consumption factors to dynamically generate corrections.

[0017] The dynamic execution module is used to online control the UAV to move along the planned trajectory, synchronously adjust the RIS phase shift matrix, monitor obstacles and GTs state changes, and trigger trajectory replanning and phase realignment.

[0018] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the trajectory planning and data collection method based on collaboration between multiple UAVs and RIS are implemented.

[0019] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the trajectory planning and data collection method based on collaboration between multiple UAVs and RIS.

[0020] Beneficial Effects: This paper proposes a reinforcement learning-based mobile edge system for multi-UAV and RIS collaboration, suitable for low-altitude data collection and real-time transmission tasks in urban scenarios. This method achieves efficient data collection and risk avoidance in complex three-dimensional environments by jointly optimizing UAV trajectory planning and RIS phase alignment. The designed MADDPG framework can dynamically balance data collection efficiency and flight safety, ensuring the generation of the shortest risk-aware trajectory; the RIS phase shift closed-form solution derivation and real-time alignment technology significantly improve the channel gain under NLoS conditions and enhance data transmission reliability. Simulation results show that compared with traditional RRT and A-Star algorithms, this method can reduce the cumulative risk by at least 31.2%, while improving the equivalent service efficiency (EoS) and service quality (QoS), making it suitable for scenarios with strict real-time and safety requirements, such as smart city monitoring and emergency communications. In addition, the system supports multi-UAV collaboration and dynamic environment adaptation. Through a reinforcement learning-driven online decision-making mechanism, it effectively responds to changes in GTs distribution and sudden obstacle interference, improving the robustness and scalability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of a mobile edge system scenario in which multiple drones collaborate with RIS based on reinforcement learning in an embodiment of the present invention.

[0022] Figure 2 Schematic diagram of the training process of various intelligent agents in an embodiment of the present invention.

[0023] Figure 3This is a comparison chart of equivalent EoS, QoS, and cumulative risk of different algorithms in an embodiment of the present invention.

[0024] Figure 4 This is a graph showing the changes in EoS and QoS with data volume under the CoUR algorithm with and without RIS in an embodiment of the present invention.

[0025] Figure 5 This is a UAV motion trajectory diagram under the CoUR algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0027] Combine Figure 1 In the scenario shown, an embodiment of the present invention discloses a reinforcement learning-driven mobile edge system based on the collaboration of multiple UAVs and reconfigurable intelligent surfaces. For the UAV trajectory planning and data collection services in urban scenarios, a multi-UAV collaboration and RIS-assisted communication architecture is introduced to solve the problems of high energy consumption, low data collection efficiency, and weak risk avoidance ability of UAVs in complex three-dimensional environments. This method considers maximizing the equivalent efficiency of UAV services under the constraints of energy consumption, data collection volume, flight risk, and channel gain. In order to solve this multi-objective optimization problem, a collaboration algorithm based on reinforcement learning (Collaboration-of-Multi-UAVs-and-RIS-basedAlgorithm, CoUR) is designed. The algorithm combines the iterative update of the penalty matrix with the RIS phase alignment mechanism to solve the dynamic obstacle avoidance and channel enhancement problems in trajectory planning, and optimizes the reflection link performance through the real-time phase adjustment of RIS. Through offline rasterized environment modeling and online adaptive decision-making, UAVs are allowed to efficiently perform data collection tasks in dynamic urban environments.

[0028] An embodiment of the present invention describes a reinforcement learning-driven mobile edge system based on collaboration between multiple drones and a reconfigurable intelligent surface. First, based on a mobile edge architecture that collaborates with multiple drones and a reconfigurable intelligent surface (RIS), an optimization problem model is developed in a three-dimensional urban scenario, taking into account drone propulsion energy consumption, data collection volume, flight risk, and channel gain constraints. The optimization problem model is used as decision variables, including drone trajectory design, RIS phase alignment, and data transmission rate optimization, with maximizing service equivalent efficiency as the optimization objective. The optimization problem model is then converted into a reinforcement learning-driven Markov decision process, and a collaborative intelligent agent is constructed for drone motion and RIS phase adjustment. Based on a penalty matrix iterative update mechanism, a RIS phase alignment strategy is embedded in the drone action sequence to perform offline environmental modeling and online dynamic optimization. Specifically, in its implementation, service equivalent efficiency can be expressed by the amount of data collected per unit path length; UAV trajectory design can be represented by a sequence of step vectors within a horizontal gridded area, combined with a centralized critic network update to mitigate obstacle risks; RIS phase alignment is achieved by deriving an optimal closed-form solution for passive phase shifts, which is adjusted in real time after each UAV maneuver to maximize the reflection link channel gain; and data transmission rate optimization is represented by dynamically updating the channel model and data remainder of the GT-RIS-UAV link. Simulation results demonstrate that compared to existing mainstream solutions, the proposed system can reduce cumulative risk by at least 31.2% and significantly improve equivalent service efficiency and service quality.

[0029] The specific steps of the embodiment of the present invention are described in detail below in conjunction with a specific system model.

[0030] Step (1) is to establish a RIS-assisted multi-UAV trajectory optimization and data collection architecture, including a ground terminal model, a UAV model, and a RIS deployment model; the specific steps are as follows:

[0031] (1a) Design a RIS-assisted multi-UAV collaborative trajectory optimization and data collection system architecture, which includes multiple randomly distributed ground terminals (GTs), multiple unmanned aerial vehicles (UAVs), and reconfigurable smart surfaces (RISs) deployed on the surface of buildings. GTs serve as data sources and generate large-scale monitoring data that needs to be transmitted to the cloud. UAVs are equipped with edge computing modules, which can collect GT data during flight and forward it to the cloud through RIS-enhanced links. RIS optimizes the channel gain of the GT-UAV link by dynamically adjusting the reflection phase, thereby improving data transmission efficiency. The cloud server is responsible for storing and analyzing data and feeding back control instructions to the terminal. In this embodiment of the present invention, the target area is set as a 1000m×100m 3D urban scene, which includes 10 GTs, 4 UAVs, and 1 RIS. The area is divided into 50×50 grids, each grid represents a 20m×20m plane area, and the grid height is the building height or the fixed flight height of the UAV (100m).

[0032] (1b), establish a ground terminal model. Assume that various types of ground terminals (GTs) are randomly distributed, defined as I = {1, 2, ..., I}. Each GT carries a large amount of data waiting to be collected by the UAV. The present invention considers low-speed scenarios. In each tiny time slot, the position of the GTs can be regarded as static. Secondly, define the building set as J = {1, 2, ..., J}. Assume that the buildings follow a Gaussian distribution and are high enough to be regarded as obstacles in the UAV flight trajectory. In order to avoid channel congestion, a UPA passive RIS is deployed on the surface of a high-rise building. The RIS consists of Mr×Mc reflective elements. The distance between adjacent elements is equal, denoted by D.

[0033] (1c), the present invention adopts a three-dimensional Cartesian coordinate system to describe the positions of GTs, obstacles, RIS and UAVs. Specifically, the positions of GTi and obstacle j are expressed as It is used to represent the position of the first element of the main diagonal in RIS. For safety reasons, UAVn is only allowed to fly at a fixed altitude, so its position in time slot t can be expressed as

[0034] (1d), establish the MADDPG framework. It includes a distributed actor network architecture and a centralized critic network architecture. The distributed actor network architecture transforms the state space of the i-th drone into Input to the fully connected layer network, where is a three-dimensional coordinate, is the motion velocity vector, is the phase matrix of RIS, is a local obstacle map, the fully connected layer outputs a standardized action vector through the tanh activation function; the centralized critic network combines the states of all drones {s1, ..., s M} with actions {a1, ..., a M} as the joint input, and output the global state-action value function through two fully connected layers.

[0035] Step (2): Based on the above architecture, establish a data collection model, a channel model, and a risk avoidance model; including the following specific steps:

[0036] (2a) Establish a data collection model. In the 3D urban scene proposed by this invention, GTs, such as users’ smart devices, smart vehicles, road monitoring, office servers, etc., have specific tasks (large amounts of data) that need to be transmitted to cloud servers for storage or calculation through UAVs. The data volume of GTs’ tasks is defined as The following considers the data transmission process between GTs and UAVs. The average data transmission rate between GTi and UAVn in time slot t can be expressed as:

[0037]

[0038] Where W represents the channel bandwidth, p tr Indicates the transmission power of the signal. 2 Represents the power of Gaussian white noise in the channel. When GTi enters the service range R of UAVn s The data transmission service will start within t. The amount of data remaining on the GTi side will decrease at the transmission rate. The amount of data remaining at time slot t can be written as:

[0039]

[0040] Here, u{·} represents a step function.

[0041] (2b) Establish a RIS auxiliary channel model. In a mobile edge system with multiple UAVs and RIS, to avoid non-line-of-sight (NLoS) links, the direct link between the GT and the UAV is redirected by the RIS into a GT-RIS-UAV reflection link. It is worth noting that in reality, the links between the GT and the RIS, and between the RIS and the UAV, are not necessarily line-of-sight (LoS) links. However, in this invention, only the case where the reflection link is a LoS link is considered.

[0042] Generally, consider the LoS / NLoS probability model. The probability that the direct connection between GTi and UAVn is a LoS link is:

[0043]

[0044] Therefore, the probability of an NLoS link is in, represents the three-dimensional distance between GTi and UAVn at time slot t. B and C are constants determined by the environment. Furthermore, the channel gain of the GT-UAV LoS link can be expressed as:

[0045]

[0046] Where ξ is the path loss when the reference distance is 1m. α is the path loss factor, which is generally set to 2.

[0047] In order to improve the channel gain under NLoS conditions, the GT-RIS-UAV reflection link is introduced. To simplify the model, it is assumed that the number of reflection elements assigned to UAVn is M×M, so the channel gain of the GT-RIS link is Expressed as a Cartesian product:

[0048]

[0049] in, represents the distance between GTi and the first element of RIS. λ is the carrier wavelength. The cosine of the horizontal angle of arrival (AoA) of the data stream signal at the RIS is Sine is represents the sine of the vertical arrival angle of the signal at the RIS. Similarly, the channel gain of the RIS-UAV link at time slot t can be written as:

[0050]

[0051] in, They represent the cosine and sine of the horizontal angle of departure (AoD) of the signal sent from the RIS to the UAVn at time slot t, respectively. represents the sine of the vertical AoD at time slot t. Therefore, the channel gain of the GT-RIS-UAV reflection link can be expressed as:

[0052]

[0053] in, represents the phase shift matrix of RIS at time slot t, and Represents the attenuation coefficient. For the mth r Row m c Phase shift of the column reflection element.

[0054] Therefore, with the help of RIS, the expectation of the channel gain between GTi and UAVn at time slot t can be written as:

[0055]

[0056] (2c) Establish a risk avoidance model. In the present invention, UAVs not only need to design trajectories to ensure that more complete data information is collected, but also need to avoid the risks caused by building obstacles. The closer to the obstacle, the higher the risk. In order to quantify the risk level, the present invention uses the following probability model to describe the UAVn on the horizontal plane. The risk of obstacles on a straight line path from point A to any other point is:

[0057]

[0058] in, represents the horizontal distance between obstacle j and UAVn at time slot t. Note that only the obstacle j within the observation radius R o Only obstacles within the range are considered as risk sources. In addition, the flight propulsion energy consumption of UAVn can be expressed as: Among them, P0 and U tip represent the blade power and the rotor tip speed respectively.

[0059] The optimization goal is to maximize the UAV's efficiency of service (EoS), that is, the amount of data collected per unit energy consumed. To simplify the model, this paper assumes that the UAV's speed is constant. Therefore, the UAV's propulsion power will be constant, and the UAV's total energy consumption will be proportional to the number of required time slots. In addition, the UAV's path length L is proportional to time. Therefore, this paper equates EoS to the amount of data collected per unit path:

[0060]

[0061] At the same time, it is also necessary to ensure that the UAVn moves a fixed distance in each time slot and the destination is reachable:

[0062]

[0063] In addition, in order to achieve the ideal training effect, the total risk size in the UAVn trajectory is restricted:

[0064] Risk n ≤Risk max

[0065] Step (3): Based on the above model, construct a multi-UAV collaboration and RIS-based algorithm; including the following specific steps:

[0066] (3a), update the MADDPG algorithm network parameters. First, prioritize experience replay, that is, build an experience replay buffer Using the priority sampling strategy, the sampling probability is calculated as follows:

[0067]

[0068] Among them, α=0.6 is the priority coefficient, ò=1e-5 is a constant, is the timing difference error;

[0069] Next, the actor network gradient is updated. For the mth drone agent, the policy gradient is calculated:

[0070]

[0071] Where B is the batch number of samples, is the critic network output, Output for actor network;

[0072] Then update the critic network and first calculate the target Q value:

[0073]

[0074] Where γ = 0.99 is the discount factor, is the target critic network;

[0075] Then calculate the critic loss function and network gradient. The calculation formula is as follows:

[0076]

[0077] The regularization coefficient β=1e-4 to prevent overfitting.

[0078] (3b), MADDPG optimization model for UAVs motion. The optimization problem is expressed as:

[0079]

[0080] Among them, λs is the data collection service factor, L current，t is the cumulative flight path length up to time t, λr is the risk barrier factor, ||α-a′|| represents the distance from grid α to the specified target point α′, Risk n represents the risk barrier result, λ E stands for energy loss factor.

[0081] (3c) , establish the phase alignment model of RIS. During the movement of UAVs, the phase alignment of RIS can improve the channel gain to the greatest extent, thereby maximizing the data transmission rate of each time slot. Based on the above derivation, it can be derived that:

[0082]

[0083] At this time, the optimal closed-form solution of the PIS phase shift is:

[0084]

[0085] The maximum channel gain of the GT-RIS-UAV reflection link is derived as:

[0086]

[0087] In step (4), based on the above modeling, the specific process of the collaboration (CoUR) algorithm based on multiple UAVs and RIS is shown in the following table.

[0088]

[0089] The following simulation experiments are carried out using the MATLAB platform, and the present invention is compared with the traditional algorithm to verify the beneficial effects of the present invention.

[0090] exist Figure 3 In the paper, the performance comparison between the embodiments of the present invention and the existing mainstream solutions RRT and A-Star and the proposed algorithm is described. Assuming λs=1, λr=20, the equivalent EoS is the data collected per unit path, and the quality of service (QoS) refers to the data collection rate, that is, the ratio of the collected data to the total amount of data. Compared with the mainstream solutions, the CoUR algorithm proposed in the present invention considers risk avoidance and data collection at the same time, which can effectively reduce the cumulative risk by at least 31.2% and maintain higher equivalent EoS and QoS. Not only obstacle avoidance but also risks in the flight trajectory are considered, while taking into account data transmission efficiency. Such a design is of practical significance.

[0091] exist Figure 4 The following graphs illustrate how EoS and QoS vary with data volume under the CoUR algorithm, with and without RIS. When the expected data size carried by GTs is less than 4 Mb, the presence of RIS does not significantly impact performance. However, when it exceeds 4 Mb, RIS enhances data collection, which is of great practical significance in scenarios with exploding data volumes.

[0092] exist Figure 5 Figure 2 illustrates a UAV trajectory diagram using the CoUR algorithm according to an embodiment of the present invention. Triangles represent GTs, and their size indicates the amount of data. The trajectories of two drones are represented by lines of different colors. It can be seen that the drones approach as many GT nodes as possible and avoid obstacles, demonstrating the system's coverage efficiency and path optimization.

[0093] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the UAV and PIS collaborative edge computing algorithm designed by the present invention based on deep reinforcement learning can effectively reduce the system's cumulative risk and achieve reasonable trajectory optimization.

[0094] Based on the same inventive concept, an embodiment of the present invention also discloses a UAV and RIS collaborative edge computing system based on deep reinforcement learning, including: an environmental modeling module: generating a three-dimensional urban scene containing randomly distributed ground terminals, building obstacles and RIS equipment, and dynamically updating the terminal data volume and obstacle location; a deep reinforcement learning decision module: adopting the critic network and actor network iterative mechanism, the distributed actor network architecture inputs the state space of the i-th UAV into the fully connected layer network, and outputs the next movement direction decision. The centralized critic network takes the state and action of all UAVs as joint input, and outputs the global state-action value function through two fully connected layers; the RIS phase control module: calculating the optimal phase shift parameter based on the real-time coordinates of the UAV and the ground terminal position and driving the RIS controller to adjust the reflection unit state; the collaborative scheduling module: managing the collision avoidance rules, RIS resource allocation strategy and environmental state sharing mechanism among multiple UAVs to ensure the global efficiency and safety of the system.

[0095] An embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method for collaborative edge computing between a drone and RIS based on deep reinforcement learning are implemented.

[0096] An embodiment of the present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the deep reinforcement learning-based UAV and RIS collaborative edge computing method.

[0097] The program code for implementing the inventive method can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, causes the steps of the inventive method to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server. The present invention is not described in detail herein, and all of these are known techniques to those skilled in the art.

Claims

1. A UAV and RIS collaborative edge computing method based on deep reinforcement learning, characterized in that: The following steps are involved: A multi-UAV cooperative communication system model is established in a three-dimensional urban scenario, in which reconfigurable smart surfaces (RIS) are fixedly deployed on building surfaces. The dynamic planning of UAV trajectories and the passive phase shift adjustment of RIS are jointly optimized to construct a joint optimization problem with the goals of maximizing the amount of data collected per unit path length (equivalent service efficiency) and minimizing risk obstacles and flight energy consumption. A distributed decision-making model for UAV agents is constructed using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm framework. A centralized critic network and a distributed actor network are designed. The critic network receives global state information, including the positions of all UAVs, obstacle distribution, the amount of ground terminal (GT) data within the service radius, and RIS phase parameters, and outputs a joint Q-value evaluation. Each UAV independently trains an actor network to generate a trajectory decision vector in a continuous action space based on its local observation state (its own coordinates, sensor-perceived obstacle distribution, and the amount of neighboring GT data). The reward function is defined as the weighted sum of the equivalent service efficiency factor, the risk aversion factor, and the target distance factor; the loss function is defined as the joint optimization objective of the temporal difference error of the critic network and the policy gradient of the actor network; In the RIS phase optimization stage, based on the real-time channel state information between the UAV and the target terminal, a closed-form solution algorithm is used to dynamically calculate the RIS phase shift matrix to maximize the reflection link channel gain. During the execution phase, the UAV generates a dynamic obstacle avoidance trajectory based on the trained MADDPG model. Meanwhile, the RIS controller calculates the closed-form solution of the phase shift in real time according to the channel status, achieving coordinated optimization of trajectory planning and phase adjustment.

2. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 1 is characterized in that: The MADDPG framework includes a distributed actor network architecture and a centralized critic network architecture. The distributed actor network architecture transforms the state space of the i-th drone into Input to the fully connected layer network, where is a three-dimensional coordinate, is the motion velocity vector, is the phase matrix of RIS, is a local obstacle map, the fully connected layer outputs a standardized action vector through the tanh activation function; the centralized critic network combines the states of all drones {s1, ..., s M } with the action {a1, ..., a M } as the joint input, and output the global state-action value function through two fully connected layers.

3. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 1 is characterized in that: The RIS phase adjustment strategy achieves coherent superposition of reflection links based on the following formula: Among them, m r With m c Represents a passive RIS of UPA, RIS consists of m r ×m c It is composed of reflective elements. Represents RIS No. (m r ,m c ) units at time t, λ represents the wavelength of the electromagnetic wave, D represents the RIS unit spacing, is the cosine of the horizontal arrival angle of the data stream signal at the RIS, is the sine value, represents the sine of the vertical angle of arrival of the signal at the RIS, denote the cosine and sine, respectively, of the horizontal departure angle of the signal transmitted from the RIS to the UAVn at time slot t.

4. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 1 is characterized in that: The optimization problem is expressed as: in, Represent the x-coordinate and y-coordinate of UAVn at time slot t, They represent the x-coordinate and y-coordinate of the GTi location, λs is the data collection service factor, and L current,t is the cumulative flight path length up to time t, λr is the risk obstacle factor, ||ɑ-a′|| represents the distance from grid α to the specified target point α′, Represents the amount of remaining data in time slot t, Risk n represents the risk barrier result, λ E Represents the energy loss factor, E n Represents the energy loss of the drone flight.

5. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 1 is characterized in that: The phase optimization goal is to maximize the channel gain: in, represents the channel gain of the GT-UAV LoS link, assuming that the number of reflective elements in the GT-RIS-UAV reflection link assigned to UAVn is M × M, ξ is the path loss at a reference distance of 1 m, represents the distance between GTi and the first element of RIS, represents the distance from UAVn to RIS at time t.

6. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 1 is characterized in that: The MADDPG algorithm network parameter update includes: first giving priority to experience playback, that is, building an experience playback buffer Using the priority sampling strategy, the sampling probability is calculated as follows: Among them, α=0.6 is the priority coefficient, ε=1e-5 is a constant, is the timing difference error; Next, the actor network gradient is updated. For the mth drone agent, the policy gradient is calculated: Where B is the batch number of samples, is the critic network output, Output for actor network; Then update the critic network and first calculate the target Q value: Where γ = 0.99 is the discount factor, is the target critic network; Then calculate the critic loss function and network gradient. The calculation formula is as follows: The regularization coefficient β = 1e-4 is used to prevent overfitting.

7. The UAV and RIS collaborative edge computing method based on deep reinforcement learning according to claim 4 is characterized in that: The risk barrier quantification model is: in, represents the horizontal distance between obstacle j and UAVn at time slot t. The UAV flight propulsion energy consumption model is: Among them, P0 and Utip represent the blade power and the rotor tip speed respectively, v n,t represents the speed of UAVn at time t.

8. A UAV and RIS collaborative edge computing method based on deep reinforcement learning that implements the method described in claims 1-7, characterized in that: include: A problem-building module builds a multi-UAV cooperative communication system model in a three-dimensional urban scenario, where reconfigurable smart surfaces (RIS) are fixedly deployed on building surfaces. The dynamic planning of UAV trajectories and the passive phase shift adjustment of RIS are jointly optimized to construct a joint optimization problem with the goals of maximizing the amount of data collected per unit path length (equivalent service efficiency) and minimizing risk obstacles and flight energy consumption. The neural network building module uses the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm framework to construct a distributed decision-making model for UAV agents. A centralized critic network and a distributed actor network are designed. The critic network receives global state information, including the positions of all UAVs, obstacle distribution, the amount of ground terminal (GT) data within the service radius, and RIS phase parameters, and outputs a joint Q-value evaluation. Each UAV independently trains the actor network to generate a trajectory decision vector in a continuous action space based on its local observation state (its own coordinates, the obstacle distribution perceived by sensors, and the amount of neighboring GT data). The network training module defines the reward function as the weighted sum of the equivalent service efficiency factor, the risk aversion factor, and the target distance factor; and defines the loss function as the joint optimization objective of the critic network's temporal difference error and the actor network's policy gradient, and updates the actor and critic neural networks. The RIS phase optimization module is used during the execution phase. The UAV generates dynamic obstacle avoidance trajectories based on the trained MADDPG model. At the same time, the RIS controller calculates the closed-form solution of phase shift in real time according to the channel status, achieving coordinated optimization of trajectory planning and phase adjustment.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images