Task offloading optimization method based on unmanned aerial vehicle and ris assisted MEC system

Abstract

This invention discloses a task offloading optimization method based on a UAV and RIS-assisted MEC system, comprising: constructing a UAV-assisted and RIS-assisted mobile edge computing system (MEC) model; based on the MEC, constructing a channel model, task generation and priority model, offloading latency and energy consumption model, and UAV flight and charging scheduling model for the system; based on the MEC, under the condition of satisfying system constraints, constructing a joint optimization objective function P for the system with the objective of minimizing the maximum processing latency and system energy consumption of all user devices in the system; and solving the joint optimization objective function P using a reinforcement learning algorithm based on a TD3 network to obtain the task offloading optimization scheme for the MEC. This invention can minimize task processing latency and system energy consumption, thereby improving the performance of the mobile edge computing system.

Description

Technical Field

[0001] This invention belongs to the field of edge computing technology, and in particular relates to a task offloading optimization method based on UAVs and RIS-assisted MEC systems. Background Technology

[0002] With the development of 5G technology, new applications characterized by computational intensity and latency sensitivity are constantly emerging, such as online games, autonomous driving, virtual reality, and augmented reality. Mobile edge computing (MEC) is needed to assist in handling resource-intensive applications, helping to reduce the computational resources and energy consumption of running such applications. By deploying computing resources with data processing and service functions closer to user terminals, MEC can provide users with faster service computing power, thereby reducing the energy consumption and computing time of user devices. It also reduces the need to offload large amounts of data to the cloud, thus alleviating pressure on the core network and improving the quality of service in real-time response scenarios.

[0003] Drones not only possess extensive geographical coverage capabilities but also offer unique advantages such as rapid deployment and ease of programming. Various payloads can be mounted on drones, including IoT sensors (such as cameras), miniaturized base stations, and embedded computing modules, to perform a variety of sensing, communication, and computing tasks. Proper deployment and operation of drones can provide reliable and cost-effective wireless communication solutions for many practical scenarios.

[0004] Reconfigurable Intelligent Surfaces (RIS) bring revolutionary capabilities to wireless communication by providing fine-grained control over electromagnetic wave propagation. RIS is particularly beneficial in environments with obstructed line of sight or weak signal strength. RIS can optimize signal paths, bypassing obstacles that reflect or refract signals, ensuring robust and reliable communication links. This capability is especially valuable for drones operating in complex and dynamic environments, where the integration of RIS offers significant advantages in communication reliability, energy efficiency, and computational load. By improving signal transmission efficiency, RIS reduces the need for intensive onboard processing, saving power and extending flight time. Furthermore, RIS optimizes signal paths, ensuring drones maintain robust communication links without consuming excessive energy, directly contributing to extended runtime and reduced operating costs.

[0005] Most existing studies typically consider the auxiliary role of UAVs or RIS in mobile edge systems in isolation, and their task scheduling often relies on fixed rules or single algorithms to generate tasks and allocate resources, making it difficult to adapt to dynamically changing task load scenarios. Furthermore, current research only considers flight and computing energy consumption for UAV energy management, failing to integrate charging station planning into a comprehensive charging strategy. This results in limited UAV endurance and impacts system service continuity. To further unlock the potential of mobile edge computing technology, this study proposes a system that combines UAVs and RIS to assist in task offloading, and optimizes the task generation and scheduling mechanism and charging strategy. This system design has significant practical implications and application value. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a task offloading optimization method based on UAVs and RIS-assisted MEC systems, which includes:

[0007] S1: Construct a Mobile Edge Computing (MEC) model based on UAV and RIS assistance. This system includes: one edge cloud base station and one UAV equipped with RIS. RIS One unmanned aerial vehicle (UAV) to perform unloading tasks, two fixed charging stations, and several user equipment (UEs);

[0008] S2: Based on the Mobile Edge Computing (MEC) system, construct the system's channel model, task generation and priority model, offload latency and energy consumption model, and UAV flight and charging scheduling model;

[0009] S3: Based on the Mobile Edge Computing (MEC) system, minimize the maximum processing latency of all user devices in the system while satisfying system constraints. and system energy consumption To achieve the goal, construct the joint optimization objective function P for this system;

[0010] S4: Solve the joint optimization objective function P using a reinforcement learning algorithm based on the TD3 network to obtain the task offloading optimization scheme of the mobile edge computing system (MEC).

[0011] The joint optimization objective function P is:

[0012]

[0013] in, This indicates the user of the drone task scheduling service. Indicates in time slot The maximum processing latency for unloading computation. Indicates system energy consumption. This represents the set of user devices in the system. This represents the two-dimensional coordinates of the UAV carrying the RIS on the ground in time slot t+1. This represents the two-dimensional coordinates of the UAV's ground projection during the unloading of its mission in time slot t+1. The weights represent the maximum processing latency. The weights representing system energy consumption express Task generation for time-slot user equipment follows a Poisson distribution. Represents the random number generation function of the Poisson distribution. Indicates the first Task arrival rate per time slot Indicates the first Task arrival rate per time slot This indicates the task arrival rate during peak hours. This indicates the task arrival rate during off-peak hours. This indicates RIS phase constraint. Indicates in The proportion of tasks that will be offloaded to the drone server that will execute the task will be determined by the time slot. Indicates in The proportion of tasks that are offloaded to edge servers in each time slot. Let T represent the number of time slots in the entire communication cycle, and M represent the number of user equipment in the system. This indicates the number of reflection units on the RIS. This represents the total quota of tasks generated by all user devices in the system. C1 represents the first constraint, C2 represents the second constraint, C3 represents the third constraint, C4 represents the fourth constraint, C5 represents the fifth constraint, C6 represents the sixth constraint, C7 represents the seventh constraint, C8 represents the eighth constraint, C9 represents the ninth constraint, C10 represents the eleventh constraint, C11 represents the eleventh constraint, C12 represents the twelfth constraint, and C13 represents the thirteenth constraint.

[0014] The beneficial effects of this invention are:

[0015] 1. This invention considers dynamic task generation and intelligent scheduling in mobile edge computing systems. It uses Poisson distribution to generate tasks with differentiated probabilities in different time slots, which can realistically simulate scenarios with dynamic changes in high and low task loads. Based on a priority scheduling mechanism weighted by the ratio of energy consumption and latency, it can prioritize providing services to users in high-energy-consumption and high-latency scenarios, thereby improving system resource utilization efficiency and service quality.

[0016] 2. This invention uses a UAV equipped with a MEC server as an airborne base station to collaboratively offload tasks with the ground base station, overcoming the problem of long service response latency when the base station and user equipment are far apart. The UAV will partially compute user equipment tasks through line-of-sight links and reconfigurable intelligent surface RIS reflection links, and forward some computing tasks to the ground base station (edge ​​cloud base station).

[0017] 3. This invention uses a reliable charging strategy to ensure the drone's endurance. It proposes an energy threshold-triggered charging mechanism, which triggers charging when the drone's energy is less than twice the energy required to fly to the charging station, thus avoiding service interruption due to energy depletion. The invention also uses a reinforcement learning algorithm to select the optimal charging station and replan the trajectory, minimizing energy consumption and time overhead during the charging process and improving system service continuity.

[0018] This invention first constructs a mobile edge computing (MEC) system based on UAV and RIS (Reference-Based Component Analysis) assistance. Then, it employs a deep reinforcement learning algorithm based on a TD3 network to jointly optimize task scheduling, charging strategies, UAV trajectories, RIS phase, and task offloading ratio, effectively addressing the challenges of non-convexity and high dimensionality in the optimization problem. Simulation results demonstrate that, compared to traditional solutions, this system exhibits minimal processing latency and energy consumption, thereby improving the performance of the mobile edge computing system and validating the effectiveness of the proposed algorithm. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a system model according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0021] Figure 3 This is a flowchart of reinforcement learning processing based on the TD3 network in an embodiment of the present invention;

[0022] Figure 4 This is the reward convergence graph of the TD3 algorithm during simulation verification of an embodiment of the present invention. Detailed Implementation

[0023] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention proposes a task offloading optimization method based on UAVs and RIS-assisted MEC systems, the method comprising:

[0026] S1: Construct a mobile edge computing system based on unmanned aerial vehicles (UAVs) and reconfigurable smart surfaces (RIS).

[0027] like Figure 1 As shown, a mobile edge computing system assisted by a drone and RIS is constructed. This mobile edge computing system includes: an edge cloud base station GT (equipped with an edge cloud server) for processing computing tasks, and a drone (UAV) carrying a RIS. RIS The system consists of one unmanned aerial vehicle (UAV) carrying a mobile edge computing (MEC) server to perform task unloading, two fixed charging stations, and a service area with numerous user equipment (UEs). Each UAV has a limited battery capacity and needs to be charged within the task unloading schedule to assist in completing the task unloading. The UAV uses trajectory planning to implement energy consumption control and charging scheduling strategies.

[0028] Assuming that the user equipment (UE) tasks use a Poisson arrival model, and the tasks occur within a time interval... The probability P(k) of reaching k tasks within a given timeframe is expressed as: Where k represents the number of tasks. Indicates the first Task arrival rate per time slot Denotes the factorial of k. It represents the base of the natural logarithm.

[0029] The time interval between the arrival of any two tasks Follows exponential distribution The calculation formula is as follows:

[0030] , .

[0031] Drones and edge cloud base stations provide computing services (including task offloading) to terminals (including user equipment) in a time-division manner, covering the entire communication cycle. Divided into Each time slot, that is, for any time t, .

[0032] In mobile edge computing systems, the set of User Equipments (UEs) is represented as: , M represents the number of user equipments in the system; the height of the user equipments (UEs) above the ground is expressed as... Its three-dimensional Cartesian coordinates are represented as: ; Indicates that the user equipment is in the The two-dimensional ground coordinates corresponding to each time slot Represents the two-dimensional coordinates of the user equipment. This indicates a transpose expression.

[0033] UAV equipped with RIS RIS The height above the ground remains fixed. The drone performing the unloading mission maintains a fixed altitude above the ground. .

[0034] The RIS is represented by the time slot start coordinate as follows ,in, , This indicates the number of reflection units on the RIS. This represents the two-dimensional coordinates of the RIS projected onto the ground, at the end of the time slot. Represented as: .

[0035] The starting coordinates of the UAV performing the unloading mission in time slot t are represented as follows: ,in, , This represents the two-dimensional coordinates of the UAV's projection onto the ground in time slot t. At the end of that time slot, the coordinates are represented as follows: .

[0036] Assuming the edge cloud base station is on the ground, its two-dimensional coordinates are fixed and represented as P. s : The charging station is fixed on the ground, and its two-dimensional coordinates are fixed as follows: : .

[0037] S2: Based on the mobile edge computing system, construct the system's channel model, task generation and priority model, offload latency and energy consumption model, and UAV flight and charging scheduling model.

[0038] S201: Based on the mobile edge computing system, construct the channel model of the system.

[0039] Assuming User Equipments (UEs) move randomly at low speeds within a defined area, in each time slot, a drone flies to a fixed location, hovers, and establishes communication with one of the UEs. The terminal offloads computing tasks to the server according to the allocation plan, and the remaining computing tasks are executed locally. Assuming that the transmitted computing tasks can be partially executed using the drone's server computing resources, the drone can transfer the remaining computing tasks to an edge server for computation.

[0040] Assuming user equipment In uplink communication with a drone, the channel gain of its line-of-sight link is expressed as: The specific calculation formula is as follows:

[0041] (1)

[0042] (2)

[0043] in, Indicates the first Each time slot The European distance between the drone and the drone Represents the Euclidean norm. This represents the two-dimensional coordinates of the drone projected onto the ground during the (t+1) time slot, where the drone is either unloading its task or carrying a mobile edge computing (MEC) server. Indicates user equipment In the The two-dimensional ground coordinates corresponding to each time slot This indicates the altitude of the drone above the ground during the unloading mission. Indicates user equipment Channel gain of line-of-sight link in uplink communication with UAVs This represents the channel gain at a reference distance d=1m.

[0044] The RIS deployed on the drone redirects the signal between the drone and the edge cloud base station GT. The RIS has The reflective elements form a uniform linear array, and the passive phase shift of the RIS requires calculating the phase array that makes up the rectangular array. To simulate the phase shift of the RIS, it is assumed that... Indicates the first The reflecting element at the th ... The phase of each time slot, express Each element in the time slot The phase array at the location. The communication link between the UAV performing the unloading task and the RIS is at the location. Channel gain at each time slot Represented as:

[0045] (3)

[0046] (4)

[0047] in, Or does it mean in the first The Euclidean distance between the time-slot drone and the RIS This represents the two-dimensional coordinates of the UAV projected onto the ground in time slot (t+1). This represents the two-dimensional coordinates of the UAV carrying the RIS projected onto the ground in time slot (t+1). This indicates the altitude of the drone above the ground during the unloading mission. Indicates the height of RIS above the ground. This indicates that the communication link between the drone performing the unloading task and the RIS is in the [missing information - likely a specific location or stage]. Channel gain at each time slot Indicates the signal number The cosine of the angle of arrival (AoA) for each time slot. Indicates the antenna spacing. Indicates the carrier wavelength.

[0048] User equipment The communication link with RIS is in the first Channel gain at each time slot Represented as:

[0049] (5)

[0050] (6)

[0051] in, Indicates the first The Euclidean distance between the time-slot drone and the RIS Indicates user equipment The communication link with RIS is in the first Channel gain at each time slot Indicates the signal number The cosine of the departure angle (AoD) of each time slot.

[0052] When the communication link between the drone and the user equipment is blocked, the channel gain is achieved with the assistance of RIS. Represented as:

[0053] (7)

[0054] in, Indicates the first User equipment serving a time slot, .

[0055] To assess the likelihood of line-of-sight link (LAS) congestion between the drone and the user, an air-to-ground corridor model in an urban environment was considered, where the drone and user equipment... Between time slots Yes, the probability of blocking. Represented as:

[0056] (8)

[0057] in, ;

[0058] This indicates the first constant value that depends on the environment. This represents a second constant value that depends on the environment. Indicates the height of the drone above the ground. express.

[0059] Therefore, the cascaded execution of the task offloading of the drone and user equipment The average channel gain that can be achieved between Represented as:

[0060] (9)

[0061] In the downlink between drones and edge servers (edge ​​cloud base stations), the channel gain of their line-of-sight links. Represented as:

[0062] (10)

[0063] (11)

[0064] in, Indicates the first The Euclidean distance between the time-slot drone and the edge server.

[0065] Assuming drone flight is limited to Within the area, drones and user equipment The communication range is defined as follows:

[0066] (12)

[0067] in, Indicates drones and user equipment The maximum communication range between them.

[0068] User equipment considering occlusion due to obstacles uplink wireless transmission rate with drone Represented as:

[0069] (13)

[0070] in, Indicates user equipment Communication bandwidth with drones This indicates the uplink transmit rate of user equipment (UEs). Indicates noise power.

[0071] The downlink wireless transmission rate between the drone and the edge server (edge ​​cloud base station) is expressed as:

[0072] (14)

[0073] in, This indicates the channel bandwidth between the drone and the edge server. This indicates the downlink transmission power of the drone.

[0074] S202: Based on the mobile edge computing system, construct the task generation and priority model of the system.

[0075] In a mobile edge computing (MEC) system, for each time slot of the user equipment Task generation follows a Poisson distribution. The average number of tasks generated in different time slots is adjusted to simulate peak / slow peak task volume. The amount of tasks generated at that time Represented as:

[0076] ,

[0077] in, Represents the random number generation function of the Poisson distribution. Indicates the first Task arrival rate per time slot This indicates the task arrival rate during peak hours, corresponding to scenarios with high system load. This indicates the task arrival rate during off-peak hours, corresponding to scenarios with lower system load.

[0078] In the In each time slot, when tasks of multiple user devices need to be offloaded, the task offloading strategy for each user device adopts a weighted priority sorting method. The priority function adopts a latency and energy consumption weighted method, which is consistent with the overall optimization goal.

[0079] ,

[0080] in, Indicates the first User equipment in each time slot Weighted priority of task unloading This represents the weighting coefficient between energy consumption and latency. Indicates the first User equipment in each time slot Task processing energy consumption Indicates the first User equipment in each time slot Task processing latency, Indicating the system The average total energy consumption for processing all tasks in a time slot Indicating the system The average total latency for processing all tasks in one time slot.

[0081] By weighted normalizing energy consumption and latency, high-energy-consuming and urgent tasks are processed first, avoiding queuing issues caused by heavy loads and long latency in the mobile edge computing (MEC) system. Simultaneously, prioritizing high-energy-consuming and urgent tasks allows UAVs (User Aerial Vehicles) to be completed when their batteries are fully charged, reducing the secondary waiting costs associated with UAV charging.

[0082] S203: Based on the mobile edge computing system, construct the system's offload latency and energy consumption model.

[0083] In a mobile edge computing (MEC) system, for each time slot of the user equipment The task employs a partial unloading strategy, making Indicates in The proportion of tasks that are offloaded to the drone server that performs the task in a time slot. This indicates the proportion of remaining tasks executed locally on the user equipment terminal, i.e., within a time slot. User device local task processing latency Energy consumption for task processing They are represented as follows:

[0084] (15)

[0085] (16)

[0086] in, Indicates the first User equipment in each time slot The size of the computation task, This indicates the CPU cycles required for the user equipment to process each unit byte. This indicates the calculation frequency of User Equipments (UEs). Indicates user equipment CPU effective capacitance coefficient.

[0087] The user equipment transmits the remaining computing tasks to the drone performing the task offloading, and the user equipment's transmission latency... Represented as:

[0088] (17)

[0089] make Indicates in The proportion of tasks that are offloaded to edge servers in each time slot. This represents the proportion of remaining tasks performed by the drone, and the computational latency generated on the drone's MEC server. Represented as

[0090] (18)

[0091] in, Indicates the first User equipment in each time slot The size of the computation task, This indicates the CPU cycles required for the drone server to process each byte. This indicates the computing frequency of the drone server's CPU.

[0092] In the time slot In the context of offloading computing tasks to a drone server, the transmission energy consumed is... Represented as:

[0093] (19)

[0094] Indicates user equipment Uplink transmit power, This indicates the transmission delay of the remaining computing tasks from the user equipment to the drone.

[0095] The drone MEC server (the drone performing the offloading mission) is in the... Computational energy consumption per time slot Represented as

[0096] (20)

[0097] Where, k u f represents the effective capacitance coefficient of the CPU in the drone MEC server. u This indicates the computing frequency of the drone server's CPU. This indicates the computational latency generated on the drone's MEC server.

[0098] Assuming in time slot In the process, the drone, while handling some offloading tasks, further offloads the data to an edge server (installed on an edge cloud base station) for processing, thus reducing transmission latency. Represented as:

[0099] (twenty one)

[0100] In the time slot The computational latency caused by offloading computational tasks to edge servers Represented as:

[0101] (twenty two)

[0102] in, This indicates the CPU cycles required for the edge server to process each unit byte. This indicates the computing frequency of the edge server's CPU.

[0103] Assuming in time slot The data transfer energy consumption is only considered in relation to the energy consumed in offloading computing tasks from the drone to the edge server, without considering the computing energy consumption of the edge server itself, or the energy consumption of transmitting data to the edge server. Represented as:

[0104] (twenty three)

[0105] in, Indicates in time slot Energy consumption for offloading computing tasks from drones to edge servers This indicates the transmit power of the downlink between the drone and the edge cloud server. This indicates the transmission latency when the drone further unloads the data to the edge server for processing while handling some unloading tasks.

[0106] S204: Based on the mobile edge computing system, construct a drone flight and charging scheduling model.

[0107] In the In each time slot, the drone flies from its initial position to a new hovering position, which is represented as:

[0108] ,

[0109] in, Indicates that the drone is in The ground x-coordinate corresponding to each time slot Indicates that the drone is in Flight speed per time slot This indicates the fixed flight time of the drone. Indicates that the drone is in The flight angle of each time slot.

[0110] Drones with speed and angle The parameters are used for flight motion, and the energy consumed during each flight and hovering is expressed as follows:

[0111] (twenty four)

[0112] (25)

[0113] in, This indicates the flight energy consumption of the drone. This indicates the power factor of the drone. , Indicates the quality of the drone. This indicates the fixed flight time of the drone. The Euclidean norm of a vector. The hovering energy consumption of drones Represents Earth's gravity. This indicates the radius of the rotor blades of a rotary-wing drone. Indicates the number of propellers, This represents the air density at standard atmospheric pressure. This indicates the hovering time of the drone.

[0114] In mobile edge computing (MEC) systems, the size of the server's computation results is usually very small and can be ignored. Transmission latency and energy consumption are not considered in the downlink between the drone and the user.

[0115] In the time slot In the context of the mission, the distance from the drone (equipped with a MEC server) performing the unloading operation to the fixed charging station is... Represented as:

[0116] ,

[0117] in, The Euclidean norm of a vector. Indicates the first The two-dimensional coordinates of the ground projection of the drone charging station in each time slot. This represents the two-dimensional coordinates of the drone projected onto the ground during the (t+1) time slot, where the drone is either unloading its task or carrying a mobile edge computing (MEC) server. This indicates the altitude above the ground of the drone performing the task, either unloading or carrying a mobile edge computing (MEC) server.

[0118] In the time slot In the middle, the distance from the RIS-equipped drone to the fixed charging station Represented as:

[0119] ,

[0120] in, The two-dimensional coordinates of the UAV carrying the RIS projected onto the ground in time slot (t+1). This indicates the height of RIS above the ground.

[0121] In the time slot In the process, the remaining energy of the drone (equipped with a MEC server) that has completed its mission unloading is transferred to a fixed charging station. Represented as:

[0122] ,

[0123] in, Indicates the initial battery capacity of the drone. This indicates the flight energy consumption of the drone. This indicates the hovering energy consumption of the drone. This indicates the energy consumption for transferring computing tasks from the drone to the edge server. This indicates that the drone (equipped with an MEC server) that performed the unloading mission was in the [number]th [year]. The computational energy consumption per time slot.

[0124] exist During the time slot, the remaining energy of the drone equipped with RIS at the charging station is represented as follows:

[0125] .

[0126] Drones equipped with RIS or performing mission offloading in time slots The conditions for triggering charging are:

[0127] ,

[0128] in, This indicates the unit flight energy consumption of the drone. This indicates the distance between the drone performing the unloading mission or the drone equipped with RIS and the charging station.

[0129] S3: Based on the mobile edge computing system, minimize the number of user devices in the system while satisfying system constraints. We construct a joint optimization objective function P for the system, weighted by the maximum processing latency and system energy consumption.

[0130] To ensure the effective utilization of the system's limited computing resources, this invention optimizes UAV scheduling, UAV maneuverability, and computing task allocation within the system jointly to minimize the impact on all user devices. The target is a weighted average of maximum processing latency and system energy consumption.

[0131] exist During time slots, User Equipment (UE) m Maximum processing latency for unloading computing tasks Represented as:

[0132] (26)

[0133] Where max{.} represents finding the maximum value. This indicates the processing latency of tasks executed locally on the user device. This indicates the transmission latency from the user equipment to the drone for the remaining computational tasks. This indicates the transmission latency of a portion of the computational tasks transmitted by the drone performing the offloading mission to the edge cloud server (i.e., the edge cloud base station). This indicates the processing latency of tasks performed by the edge cloud server (i.e., edge cloud base station).

[0134] In the time slot At that time, User Equipment (UE) m System power consumption when unloading computing tasks Represented as:

[0135] (27)

[0136] in, This indicates the energy consumption for transmitting the remaining computing tasks from the user equipment to the drone. This represents the transmission energy consumed when offloading computational tasks to the drone server. This indicates the transmission power consumption of the drone server in processing tasks. This represents the energy consumption of transmitting a portion of the computational tasks from the drone performing the offloading mission to the edge cloud server (i.e., the edge cloud base station). This indicates the flight energy consumption of the drone. This indicates the hovering energy consumption of the drone.

[0137] Based on the aforementioned mobile edge computing system, the maximum processing latency of all user devices in the system is minimized while satisfying system constraints. and system energy consumption To achieve this, we construct a joint optimization objective function P for the system.

[0138] The joint optimization objective function P is expressed as:

[0139] (28)

[0140] in, express The time-slot system provides task offloading services to only one user device. The weights representing system processing latency Indicates in time slot User Equipment (UE) m Maximum processing latency for unloading computing tasks Indicates in time slot User Equipment (UE) m System energy consumption when unloading computing tasks. The weights representing system energy consumption are: T represents the total number of time slots in the entire communication cycle, and M represents the number of user devices in the system. Indicates the first User Equipment (UE) in each time slot m The workload, express Task generation for time-slot user equipment follows a Poisson distribution. Represents the random number generation function of the Poisson distribution. Indicates the first Task arrival rate per time slot This indicates the task arrival rate during peak hours, corresponding to scenarios with high system load. This indicates the task arrival rate during off-peak hours, corresponding to scenarios with lower system load. Indicates the first The overall phase configuration of the RIS in each time slot, This represents the total number of RIS reflection elements. The system represents the total quota of tasks generated by all user equipment in the system. C1 is the first constraint, indicating whether user equipment m is served (i.e., task offloading) in time slot t of the MEC system. C2 is the second constraint, indicating that the MEC system only serves user equipment m in time slot t. C3 is the third constraint, indicating that the tasks generated by user equipment must satisfy a Poisson distribution. C4 is the fourth constraint, indicating the phase constraint that the RIS must satisfy. C5 is the fifth constraint, indicating the position constraint of the UAV carrying the RIS. C6 is the sixth constraint, indicating the position constraint of the UAV performing task offloading. C7 is the seventh constraint, indicating the position constraint of user equipment. C8 is the eighth constraint, indicating the energy constraint of the UAV. C9 is the ninth constraint, indicating the system's task processing volume constraint. C10 is the tenth constraint, indicating the number of charging stations. C11 is the eleventh constraint, indicating the ratio constraint between user equipment offloading tasks and UAVs performing task offloading. C12 is the twelfth constraint, indicating the ratio constraint between UAVs performing task offloading and edge cloud base stations. C13 is the thirteenth constraint, indicating whether two charging stations provide charging services in time slot t.

[0141] S4: Solve the joint optimization objective function P using a reinforcement learning algorithm based on the TD3 network to obtain the task offloading optimization scheme of the mobile edge computing system (MEC).

[0142] The system optimizes in real time the drone scheduling, drone-edge cloud collaborative unloading ratio, and the deflection angle and travel distance of RIS drones and task unloading drones in the system model based on the optimal unloading scheme.

[0143] S401: Construct a Markov decision process based on the joint optimization objective function P.

[0144] Define the state space: in the time slot System state at time , S represents the state space in reinforcement learning, which is the set of all possible states of the system, including the energy and position of the RIS UAV, the energy and position of the task-unloading UAV, the position of the user equipment, the remaining task load of the system, and the task load of the user equipment. Therefore, the time slot... System state at time Represented as:

[0145] (29)

[0146] in, Indicates in time slot The remaining energy of the RIS-equipped drone battery. Indicates in time slot Location information of drones equipped with RIS. Indicates in time slot The remaining energy of the drone's battery after the mission is completed. Indicates in time slot Location information of the drone that performed the unloading mission. Indicates in time slot User equipment served by drones Location information, This indicates the remaining amount of tasks the system needs to complete within the entire time period. Indicates in time slot User equipment The size of the randomly generated task. When hour, , , This represents the total quota for tasks generated by all user devices in the system. express Whether drones equipped with RIS or drones performing mission unloading are charged. When c=0, it means no charging is needed; when c=1, it means charging is needed. C5, C6, C7, C8, C9, and C10 represent the fifth, sixth, seventh, eighth, ninth, and tenth constraints, respectively.

[0147] State normalization is performed to preprocess the observed states, enabling deep neural networks to be trained more effectively. The difference between the maximum and minimum values ​​of each variable is used as a scaling factor to address the magnitude difference between input variables.

[0148] and Having states ranging from different magnitudes can lead to convergence problems during training. Normalizing the input variables to the states helps the training converge.

[0149] Five scaling factors were used in the state normalization algorithm. To reduce the battery capacity of drones. Due to user equipment The drone equipped with RIS and the drone performing the unloading mission have the same x and y coordinate ranges, therefore they use... and To scale up user equipment The x and y coordinates of the RIS-equipped drone and the drone performing the unloading mission. (Using...) To scale the remaining tasks within the entire time period, use To shorten the time period Task size for each User Equipment (UE);

[0150] Defining the action space: The agent selects actions, including those within time slots, based on the system's current state and the observed environment. User equipment to be served The deflection angle and travel distance of the RIS-equipped drone and the drone performing task offloading, the drone task offloading ratio, and the task offloading ratio of further offloading to the edge cloud base station, and the action. It can be represented as:

[0151] (30)

[0152] in, express Time-slot drones select normalized terminal devices for their services. This indicates the flight angle of the drone performing the unloading mission. This indicates the flight angle of the drone equipped with RIS. This indicates the flight speed of the drone performing the unloading mission. This indicates the flight speed of the drone equipped with RIS. Indicates user equipment The proportion of tasks that are offloaded to the server on the drone performing the task offloading. This indicates that the drone performing the task offloading will partially offload the task to another device. T represents the total number of time slots in the entire communication cycle, and M represents the number of user devices in the system. express When a drone equipped with RIS or a drone performing a mission unloading is charging, the charging station selection is as follows: when l=0, it means charging at charging station 1; when l=1, it means charging at charging station 2.

[0153] The TD3 actor network outputs continuous actions for user equipment. The agent selected Discretization is required if ,but ;if ,but ,in It is a rounding function. The drone's flight angle, flight speed, and mission unloading ratio can be precisely optimized in a continuous action space, i.e., angle... ,speed The proportion of unloading to drones and the offloading ratio to the base station The selection of charging stations for devices that need charging C11, C12, and C13 represent the eleventh, twelfth, and thirteenth constraints, respectively.

[0154] Defining the reward function: The agent's policy is reward-based, and choosing an appropriate reward function plays a crucial role in the performance of the TD3 framework. The goal is to maximize the reward by minimizing the weighted average of processing latency and system energy consumption, achieving immediate reward. Represented as:

[0155] (31)

[0156] Among them, in time slots The processing delay is expressed as follows:

[0157] ,

[0158] in, Indicates in time slot Time service user equipment The system processing latency,

[0159] In the time slot The system energy consumption at that time is expressed as:

[0160] ,

[0161] in, Indicates in time slot Time service user equipment The system energy consumption, and if when ,but ,otherwise .

[0162] S402: Based on the TD3 network structure, construct the corresponding Actor network and Critic network structures.

[0163] The Actor network uses the state as the input layer, three fully connected layers, and the output layer, and uses ReLU and tanh as activation functions, specifically including:

[0164] (32)

[0165] in, Let W1 represent the output vector of the i-th fully connected layer, ReLU(.) represent the ReLU activation function, and W1, W2, and W3 represent the weight matrices of the 1st, 2nd, and 3rd fully connected layers in the Actor network, respectively. , , Let represent the bias vectors for the 1st, 2nd, and 3rd fully connected layers, respectively, and let Tanh(.) represent the Tanh activation function. This represents the weight matrix of the output layer of the Actor network. This represents the bias vector of the output layer of the Actor network. This indicates the final scaling action. This represents a scaled action vector used for interaction with the environment. This represents the raw action output of the Actor network.

[0166] The Actor network uses the output layer of the tanh activation function to generate corresponding actions. The Actor network scales the output action within a range; the Critic network takes the state and action as inputs and outputs of the fully connected layers respectively. The concatenated outputs are then mapped to Q-values ​​through the last fully connected layer and a linear activation function in the output layer. This is used to evaluate the quality of the actions generated by the Actor network.

[0167] S403: Based on Markov decision-making, a dynamic optimization algorithm based on TD3 is used to optimize the drone-edge cloud collaborative offloading ratio and the optimal trajectory of the RIS drone and the mission offloading drone.

[0168] The overall framework of the proposed algorithm is as follows: Figure 3 As shown, the network training process specifically includes:

[0169] (1) Initialize network parameters: using parameters , , , , and Initialize the Actor network separately Actor Target Network Critic1 network Critic2 network Critic1 target network and Critic2 target network At the same time, initialize the experience replay buffer;

[0170] (2) The Actor network is based on the current state Generate Actions The intelligent agent performs actions The state of the next time slot was then observed. and instant rewards and update status Then, the experience tuples Stored in an experience replay buffer of size M;

[0171] (3) The agent randomly samples a batch of experience tuples from the experience replay buffer. The next action is generated through the target Actor network, and the target Q-value is calculated as follows:

[0172] (33)

[0173] (34)

[0174] in, The parameters represent the target Actor network. The parameters represent the target Critic network. It is noise added to the output of the Actor target network to prevent overfitting of the Actor. The strategy generated for the target Actor network. For instant rewards, As a discount factor, For the next time slot state, The Q-function represents the target Q-value network.

[0175] TD error is expressed as:

[0176] (35)

[0177] (36)

[0178] in, Represents the target Q value. For time slots The system status, This represents the Q-value function under the current policy. Assigning actions to the drone for task scheduling, flight maneuvers, and task unloading. These are the parameters for the Critic network.

[0179] Update the Critic network using gradient descent:

[0180] (37)

[0181] (38)

[0182] in, This represents the parameters of the first Critic network Q1. This represents the learning rate of the Critic network, controlling the step size for parameter updates. (.) denotes the gradient operator with respect to the Q1 parameters of the Critic network, and N represents the number of samples (batch size) sampled from the empirical replay pool at each update. This represents the squared temporal difference (TD) error of the Critic network Q1. This represents the parameters of the second Critic network Q2. (.) denotes the gradient operator with respect to the Q2 parameters of the Critic network. This represents the squared temporal difference (TD) error of the Critic network Q2.

[0183] Update the Actor network every d steps and calculate the Actor policy gradient. Represented as:

[0184] (39)

[0185] Update the Actor network using gradient ascent:

[0186] (40)

[0187] Update the target network using the Polyak Averaging method:

[0188] (41)

[0189] (42)

[0190] (43)

[0191] in, This represents the coefficient for soft updates.

[0192] Repeat the above steps until training is complete to optimize the drone's flight path, including flight distance and angle, and ultimately obtain the optimal trajectory planning.

[0193] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0194] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A task offloading optimization method based on UAVs and RIS-assisted MEC systems, characterized in that, include: Construct a mobile edge computing (MEC) system model based on UAV and RIS assistance. The system includes: one edge cloud base station and one UAV equipped with RIS. RIS One unmanned aerial vehicle (UAV) to perform unloading tasks, two fixed charging stations, and several user equipment (UEs); Based on the Mobile Edge Computing (MEC) system, construct the system's channel model, task generation and priority model, offload latency and energy consumption model, and UAV flight and charging scheduling model. Based on the aforementioned Mobile Edge Computing (MEC) system, the maximum processing latency of all user devices in the system is minimized while satisfying system constraints. and system energy consumption To achieve the goal, construct the joint optimization objective function P for this system; The joint optimization objective function P is solved using a reinforcement learning algorithm based on the TD3 network, resulting in a task offloading optimization scheme for the mobile edge computing system (MEC).

2. The task unloading optimization method according to claim 1, characterized in that, The joint optimization objective function P is: ， ， in, This indicates the user of the drone task scheduling service. Indicates in time slot The maximum processing latency for unloading computation. Indicates system energy consumption. This represents the set of user devices in the system. This represents the two-dimensional coordinates of the UAV carrying the RIS on the ground in time slot t+1. This represents the two-dimensional coordinates of the UAV's ground projection during the unloading of its mission in time slot t+1. The weights represent the maximum processing latency. The weights representing system energy consumption express Task generation for time-slot user equipment follows a Poisson distribution. Represents the random number generation function of the Poisson distribution. Indicates the first Task arrival rate per time slot This indicates the task arrival rate during peak hours. This indicates the task arrival rate during off-peak hours. This indicates RIS phase constraint. Indicates in The proportion of tasks that will be offloaded to the drone server that will execute the task will be determined by the time slot. Indicates in The proportion of tasks that are offloaded to edge servers in each time slot. Let T represent the number of time slots in the entire communication cycle, and M represent the number of user equipment in the system. This indicates the number of reflection units on the RIS. This represents the total quota of tasks generated by all user devices in the system. C1 represents the first constraint, C2 represents the second constraint, C3 represents the third constraint, C4 represents the fourth constraint, C5 represents the fifth constraint, C6 represents the sixth constraint, C7 represents the seventh constraint, C8 represents the eighth constraint, C9 represents the ninth constraint, C10 represents the eleventh constraint, C11 represents the eleventh constraint, C12 represents the twelfth constraint, and C13 represents the thirteenth constraint.

3. The task unloading optimization method according to claim 1, characterized in that, The channel model includes: In user equipment In uplink communication with a UAV, calculate the channel gain of its line-of-sight link in time slot t. ; In the communication link between the UAV performing the unloading task and the RIS, the calculation of the first... Channel gain at each time slot ; In user equipment In the communication link with RIS, the calculation is performed in the first... Channel gain at each time slot ; When the communication link between the drone and the user equipment is blocked, calculate the channel gain implemented with RIS assistance. ; Calculate the unloading of drones and user equipment for the task execution achievable average channel gain ; In the downlink communication link between the drone and the edge server (edge ​​cloud base station), calculate the channel gain of the line-of-sight link in time slot t. ; Assuming drone flight is limited to Within the area, drones and user equipment The communication range is defined as follows: ， in, This represents the two-dimensional coordinates of the UAV's ground projection during the unloading of its mission in time slot t+1. Indicates user equipment In the The two-dimensional ground coordinates corresponding to each time slot Indicates drones and user equipment The maximum communication range between them.

4. The task unloading optimization method according to claim 1, characterized in that, The task generation and prioritization model includes: exist Time slot user equipment Generated task volume Represented as: ， in, Represents the random number generation function of the Poisson distribution. Indicates the first Task arrival rate per time slot This indicates the task arrival rate during peak hours. This indicates the task arrival rate during off-peak hours; In the In a time slot, when tasks from multiple user devices need to be offloaded, the system adopts a weighted priority sorting method for the task offloading strategy of each user device, specifically as follows: ， in, Indicates the first User equipment in each time slot Weighted priority of task unloading This represents the weighting coefficient between energy consumption and latency. Indicates the first User equipment in each time slot Energy consumption for task processing Indicates the first User equipment in each time slot Task processing latency, Indicating the system The average total energy consumption for processing all tasks in a time slot Indicating the system The average total latency for processing all tasks in one time slot.

5. The task unloading optimization method according to claim 1, characterized in that, The unloading delay and energy consumption models respectively include: In a time slot, the user equipment executes a task locally and calculates its task processing latency. Energy consumption for task processing ; In the time slot, the user equipment transmits the remaining computing tasks to the drone that has performed the task unloading, and calculates the transmission latency of the user equipment. Calculate the computational latency of the drone performing the unloading task. ; In time slots, computational tasks are offloaded to the drone performing the offloading task, and the transmission energy consumed is calculated. Calculate the computational energy consumption of the drone performing the unloading task. ; In the time slot, the drone performing the task offloading will offload part of the task to the edge cloud base station for processing, and the transmission delay will be calculated separately. Calculation delay and transmission energy consumption .

6. The task unloading optimization method according to claim 1, characterized in that, The UAV flight and charging scheduling model includes: Time slots are used to calculate the flight energy consumption of drones. and hovering energy consumption ; The time slot is used to calculate the distance from the drone unloading its mission to the drone charging station. Calculate the distance from the RIS-equipped drone to the drone charging station. ; Time slots are used to calculate the remaining energy of drones that have been unloaded from their missions and are now at drone charging stations. Calculate the remaining energy of the RIS-equipped drone to the drone charging station. ; During a time slot, the conditions for triggering charging for drones equipped with RIS or drones performing mission unloading are as follows: ， in, This indicates the unit flight energy consumption of the drone. This indicates the distance between the drone performing the unloading mission or the RIS-equipped drone and the charging station.

7. The task unloading optimization method according to claim 1 or 2, characterized in that, exist Time slot, User Equipment (UE) m Maximum processing latency for unloading computing tasks Represented as: ， Where max{.} represents finding the maximum value. This indicates the processing latency of tasks executed locally on the user device. This indicates the transmission latency from the user equipment to the drone for the remaining computational tasks. This indicates the transmission latency of a portion of the computational tasks transmitted by the drone performing the offloading mission to the edge cloud server (i.e., the edge cloud base station). This indicates the processing latency of tasks executed by the edge cloud server (i.e., the edge cloud base station). exist Time slot, User Equipment (UE) m System power consumption when unloading computing tasks Represented as: ， in, This indicates the energy consumption for transmitting the remaining computing tasks from the user equipment to the drone. This represents the transmission energy consumed when offloading computational tasks to the drone server. This indicates the transmission power consumption of the drone server in processing tasks. This represents the energy consumption of transmitting a portion of the computational tasks from the drone performing the offloading mission to the edge cloud server (i.e., the edge cloud base station). This indicates the flight energy consumption of the drone. This indicates the hovering energy consumption of the drone.

8. The task unloading optimization method according to claim 2, characterized in that, The joint optimization objective function P is solved using a reinforcement learning algorithm to obtain the optimal UAV-edge cloud collaborative offloading ratio, the charging schedule of the RIS UAV and the task offloading UAV, and the optimal trajectory, which specifically includes: A Markov decision process is constructed based on the joint optimization objective function P, defining the state space, action space, and reward function; Based on the TD3 network structure, construct the corresponding Actor network and Critic network structures; Based on Markov decision-making, a dynamic optimization algorithm based on TD3 is used to optimize the unloading ratio of UAV-edge cloud collaborative operation, the charging scheduling of RIS UAV and mission unloading UAV, and the optimal trajectory.

Images