Federal learning resource management method for unmanned aerial vehicle assisted mobile edge computing MEC

Abstract

The invention belongs to the technical field of unmanned aerial vehicle communication networks, and discloses a federated learning resource management method for unmanned aerial vehicle assisted mobile edge computing MEC, and the method comprises the steps: modeling a federated learning resource management problem in an unmanned aerial vehicle assisted MEC system as a Markov decision process MDP; based on the Markov decision process, strategy optimization is carried out by using a deep reinforcement learning algorithm; according to the deep reinforcement learning algorithm, a framework based on a standard deviation Q target SQT is adopted, a plurality of Q networks are integrated, and the standard deviation of Q values of the Q networks is calculated, so that a target Q value containing uncertainty punishment is generated, and network parameters are updated; and outputting unmanned aerial vehicle selection and resource allocation actions according to the real-time unmanned aerial vehicle auxiliary MEC system state through the trained strategy network. According to the method, federal learning precision and system resource consumption can be effectively balanced, and resource management efficiency is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unmanned aerial vehicle communication networks, and particularly relates to a federated learning resource management method for unmanned aerial vehicle assisted mobile edge computing (MEC). BACKGROUND

[0002] With the continuous expansion of the scale of mobile intelligent devices and the increasing demand for low-latency and high-bandwidth services, mobile edge computing (MEC) as a new paradigm that sinks computing and storage capabilities to the network edge can effectively alleviate the data transmission pressure of the core network and improve service response speed, and has become a key technology to support local intelligent processing. In this context, unmanned aerial vehicles are widely considered as an important component of building dynamic and flexible edge networks due to their inherent high mobility, rapid deployment capability and flexible line-of-sight link advantages, and are particularly suitable for application scenarios where ground infrastructure is scarce or damaged, such as emergency communication and temporary regional coverage. Therefore, unmanned aerial vehicle assisted MEC systems provide a feasible infrastructure solution for edge intelligent applications such as autonomous driving and industrial Internet of Things that have strict real-time requirements.

[0003] At the same time, the deepening deployment of edge intelligent applications faces the dual challenges of data privacy and communication overhead. Federated learning as a distributed machine learning framework, by distributing the model training process to local user devices for execution, cooperatively learns only by exchanging model parameters rather than raw data, significantly reducing data transmission volume while effectively protecting user data privacy and security. Introducing federated learning mechanism into unmanned aerial vehicle assisted MEC network is expected to achieve efficient intelligent collaboration among edge nodes and provide a new technical path for model training tasks in resource-constrained and topology time-varying network environments.

[0004] However, as a mobile edge node, the practical application of unmanned aerial vehicles is strictly restricted by on-board energy reserves and hardware computing capabilities. Limited battery capacity directly determines the time that the unmanned aerial vehicle can sustainably provide services, and limited processor performance affects the efficiency of local computing tasks. These physical constraints make it impossible to infinitely expand the computing and communication resources in the unmanned aerial vehicle assisted MEC system, and efficient scheduling and management must be carried out within strict resource boundaries.

[0005] In addition, the wireless channel state between the UAV and the edge server has significant time-varying characteristics. In actual systems, the acquisition of channel state information usually relies on periodic measurement and feedback mechanisms, and there are inevitably estimation errors and feedback delays. Therefore, the resource scheduling decision is often made based on imperfect channel information, which causes the power control and rate matching strategy to deviate from the optimal state, thereby causing waste of system resources. The existence of imperfect channel state information brings severe challenges to the efficiency of resource management in dynamic wireless environments.

[0006] On the other hand, during the federated learning training process, the server needs to dynamically select part of the UAVs to participate in global model updating in each training round. The number and quality of participating UAVs directly affect the model convergence performance and system energy efficiency. If too few UAVs are selected, the model accuracy may decrease and the convergence may be slow due to insufficient training data. If too many UAVs are selected, the total system energy consumption will be significantly increased, and the training may be timed out due to waiting for the slowest node. Therefore, how to intelligently select the participating UAV set and efficiently allocate multi-dimensional resources while meeting the model accuracy requirement is the key to improving the overall performance of the system.

[0007] In summary, the existing technology has not effectively solved the problem of how to balance the federated learning training accuracy and the total system energy consumption in a complex dynamic environment where the UAVs are energy-limited, the computing power is limited, and the channel state information is imperfect. This core problem restricts the further improvement of the federated learning performance in the UAV-assisted MEC system, and there is an urgent need for a solution that can adapt to environmental changes and implement intelligent management of cross-layer resources. SUMMARY

[0008] In view of the problem that in the existing UAV-assisted mobile edge computing system, the federated learning resource allocation cannot effectively balance the model training accuracy and the total system energy consumption in a dynamic environment where the energy is limited, the computing resources are limited, and the channel state information is imperfect, the present application provides a federated learning resource management method for a UAV-assisted mobile edge computing (MEC) system. The method is based on the joint optimization of Markov decision process and standard deviation Q target deep reinforcement learning, and aims to adaptively adjust the UAV selection, transmission power and computing frequency through an intelligent decision mechanism to improve the resource management efficiency and system performance.

[0009] To achieve the above technical purposes, the present application specifically adopts the following technical solutions: In one aspect of the present application, a federated learning resource management method for a UAV-assisted mobile edge computing (MEC) system is provided, comprising the following steps: The federated learning resource management problem in the UAV-assisted MEC system is modeled as a Markov decision process (MDP); wherein, a state space is defined to represent the channel state information, local data volume and residual energy of each UAV before each training round; an action space is defined to represent the selection decision of each UAV, the allocation of uplink transmission power and CPU frequency in each training round; and a reward function is defined to balance the training accuracy of the federated learning model and the total energy consumption of the system. Based on the Markov decision process, a deep reinforcement learning algorithm is used for policy optimization; the deep reinforcement learning algorithm adopts a standard deviation Q target (SQT) based framework, integrates multiple Q networks and calculates the standard deviation of the Q values to generate a target Q value containing uncertainty penalty, thereby updating the network parameters. Through the trained policy network, UAV selection and resource allocation actions are output according to the real-time UAV-assisted MEC system state.

[0010] In one embodiment, the state space is specifically represented as:

[0011] wherein, , respectively represent the uplink channel estimation value and the estimation error of the UAV u; , respectively represent the downlink channel estimation value and the estimation error of the UAV u; represents the local data set size; represents the allocated bandwidth; represents the residual battery energy; represents the set of UAVs.

[0012] In one embodiment, the uncertainty of the channel state information is represented as a channel estimation error, and the actual channel state information is represented as a channel estimation value and a channel estimation error :

[0013] wherein, the channel estimation error obeys a circularly symmetric complex Gaussian distribution.

[0014] In one embodiment, when calculating the effective noise of the uplink signal, the interference caused by the channel estimation error is taken into account, and the effective noise is represented as:

[0015] wherein, is an additive white Gaussian noise, P (t) is the transmission power of the UAV u in the t-th round, is the amplitude of the signal.

[0016] In one embodiment, based on the effective noise, the lower limit of the signal-to-noise ratio of the uplink is:

[0017] wherein, denotes the bandwidth allocated by the server to the UAV u in the t-th round of training, is the noise power spectral density, denotes the total noise and interference in the signal transmission.

[0018] In one embodiment, the action space is specifically represented as:

[0019] wherein, denotes the UAV selection indicator variable, denotes the uplink transmission power, denotes the CPU frequency.

[0020] In one embodiment, the reward function is defined as:

[0021] wherein, denotes the accuracy of the federated learning in the t-th round; denotes the weight coefficient; denotes the total energy consumption of the selected UAV u in the t-th round; denotes the selected set of UAVs in the t-th round; denotes the state of the state space at time t; denotes the action of the action space at time t.

[0022] In one embodiment, the target Q value is calculated according to the formula:

[0023] wherein, denotes the immediate reward; is the discount factor; is the set-based Q value operator; is a penalty parameter between 0 and 1; denotes the average of the standard deviations of the Q values of the batches .

[0024] In one embodiment, the is calculated by:

[0025]

[0026] wherein, is a batch sampled from the experience replay buffer; the i-th Q-network predicts the value of state-action pair ; n denotes the total number of Q-networks.

[0027] In an embodiment, the objective function optimized by the method is the total accuracy-energy gain AE of all training rounds:

[0028] wherein, is the accuracy of the t-th round of federated learning, is a weight coefficient, denotes the total energy consumption of the selected drones u in the t-th round, denotes the set of selected drones in the t-th round.

[0029] The application has the following beneficial effects: 1) By modeling the federated learning resource allocation problem as a Markov decision process and constructing a deep reinforcement learning algorithm based on the standard deviation Q target framework, joint dynamic optimization of drone selection, transmission power and calculation frequency is realized under the condition of imperfect channel state information. This method effectively overcomes the lack of dynamic adaptability of traditional resource allocation schemes to the environment.

[0030] 2) The use of a multi-Q network integrated architecture and an uncertainty penalty mechanism significantly reduces the Q value estimation bias and improves the stability and convergence performance of the strategy. The algorithm can still maintain robust performance in complex edge network environments.

[0031] 3) The proposed accuracy-energy balance optimization mechanism considers both model training accuracy and system energy consumption in the reward function, achieving collaborative optimization of federated learning performance and resource utilization efficiency. This method significantly extends the continuous working time of the drone-assisted edge computing system while ensuring model convergence accuracy.

[0032] 4) The constructed end-to-end resource management framework has online learning and adaptive adjustment capabilities, and can dynamically adjust the decision strategy according to real-time channel state, device energy level and calculation load, providing reliable resource support for federated learning applications in dynamic edge environments. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of the federated learning resource management method for drone-assisted mobile edge computing (MEC) in the embodiments of the application. DETAILED DESCRIPTION

[0034] The technical solutions of the present application will be described clearly and completely in combination with the specific embodiments below, but those skilled in the art will understand that the following described embodiments are part of the embodiments of the present application, not all the embodiments, and are only used to illustrate the present application, and should not be regarded as limiting the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts are within the scope of protection of the present application.

[0035] The present application aims to solve the problem of low efficiency of federated learning resource allocation under limited computing resources and imperfect channel conditions in a UAV-assisted mobile edge computing environment. By designing a resource management algorithm that can adapt to dynamic changes in the environment, joint optimization of UAV selection, computing and communication resource allocation is achieved to balance federated learning training accuracy and system energy consumption.

[0036] Based on the federated learning architecture, the present application combines Markov decision process and standard deviation Q-target (Std Q-target, SQT) deep reinforcement learning algorithm to realize intelligent resource scheduling of UAVs in dynamic edge environment. The system is modeled under imperfect channel state information conditions, taking the energy state and local data volume of UAVs as state variables, and using deep reinforcement learning algorithm to jointly optimize the selection of UAVs, transmission power and CPU frequency, so as to balance the federated learning training accuracy and system energy consumption.

[0037] In one specific embodiment, a federated learning resource management method for a UAV-assisted mobile edge computing (MEC) system is provided, comprising the following steps: S1, model the federated learning resource management problem in the UAV-assisted MEC system as a Markov decision process (MDP); wherein, define the state space to represent the channel state information, local data volume and remaining energy of each UAV before each training round; define the action space to represent the selection decision of each UAV, the allocation of uplink transmission power and CPU frequency in each training round; define the reward function to balance the training accuracy of the federated learning model and the total system energy consumption.

[0038] In some embodiments, the UAV-assisted MEC system includes a federated learning server and multiple UAVs, and each training round includes three stages: UAV selection and global model distribution; local model training and updating; global model aggregation.

[0039] Specifically, the selection of the UAVs and the global model are issued for each round of federated learning training, which is completed by the federated learning server, and the UAV nodes participating in the training of this round are screened out, and the model is initialized. The server makes decisions according to an optimized resource management strategy, which considers the real-time state of the system, such as the residual energy of the UAV, the channel condition of each UAV, the size of the local data, etc. Through the decision, the server determines the set of UAVs selected to participate in the training in this round.

[0040] In the local model training and updating stage, the computing task is transferred from the center server to the selected edge UAVs and executed in parallel, fully utilizing distributed computing resources. Each selected UAV trains the model using its own local data set after receiving the global model. Each data set comes from different patrol areas or sensing targets of the UAV, so it usually has the characteristics of non-independent and identically distributed, which helps the global model to learn more extensive data features. The UAV uses the received global model as the initial model and executes multiple cycles of optimization algorithms such as stochastic gradient descent on its local data, finally updates and obtains a new set of local model parameters. After training is completed, all selected UAVs upload their trained local model parameters to the federated learning server through the uplink.

[0041] In the global model aggregation stage, the server waits and collects the local model parameters uploaded by all selected UAVs, and aggregates them into a new generation of global model. It can be understood that the server does not simply average all models, but performs weighted aggregation according to the local data volume of each UAV. Specifically, the larger the data volume of the UAV, the more "right to speak" its local model has in aggregation. After aggregation is completed, the server replaces the old model with the new global model, and then the system will determine whether to start the next round of training (until the preset number of rounds or the accuracy requirement is met).

[0042] In some embodiments, the federated learning resource management problem includes a communication model, a time model, an energy consumption model, and an optimization objective. Specifically as follows: In the communication model, considering the uncertainty of channel state information, the actual channel gain is represented as:

[0043] wherein, is the actual channel state information; is the channel estimation value; is the channel estimation error, which is subject to a circularly symmetric complex Gaussian distribution.

[0044] The received signal is:

[0045] wherein, is the uplink and downlink signal sent by the tth round uth UAV to the server; is the amplitude of the signal (expectation is 0, variance is 1); is the additive white Gaussian noise with power spectral density . is the transmission power of the UAV u in the tth round.

[0046] The effective noise includes Gaussian noise and additional noise caused by channel estimation error:

[0047] wherein, is the effective noise of the uplink and downlink.

[0048] The lower limit of the signal-to-noise ratio and the transmission rate are respectively:

[0049]

[0050] wherein, is the lower limit of the signal-to-noise ratio of the uplink; denotes the bandwidth allocated by the server to the UAV u in the tth round of training; is the noise power spectral density; denotes the total noise and interference in signal transmission; is the transmission rate of the UAV u in the tth round.

[0051] In the time model, the total time of each training round is determined by the slowest UAV:

[0052] wherein, denotes the time for the UAV u to download the global model in the tth round, denotes the time for the UAV u to train the local model in the tth round, denotes the time for the UAV u to upload the local model in the tth round.

[0053] In the energy consumption model, the total energy consumption of the UAV is the sum of the calculation energy consumption and the upload energy consumption:

[0054] wherein, is the energy consumed by the UAV u in training the local model in the tth round, is the energy consumed by the UAV u in uploading the local model in the tth round.

[0055] In the optimization objective, the objective function is the total accuracy-energy gain AE of all training rounds:

[0056] wherein, is the accuracy of the tth round of federated learning, is the weight coefficient, denotes the total energy consumption of the selected drones u in the tth round, denotes the selected drone set in the tth round.

[0057] and an optimization problem is established to maximize the gain while satisfying seven types of constraint conditions, including the drone transmission power constraint, the CPU frequency constraint, the drone energy constraint, the drone selection number constraint, the federated learning accuracy constraint, the system total bandwidth constraint, and the per-round training delay constraint.

[0058] In some embodiments, the resource management problem is modeled as a Markov decision process, including a state space, an action space, and a reward function.

[0059] The state space is specifically represented as:

[0060] wherein, , denote the uplink channel estimation value and estimation error of drone u, respectively; , denote the downlink channel estimation value and estimation error of drone u, respectively; denotes the local dataset size; denotes the allocated bandwidth; denotes the remaining battery energy; denotes the set of drones.

[0061] The action space is specifically represented as:

[0062] wherein, denotes the drone selection indicator variable, denotes the uplink transmission power, denotes the CPU frequency.

[0063] The reward function is defined as:

[0064] wherein, denotes the federated learning accuracy of the tth round; denotes the weight coefficient; denotes the total energy consumption of the selected drones u in the tth round; denotes the selected drone set in the tth round; denotes the state of the state space at time t; denotes the action of the action space at time t.

[0065] S2, based on the Markov decision process, a deep reinforcement learning algorithm is used for policy optimization; the deep reinforcement learning algorithm adopts a standard deviation Q target SQT-based framework, integrates multiple Q networks and calculates the standard deviation of the Q values to generate a target Q value containing uncertainty penalty, thereby updating the network parameters.

[0066] In the initialization phase, the online policy network π, the online value network Q, the target network and the experience replay buffer are initialized, and the hyperparameters such as the discount factor γ, the update frequency, the exploration noise, etc. are set.

[0067] In the training phase, before each round of training, the environment state is reset, and the action is selected according to the current policy network After the action is executed, the system collects the following information: The local training time, upload time and total energy consumption of the selected drone ; The accuracy of federated training ; The immediate reward , the next state and the termination flag d Store the experience tuple ) to the replay buffer.

[0068] When the replay buffer reaches a certain capacity, a batch is randomly sampled, and the Q value standard deviation of the state-action pair in the batch is calculated:

[0069] Wherein, The prediction value of the i-th Q network for the state-action pair , and n represents the total number of Q networks.

[0070] Calculate the average standard deviation of the batch:

[0071] Construct a target Q value containing uncertainty penalty:

[0072] Wherein, Indicates the immediate reward; Is the discount factor; Is a set-based Q value operator; Is a penalty between 0 and 1.

[0073] Repeat the above process until the preset training round threshold is reached.

[0074] S3, the policy network trained, outputs UAV selection and resource allocation actions according to the real-time system state.

[0075] Through the above training, an optimized policy network is obtained. In the real-time federated learning round, according to the current system state , the optimized policy network is used to generate actions , including UAV selection decision, uplink transmission power of each UAV, and CPU frequency of each UAV. The policy maximizes the AE gain under the constraint condition, and improves the overall performance of federated learning.

[0076] Embodiments Referring to Figure 1 , the embodiments implement the following steps: Step 1: system modeling and algorithm initialization 1) In the hierarchical federated learning system, the global model on the federated learning server and the local model on the UAV are trained in each round. The maximum value of the training round is T, denoted as =[1,2...T]. Each UAV has a local data set in each training round, which represents the local data set of UAV u in the t-th round. The t-th round of federated learning training includes the following key steps: UAV selection and global model delivery: the server selects some suitable UAVs from the resource request state (such as the remaining energy of the UAV) and sends the global model to the selected UAVs through the downlink. denotes the set of UAVs selected to participate in the t-th round of federated learning training. is the set of UAVs.

[0077] Local model update and upload: each selected UAV u uses its local data set to train the model and update the local model parameters. All selected UAVs upload their trained local models to the federated learning server.

[0078] Global model aggregation and update: the server collects the local models of all selected UAVs, and aggregates them into a new generation of global model according to the data volume weighting.

[0079] 2) In the communication process, due to the inherent time-varying nature of the channel and the delay of the feedback, it is difficult to obtain perfect channel state information. The uncertainty of the channel state information will be reflected in the channel estimation error. Therefore, the actual channel state information is represented as the estimated value of the channel plus the channel estimation error.

[0080]

[0081] wherein, is the channel estimation value of the drone u in the tth round, is the channel estimation error, which is distributed according to a Circularly Symmetric Complex Gaussian Distribution (CSCG). The uplink and downlink signals can be expressed as:

[0082] is the uplink and downlink signal sent by the u th drone to the server in the tth round. is the amplitude of the signal (expectation is 0, variance is 1). is an additive white Gaussian noise with power spectral density The effective noise includes not only the additive white Gaussian noise, but also additional noise due to the consideration of channel estimation error. This part of the additional noise is due to the inaccuracy of channel estimation, so it needs to be considered in the total noise. The effective noise is as follows:

[0083] wherein, is the effective noise of the uplink and downlink. The variance of the uplink and downlink is . denotes the bandwidth allocated by the server to the drone u in the tth round of training.

[0084] Thus, the lower bound of the effective Signal-to-Interference-plus-Noise Ratio (SINR) of the communication link can be obtained:

[0085] wherein, is the transmission power of the drone u in the tth round, denotes the bandwidth allocated by the server to the drone u in the tth round of training, is the noise power spectral density, denotes the total noise and interference in signal transmission. The transmission rate of the u th drone in the tth round is:

[0086] 3) In the federated learning framework, after the drone completes the local model training, it needs to upload the trained local model to the federated learning server. This process is considered as a parallel operation, allowing selected drones to upload local models simultaneously. In order to obtain the cumulative time cost of the tth round This embodiment mainly considers the time spent on computation and communication. Since the federated learning server has strong computing power, the time required for drone selection and global model aggregation is ignored. The time required for the completion of the tthround of federated learning is determined by the slowest drone:

[0087] where, denotes the time for drone u to download the global model in the tthround, denotes the time for drone u to train the local model in the tthround, denotes the time for drone u to upload the local model in the tthround. Similarly, because the energy of the federated learning server is very sufficient, the energy consumed on the federated learning server is not calculated, and only the energy consumption of the drone is considered. The energy consumption of the selected drone is composed of the computation energy consumption and the upload energy consumption:

[0088] where, is the energy consumed by drone u when training the local model in the tthround, is the energy consumed by drone u when uploading the local model in the tthround.

[0089] 4) In order to jointly optimize the energy consumption and the delay, it is necessary to evaluate the performance of federated learning. Training accuracy is one of the most commonly used performance indicators of federated learning. If fewer drones are selected to participate in the training process in a round of training, the effect of reducing energy consumption can be achieved, but at the same time the accuracy of federated learning will be reduced, and vice versa.

[0090] Considering the relationship between the minimization of the energy consumption of the selected drone and the maximization of the accuracy of federated learning, the accuracy-to-energy (AE) gain is defined as:

[0091] where, is the accuracy of the tthround of federated learning, is a weight coefficient. In order to optimize the drone selection and resource allocation strategy, the AE gain of the system is maximized under the premise of meeting various constraint conditions. The optimization problem can be modeled as P0:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] where, is the UAV selection indicator, = 1, means that the UAV u is selected by the federated learning server in the tth round. is the frequency of the CPU of the UAV u in the tth round. is the energy of the UAV u at the beginning of the tth round. Transmission power limit: the uplink transmission power of the UAV u must be within the power range allowed by its device. CPU frequency limit: the frequency of the CPU is limited by the hardware capability and must be within the frequency range limited by the hardware. UAV energy limit: each selected UAV must have enough energy to complete the training of the local model and the uploading of the local model. UAV selection limit: at least one UAV must be selected to participate in each round of federated learning training. Federated learning training accuracy limit: the training accuracy of each round must meet the minimum model accuracy threshold. Total bandwidth allocation limit: the sum of the bandwidth allocated by the selected UAVs in each round cannot exceed the total system bandwidth. Time limit: the time required for federated learning to complete must be less than the preset maximum delay.

[0100] 5) In each round of federated learning, each UAV acts as an agent to make UAV selection and resource optimization decisions to maximize the system's AE gain in a distributed environment. In this way, the optimization problem P0 can be re-described as a Markov decision process problem.

[0101] The state space includes: uplink channel estimation, uplink channel estimation error, downlink channel estimation, downlink channel estimation error, local dataset size, bandwidth allocation, and remaining battery energy: .

[0102] The action space includes: UAV selection variable, UAV uplink transmission power, and frequency of UAV CPU: .

[0103] The reward function is defined as the difference between the training accuracy and the weighted energy consumption to balance these two key indicators: .

[0104] 6) Initialize the online policy network, online value network, target network, and experience replay buffer. Set simulation parameters, such as discount factor, update frequency, and exploration noise. This process is crucial for the algorithm's smooth transition to the training phase, laying a solid foundation for subsequent optimization and learning.

[0105] Step 2: Environment Reset and Task Distribution Before each training round begins, the system resets the simulation environment and initializes the state space. In the t-th round of federated learning, to achieve effective interaction between each drone and the environment, the following steps need to be performed: First, based on the online policy network... Select Action This decision-making process forms the basis for the interaction between the drone and its environment; subsequently, the federated learning server determines the appropriate actions based on these actions. Select a suitable drone and allocate uplink transmission power to it for the current training round. and CPU power To support the implementation of federated learning, this allocation process ensures that drones can effectively participate in federated learning training.

[0106] Step 3: Local Training and Policy Update The selected drone begins training a local model and uploads the local model to the federated learning server; then the time is calculated. Energy consumption and training accuracy The selected drones interact with the environment to receive instant rewards. Next state and the termination flag d; finally, the result of this interaction process Collect and store in the experience replay buffer.

[0107] Throughout the round, to update the policy network and value network, the following steps are performed: Once the experience replay buffer reaches a certain size, a mini-batch of samples is randomly sampled from the replay experience buffer for training. These samples are used to calculate the action value function. Then calculate the standard deviation. and through standard deviation Obtain the target value This process is a crucial step in network learning and optimization, ensuring that the network can effectively adjust based on historical data. Finally, based on these calculation results, the parameters of the policy network and the value network are updated.

[0108]

[0109]

[0110] wherein, denotes the result of averaging the standard deviations of the Q-networks for the batch . is a batch sampled from the experience replay buffer, containing multiple state-action pairs. The i-th Q-network predicts the value for the state-action pair . n denotes the total number of Q-networks. The standard deviation of the predicted values of all Q-networks for the state-action pair is calculated:

[0111] wherein, is the target Q-value, is the immediate reward, which is the direct feedback of the behavior effect of the agent, reflecting the behavior effect of the agent in the current state . is the discount factor, which balances the importance of the current reward and the future reward, is the set-based Q-value operator. is the uncertainty penalty term, is a penalty parameter between 0 and 1, controlling the strength of the uncertainty penalty.

[0112] Step 4 iteration termination and policy output: After completing the policy update, enter the training round termination judgment stage. In order to ensure that the SQT network can fully evaluate the UAV selection and resource allocation scheme in each round, it is necessary to judge whether the preset training round number is reached. If not, the iteration number t is increased until it exceeds the set threshold . This repeated process is a key link in network optimization, the purpose of which is to gradually adjust and improve the policy through multiple iterations, so that more efficient selection and allocation can be achieved in different rounds.

[0113] Although the embodiments of the present application are described above in combination with the drawings, the present application is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and guiding, but not limiting. Those skilled in the art can make many forms under the inspiration of the present application and without departing from the scope protected by the claims of the present application, which are all within the scope of protection of the present application.

Claims

1. A federated learning resource management method for UAV-assisted mobile edge computing (MEC), characterized in that, Includes the following steps: The federated learning resource management problem in the UAV-assisted MEC system is modeled as a Markov decision process (MDP). The state space is defined to represent the channel state information, local data volume, and remaining energy of each UAV before each training round. The action space is defined to represent the selection decision, uplink transmission power, and CPU frequency allocation for each UAV in each training round. The reward function is defined to balance the training accuracy of the federated learning model with the total energy consumption of the system. Based on the Markov decision process, a deep reinforcement learning algorithm is used for policy optimization. The deep reinforcement learning algorithm adopts a framework based on standard deviation Q-objective SQT, which integrates multiple Q networks and calculates the standard deviation of their Q values ​​to generate a target Q value that includes uncertainty penalty, thereby updating the network parameters. The trained policy network outputs drone selection and resource allocation actions based on the real-time drone-assisted MEC system status.

2. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The state space is specifically represented as follows: in, , Let represent the uplink channel estimate and estimation error of UAV u, respectively; , Let U and U represent the downlink channel estimate and estimation error, respectively. Indicates the size of the local dataset; Indicates the allocated bandwidth; Indicates the remaining battery energy; This refers to a collection of drones.

3. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The uncertainty of the channel state information manifests as channel estimation error and actual channel state information. Represented as channel estimate With channel estimation error sum: Wherein, the channel estimation error It follows a cyclically symmetric complex Gaussian distribution.

4. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 3, characterized in that, When calculating the effective noise of the uplink signal, the interference caused by the channel estimation error is taken into account, and the effective noise... Represented as: in, It is additive white Gaussian noise. For the transmission power of drone u in round t, The amplitude of the signal.

5. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 4, characterized in that, Based on the effective noise, the lower limit of the uplink signal-to-noise ratio. for: in, This represents the bandwidth allocated by the server to drone u in the t-th round of training. For noise power spectral density, This represents the total noise and interference in signal transmission.

6. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The action space is specifically represented as follows: in, Indicates the selection of indicator variables for the drone. Indicates uplink transmission power. This indicates the CPU frequency.

7. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The reward function is defined as follows: in, This represents the accuracy of federated learning in round t; Indicates the weighting coefficient; This represents the total energy consumption of the selected drone u in round t; Let represent the set of drones selected in round t; This represents the state of the state space at time t; This represents the action in the action space at time t.

8. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The target Q value The calculation formula is: in, Indicates an immediate reward; It is a discount factor; It is a set-based Q-value operator; It is a penalty parameter between 0 and 1; Indicates batch The result is the average of the standard deviations of the Q values.

9. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 8, characterized in that, The Calculated in the following way: in, It is a batch sampled from the experience playback buffer; The i-th Q-network pair of state-action The predicted value; n represents the total number of Q networks.

10. The federated learning resource management method for UAV-assisted mobile edge computing (MEC) according to claim 1, characterized in that, The objective function optimized by the method is the total accuracy-energy consumption (AE) gain across all training rounds. in, It is the accuracy of the t-th round of federated learning. These are weighting coefficients. This represents the total energy consumption of the selected drone u in round t. This represents the set of drones selected in round t.

Images