Multi-network multi-granularity resource optimization method based on federated deep reinforcement learning

Abstract

The application discloses a multiple network multi-granularity resource optimization method based on federal deep reinforcement learning, and belongs to the technical field of edge computing and artificial intelligence, and steps include: a mobile device perceives a local resource state, and maps the local resource state into a structured state vector through a multi-granularity resource mapping module; local FDRL training and generation of a global prompt vector; pruning optimization is performed on the local FDRL training model, and a multiple heterogeneous hybrid network module realizes asynchronous uploading based on a heterogeneous link adaptation mechanism; global model aggregation, prompt distillation and joint optimization; policy back and iterative closed loop restart. The application introduces a three-level + multi-granularity state modeling method to enhance expression capability; an offline trajectory guiding mechanism is introduced to realize fast convergence in a policy cold start stage; experience priority sampling + prompt generation is adopted to accelerate policy convergence; the local FDRL training model is pruned to reduce uploading overhead, and a heterogeneous link adaptation mechanism is adopted to improve transmission efficiency and robustness.

Description

Technical Field

[0001] This invention relates to the fields of edge computing and artificial intelligence technology, specifically to a multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning. Background Technology

[0002] With the widespread deployment of 5G communication, industrial IoT, edge computing, and smart mobile devices, modern computing environments are rapidly moving towards a diversified, heterogeneous, and distributed development stage. Mobile devices are diverse in type and capabilities, resulting in a "multi-network" architecture where multiple communication methods, such as Wi-Fi, 5G, and satellite links coexist. This network heterogeneity brings significant scheduling complexity, especially under conditions of rapid changes in resource status and unstable link quality, making it difficult for traditional static scheduling models to effectively adapt to system operational requirements.

[0003] Meanwhile, the resource requirements of system tasks are becoming increasingly complex: for example, tasks such as factory production line fault early warning and identification, and real-time environmental status analysis require millisecond-level responses to prevent the spread of system anomalies; while tasks such as complex industrial drawing recognition and equipment status big data modeling place higher demands on computing resources and communication throughput; these intelligent sensing tasks require low latency and high reliability, and some computationally intensive tasks require high bandwidth and high computing resources. The diversified and dynamic characteristics of these tasks manifest as significant "multi-granularity resource requirements," that is, resources from the network level (bandwidth, load), device level (computing power, power consumption) to the business level (task latency, reliability) present multi-level and multi-dimensional scheduling indicators. However, existing resource scheduling mechanisms often only consider a single granularity or average load, lacking the ability to accurately model and control the diversity of resources in complex systems.

[0004] To address these challenges, reinforcement learning (RL) has been widely applied to resource allocation problems in edge computing due to its ability to optimize long-term returns. However, traditional reinforcement learning methods often rely on centralized control architectures, requiring large amounts of raw data to be transmitted back to a central server for unified training. This not only incurs high communication burdens but also poses serious privacy risks.

[0005] Federated Learning (FL), a distributed learning paradigm proposed in recent years, achieves a collaborative optimization strategy of "data not leaving the local device" by distributing model training tasks locally and uploading only model parameters. This demonstrates significant advantages in protecting data privacy and reducing communication costs. Combining Federated Learning with Reinforcement Learning forms the Federated Deep Reinforcement Learning (FDRL) system, becoming a key path to improve the intelligence level of edge systems.

[0006] However, existing federated deep reinforcement learning systems still face the following technical bottlenecks when dealing with real-world deployment scenarios characterized by "multi-network heterogeneity and multi-granularity resource constraints":

[0007] 1) Poor adaptability to mixed and multi-layered networks and low transmission efficiency;

[0008] Existing methods generally assume a single communication link (such as only 5G or Wi-Fi), making it difficult to flexibly select upload paths in heterogeneous access environments with multiple networks such as 5G, Wi-Fi, and satellite. This leads to increased communication latency and unstable system response in scenarios of network congestion or interruption.

[0009] 2) Resource modeling has a limited granularity and weak expressive power;

[0010] Most existing methods only focus on macro-level resource indicators (such as total computing power and bandwidth), lacking support for modeling micro-level, fine-grained resources such as device cache, queue length, and remaining battery power. This makes it difficult to accurately match diverse task requirements, resulting in low resource utilization.

[0011] 3) The strategy has difficulty in cold start and the model converges slowly;

[0012] Most existing methods lack effective prior knowledge guidance during the policy initialization phase and rely on random initialization, resulting in unstable initial training and slow convergence speed, especially with large performance fluctuations in scenarios with heterogeneous devices or uneven data distribution.

[0013] 4) Lack of personalized decision-making mechanisms leads to crude scheduling results;

[0014] Some existing methods use a unified model and strategy to schedule all mobile devices in a unified manner, without taking into account the differences between devices and the diversity of task requirements. This can easily lead to problems such as poor generalization of scheduling results and mismatch of resource allocation.

[0015] 5) High communication and computing overhead, resulting in a heavy system load;

[0016] In scenarios where communication resources are scarce or energy consumption is sensitive, existing methods frequently perform full model uploads and synchronizations, lacking effective model compression and asynchronous upload mechanisms, which can easily lead to communication bottlenecks and energy waste.

[0017] Based on this, the present invention designs a multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning to solve the above problems. Summary of the Invention

[0018] To address the aforementioned shortcomings of existing technologies, this invention provides a multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning.

[0019] To achieve the above objectives, the present invention provides the following technical solution:

[0020] A multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning includes the following steps:

[0021] Step 1: System initialization;

[0022] Step 2: The mobile device perceives the local resource status, performs fine modeling of heterogeneous resources, and divides the resource status into three structural levels: device level, network level, and service level. Each structural level is further subdivided into coarse-grained and fine-grained levels, which are then mapped into structured state vectors by the multi-granularity resource mapping module.

[0023] Step 3: Local FDRL training and global cue vector generation, the specific steps are as follows:

[0024] (3.1) The federated deep reinforcement learning controller is based on the local experience pool collected in real time on each mobile device i. Global cue vectors uploaded by the local FDRL training module An offline trajectory-guided mechanism is used to perform local FDRL training initialized by a federated coordination server, and a personalized local initialization strategy is built. ;

[0025] Personalized local initialization strategy The formula is as follows:

[0026]

[0027] in, For strategy fusion weights, For localized, personalized prior strategies; The initial global policy issued by the federal coordination server;

[0028] (3.2) Construct personalized prompt vectors , Generated by fusing state trajectories, action trajectories, and resource metrics recorded by the mobile device during local FDRL training:

[0029]

[0030] in, This indicates the feature extraction function; This represents the historical state sequence of mobile device i. Represents a historical sequence of actions; Represents the historical reward sequence;

[0031] Each mobile device records the training latency during local FDRL training. Training energy consumption pruning rate, transmission delay With transmission energy consumption And send it to the multi-heterogeneous hybrid network module;

[0032] Step 4: Multiple heterogeneous hybrid network modules based on , , , and Calculate the optimal pruning ratio Based on the optimal pruning ratio Structural compression is performed on the locally trained FDRL model to generate a lightweight local policy parameter vector. By calculating the utility functions of different links Mobile devices can independently choose the appropriate upload path;

[0033] Step 5: The federated coordination server periodically aggregates lightweight local policy parameter vectors. ; Integrate personalized prompt vectors from various mobile devices Complete the cue distillation and generate a unified global cue vector. ; Constructing includes , , and A multi-objective reward function;

[0034] Step 6: The federated coordination server will send the global policy parameter vector for the next round. The next round of global cue vectors and importance weight parameters Distribute to various mobile devices.

[0035] Furthermore, step 1 specifically involves the initialization phase, where the federated coordination server will uniformly set the deep reinforcement learning model parameter vector. The local FDRL training module is distributed to each mobile device i to initialize the local FDRL training model of the local FDRL training module, and is based on the initial cue vector of each mobile device i. The global hint vector is obtained by weighted aggregation according to certain weights. :

[0036]

[0037] in, is the aggregation weight of mobile device i; M represents the number of mobile devices currently participating in the federated learning task; This represents the initial cue vector for each mobile device i.

[0038] Furthermore, coarse-grained resource state vectors This includes, but is not limited to: link bandwidth utilization, device CPU computing power load, and network congestion level; fine-grained resource state vectors. This includes, but is not limited to: cache utilization, task queue length, single-task computation latency, and remaining energy level.

[0039] Furthermore, the multi-granularity resource mapping module maps the coarse-grained resource state vectors of each structural level of the mobile device. With fine-grained resource state vector After being normalized and encoded according to preset rules, the data are concatenated to generate a unified structured state vector. :

[0040]

[0041] In this context, || represents the concatenation operation.

[0042] Furthermore, local experience pool The construction method is as follows:

[0043] (1) The local FDRL training module of each mobile device i uploads the action vector. For the federated deep learning controller; action vectors Represented as:

[0044]

[0045] in, For participation, The pruning rate for training a local FDRL model. Select the action vector for the current upload channel. This indicates the action performed by the mobile device i;

[0046] (2) Each mobile device i records the local training trajectory. And cached in the built local experience pool. middle;

[0047] in, A structured state vector representing the current state. A structured state vector representing the new state entered after an action is performed. Represents the action vector of mobile device i The structured state vector in the current state Instant rewards obtained;

[0048] in, The calculation formula is:

[0049]

[0050] in, These are the importance weight parameters for training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

[0051] Furthermore, during local FDRL training, the local experience pool is used... When extracting samples, priority is assigned based on the temporal difference error of the samples:

[0052]

[0053] in, Let represent the temporal difference error of the i-th trajectory sample, and let represent the prediction bias of the current strategy for that trajectory sample. Discount factor; The new action vector represents the structured state vector of the mobile device in the new state. The next step is to select a new action vector to execute; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward;

[0054] The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula:

[0055]

[0056] Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

[0057] Furthermore, the optimal pruning ratio The calculation formula is:

[0058]

[0059] in, Indicates the optimal pruning ratio; The pruning rate for training the local FDRL model;

[0060] The formula is:

[0061]

[0062] in, For lightweight local policy parameter vectors; Prune represents pruning operations; Used to control the degree of sampling bias; For personalized local initialization strategies; The parameter vector for a uniformly defined deep reinforcement learning model.

[0063] Furthermore, utility function The calculation formula is:

[0064]

[0065] in, This indicates that the utility function value is calculated under different links, and the link with the largest result value is taken as the optimal link; k represents the link type. This represents the utility weighting coefficient; This represents the communication delay of link k; This indicates the available bandwidth of link k; This represents the energy consumption for transmitting data on link k; This represents the packet loss rate of link k.

[0066] Furthermore, step 5 specifically involves the following steps:

[0067] (5.1) The federated coordination server is based on the uploaded lightweight local policy parameter vector. Aggregate the vectors to obtain the global policy parameter vector for round t+1. :

[0068]

[0069] in, This is the sum of the local sample sizes across all mobile devices. Let be the number of data samples used locally by the i-th mobile device. Let be the lightweight local policy parameter vector of mobile device i in round t;

[0070] (5.2) Federated Coordination Server for Personalized Prompt Vectors Distillation and fusion are performed to obtain the global cue vector for the next round. :

[0071]

[0072] in, This indicates the feature extraction function. This represents the global hint vector for the next round;

[0073] (5.3) Federal coordination server collects training latency Training energy consumption Transmission delay With transmission energy consumption Design a multi-objective reward function:

[0074]

[0075] Where U represents the multi-objective reward function, Indicates whether mobile device i participates in the current round of federated training; M represents the number of mobile devices currently participating in the federated learning task. These are the importance weight parameters corresponding to training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

[0076] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning.

[0077] Compared with the prior art, the beneficial effects of this invention are as follows:

[0078] This invention introduces a "three-level (network level, device level, service level) + multi-granularity (coarse-grained and fine-grained)" state modeling method to achieve multi-dimensional dynamic abstraction of resource states, providing a more accurate and structured perception basis for policy input, realizing the unity of the breadth and depth of resource perception, improving the generalization and robustness of policy optimization in actual deployment scenarios, and helping to improve resource utilization.

[0079] This invention introduces an offline trajectory guidance mechanism (DTR) during local FDRL training. By loading historical task trajectories on the device side, a local personalized prior policy is generated and initialized with the global policy. This enables rapid convergence and stable progress during the policy cold start phase, significantly improving the system's generalization and adaptability in multi-task and multi-environment scenarios.

[0080] By prioritizing learning from samples that are difficult to predict or have large policy biases, the local FDRL training module can adjust the training scope more quickly (e.g., due to the number of mobile devices from 0 to j). (This avoids situations where the cumulative reward is significantly higher than the average, leading to a large deviation from the overall target), thus preventing the system from getting trapped in local optima. Furthermore, this method exhibits stronger adaptability in environments with multiple network access points and significantly different tasks on heterogeneous devices, contributing to improved training efficiency and generalization ability of the entire system.

[0081] This DTR module not only improves sample utilization and reduces cold start time, but also significantly enhances the policy convergence instability caused by cross-device differences and asynchronous communication in federated training. It is especially suitable for edge intelligent systems with severe latency fluctuations and inconsistent training task complexity under multiple network access.

[0082] This invention employs a parallel mechanism of "experience-first sampling + prompt generation". On the one hand, it accelerates strategy convergence by enhancing training efficiency and stability, and on the other hand, it provides rich knowledge representation for personalized distillation through prompt construction.

[0083] This invention can autonomously perform model compression and upload path selection, achieving flexible resource backhaul and communication optimization in hybrid multi-network architectures. Upload overhead can be reduced by pruning the local FDRL training model, and transmission efficiency and robustness can be improved through a heterogeneous link adaptation mechanism. This fully unleashes the participation capabilities of resource-constrained mobile devices, enhancing the adaptability and scalability of the entire system under bandwidth-constrained or multi-network heterogeneous conditions.

[0084] This invention combines federated parameter aggregation with a prompting distillation mechanism to enhance knowledge transfer and sharing between models while preserving the individuality of local policies. This addresses the problems of weak generalization ability of personalized models and poor robustness of global models in existing solutions, resulting in a more adaptable system. By setting a multi-objective reward function with multiple performance indicators, factors such as training latency, energy consumption, and transmission costs are modeled uniformly. Furthermore, a prompting mechanism guides the model's learning direction, effectively reducing energy overhead and communication rounds during policy convergence while improving policy accuracy.

[0085] Comprehensive experimental results show that the accuracy of this invention is as high as 97.23%, the communication volume is 4.65MB, and the end-to-end latency of key services is maintained at around 0.93s, demonstrating clear practical application prospects and industrialization value. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0087] Figure 1 This is a flowchart of the multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning of the present invention.

[0088] Figure 2 This is a comparison of the accuracy of this invention with other federated learning strategies.

[0089] Figure 3 This is a comparison of the communication volume of this invention with other federated learning strategies.

[0090] Figure 4 This is a comparison of the latency of this invention with other federated learning strategies.

[0091] Figure 5 This illustrates how the accuracy of this invention differs from other federated learning strategies over training rounds.

[0092] Figure 6 This illustrates how the communication volume between this invention and other federated learning strategies changes with training rounds. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0094] Example 1: Please refer to the accompanying drawings in the instruction manual. Figure 1 The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning has the following specific steps:

[0095] Step 1: System initialization;

[0096] During the initialization phase, the federated coordination server will uniformly set the parameter vectors of the deep reinforcement learning model. The local FDRL training module is distributed to each mobile device i to initialize the local FDRL training model of the local FDRL training module, and is based on the initial cue vector of each mobile device i. The global hint vector is obtained by weighted aggregation according to certain weights. :

[0097]

[0098] in, The aggregation weight of mobile device i (e.g., mobile device data volume, training quality, or mobile device contribution); M represents the number of mobile devices currently participating in the federated learning task; This represents the initial cue vector for each mobile device i.

[0099] The local FDRL training module weights and aggregates the resulting global cue vector. and local experience pools of each mobile device It is then passed to the federated deep reinforcement learning controller.

[0100] Step 2: The mobile device perceives the local resource status and maps it into a structured state vector through the multi-granularity resource mapping module; specifically, it includes the following steps:

[0101] (2.1) Resource collection and classification

[0102] Before executing a task, each mobile device i needs to be aware of its current resource status and perform detailed modeling of heterogeneous resources in advance, dividing the resource status into three structural levels:

[0103] device level (Pruning rate, buffer, power consumption, etc.);

[0104] Network level (Bandwidth, load, network type, link type, etc.);

[0105] Business level (QoS requirements such as task type, latency / reliability);

[0106] Furthermore, each structural level is further subdivided into coarse-grained (stability indicators) and fine-grained (dynamic indicators), which respectively constitute the coarse-grained resource state vector and the fine-grained resource state vector:

[0107] coarse-grained resource state vector This includes, but is not limited to: link bandwidth utilization, device CPU computing power load, and network congestion level;

[0108] Fine-grained resource state vector This includes, but is not limited to: cache utilization, task queue length, single-task computation latency, and remaining energy level;

[0109] (2.2) Structured mapping

[0110] The multi-granularity resource mapping module maps each type of structural level of the mobile device. and After being normalized and encoded according to preset rules, the data are concatenated to generate a unified structured state vector. :

[0111]

[0112] Where || represents the concatenation operation, and the structured state vector The input represents the structured state of the current environment of mobile device i (such as local cache state, channel conditions, remaining battery power, etc.).

[0113] (2.3) Structured state vector The input is embedded into the local FDRL training model of the federated deep reinforcement learning controller as training parameters.

[0114] This invention introduces a "three-level (network level, device level, service level) + multi-granularity (coarse-grained and fine-grained)" state modeling method to achieve multi-dimensional dynamic abstraction of resource states, providing a more accurate and structured perception basis for policy input, realizing the unity of the breadth and depth of resource perception, improving the generalization and robustness of policy optimization in actual deployment scenarios, and helping to improve resource utilization.

[0115] Step 3: Local FDRL Training and Global Cue Vector Generation

[0116] (3.1) In order to improve the stability of federated deep reinforcement learning and increase the utilization of policy samples, the federated deep reinforcement learning controller introduces an offline trajectory guidance (DTR) mechanism during local FDRL training.

[0117] Under the federated learning framework, each network domain has a local FDRL training module (including a local FDRL training model) deployed.

[0118] Specifically, the federated deep reinforcement learning controller is based on the local experience pool collected in real time on each mobile device. Global cue vectors uploaded by the local FDRL training module The offline trajectory guidance (DTR) mechanism is used to perform local FDRL training initialized by the federated coordination server, and a personalized local initialization strategy is built. ;

[0119] Personalized local initialization strategy The formula is as follows:

[0120]

[0121] in, For strategy fusion weights, For localized, personalized prior strategies; The initial global policy issued by the federal coordination server.

[0122] Traditional federated deep reinforcement learning often requires a large number of interactive rounds to converge when the mobile device starts learning directly from the global policy, resulting in poor performance in the early stages. Each mobile device i has different hardware capabilities, network conditions, and task preferences; relying entirely on a uniform global policy will lead to poor performance in local environments.

[0123] In the system startup phase, to improve model cold start efficiency and enhance personalized adaptability, the federated deep reinforcement learning controller first constructs a locally initialized policy model. (Local experience pool) The system provides rich offline trajectory samples for DTR, and DTR learns and generates guidance strategies based on offline trajectory data; global cue vectors. To provide additional global information and guidance signals to the Offline Trajectory Guidance (DTR), the DTR utilizes global cue vectors to better align trajectory data from different local models, improving the accuracy and consistency of trajectory guidance. Considering the interaction experience accumulated by mobile device i in historical tasks, each mobile device i loads the Offline Trajectory Guidance (DTR) mechanism locally and trains a local personalized prior policy. Subsequently, the federated deep reinforcement learning controller incorporates local personalized prior policies at a certain ratio. Global policies issued by the federal coordination server Generate local initialization strategy Used for subsequent training. Local initialization strategy. It inherits the universality of the whole system while retaining the adaptability of the device itself, so that the model has a certain task adaptability from the beginning, which can greatly reduce the performance trough during the cold start phase and improve the efficiency of local task completion.

[0124] This invention introduces an offline trajectory guidance mechanism (DTR) during local FDRL training. By loading historical task trajectories on the device side, a local personalized prior policy is generated and initialized with the global policy. This enables rapid convergence and stable progress during the policy cold start phase, significantly improving the system's generalization and adaptability in multi-task and multi-environment scenarios.

[0125] Among them, the local experience pool The construction method is as follows:

[0126] Upload action vectors to the local FDRL training module on mobile device i. Previously, the parameter vectors of the deep reinforcement learning model in the initial rounds... The local FDRL-trained model of the new federated deep reinforcement learning controller undergoes structured compression, i.e., pruning, with a pruning rate of [missing information]. The initial setting is 0.2.

[0127] Each mobile device i's local FDRL training module uploads action vectors. For the federated deep learning controller; action vectors Represented as:

[0128]

[0129] in, The participation flag (0 or 1). The pruning rate for training a local FDRL model. Select the current upload channel (5G, Wi-Fi, Satellite), and use the action vector. This indicates the action performed by the mobile device i (such as pruning rate selection, participation in decision-making, upload channel selection, etc.).

[0130] Each mobile device records the local training trajectory. And cached in the built local experience pool. middle;

[0131] in, A structured state vector representing the current state. The structured state vector representing the new state entered after the action is executed, i.e., the structured state vector of the state at the next moment; Represents the action vector of mobile device i The structured state vector in the current state The immediate rewards obtained (used to update the strategy during training) are then fed back into the local experience pool. Optimize sample quality itself;

[0132] in, The calculation formula is:

[0133]

[0134] in, These are the importance weight parameters for training latency, communication latency, training energy consumption, and communication energy consumption, respectively. They are usually negative numbers (representing costs) calculated based on metrics such as latency and energy consumption.

[0135] (3.2) In the local FDRL training process, a weighted priority replay mechanism is introduced.

[0136] From the local experience pool When drawing samples, instead of sampling with equal probability, priority is assigned based on the temporal difference error (TD-error) of the samples:

[0137]

[0138] in, Let represent the temporal difference error of the i-th trajectory sample, and let represent the prediction bias of the current strategy for that trajectory sample. Discount factor To measure the importance of future rewards; The new action vector represents the structured state vector of the mobile device in the new state. You can then choose to execute new action vectors, including actions such as pruning; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward.

[0139] The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula:

[0140]

[0141] Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

[0142] By prioritizing learning from samples that are difficult to predict or have large policy biases, the local FDRL training module can adjust the training scope more quickly (e.g., due to the number of mobile devices from 0 to j). (This avoids situations where the cumulative reward is significantly higher than the average, leading to a large deviation from the overall target), thus preventing the system from getting trapped in local optima. Furthermore, this method exhibits stronger adaptability in environments with multiple network access points and significantly different tasks on heterogeneous devices, contributing to improved training efficiency and generalization ability of the entire system.

[0143] This DTR module not only improves sample utilization and reduces cold start time, but also significantly enhances the policy convergence instability caused by cross-device differences and asynchronous communication in federated training. It is especially suitable for edge intelligent systems with severe latency fluctuations and inconsistent training task complexity under multiple network access.

[0144] (3.3) Construct personalized prompt vectors

[0145] Personalized hint vectors Capture the preferences and strategic characteristics of mobile devices. It is generated by fusing state trajectories, action trajectories, and resource metrics (such as latency, energy consumption, and pruning rate) recorded by the mobile device during local FDRL training.

[0146]

[0147] in, This indicates the feature extraction function; Represents the historical state sequence of mobile device i (i.e., several) This indicates the trend of resource status changes; It represents the historical action sequence and the device's preferred behavior (such as preference for high pruning / preferential Wi-Fi upload, etc.). This represents the historical reward sequence and the policy effect, such as latency / energy consumption performance.

[0148] Each mobile device records the training latency during local FDRL training. Training energy consumption pruning rate, transmission delay With transmission energy consumption Performance indicators, etc.

[0149] Training delay The calculation formula is as follows:

[0150]

[0151] in, The participation flag (0 or 1) indicates that C represents the amount of computational resources required per bit. Represents the dataset Number of CPU cycles required This refers to the device's computing power.

[0152] Training energy consumption The calculation formula is as follows:

[0153]

[0154] in, This represents the CPU's unit power consumption constant.

[0155] Transmission delay The calculation formula is as follows:

[0156]

[0157] in, Size for training the local FDRL model. For the transmission rate of mobile device i, The pruning rate for training a local FDRL model should satisfy... , This indicates the upper limit of the pruning rate.

[0158] Transmission power consumption The calculation formula is as follows:

[0159]

[0160] in, This refers to the transmit power of the mobile device.

[0161] After several rounds of local FDRL training, , , , , and They are then sent together to the multi-heterogeneous hybrid network module.

[0162] This invention employs a parallel mechanism of "experience-first sampling + prompt generation". On the one hand, it accelerates strategy convergence by enhancing training efficiency and stability, and on the other hand, it provides rich knowledge representation for personalized distillation through prompt construction.

[0163] Step 4: Prune and optimize the local FDRL training model; implement asynchronous uploading of multiple heterogeneous hybrid network modules based on heterogeneous link adaptation mechanism.

[0164] Multi-heterogeneous hybrid network module: refers to a hybrid heterogeneous network environment with multiple communication protocols, multi-layer architecture, and multiple service QoS requirements, including but not limited to the following network domains: 5G communication network, Wi-Fi local area network, Bluetooth low power network, etc. Each network domain is configured with an independent resource agent unit to monitor local resource status and network performance indicators.

[0165] Pruning primarily addresses the issue of high computational and communication overhead in locally trained FDRL models within federated learning scenarios. It is a method to optimize the overall federated learning framework. Pruning can compress models while maintaining their accuracy, significantly reducing the overall model size. The appropriate pruning ratio can be calculated based on the compression requirements of different models, thereby reducing communication overhead and the communication load between clients and servers in federated learning. It is suitable for low-speed networks or low-power devices.

[0166] The multiple heterogeneous hybrid network modules receive the pruning rate from the federated deep reinforcement learning controller. Training delay Training energy consumption Transmission delay and transmission energy consumption Calculate the optimal pruning ratio :

[0167]

[0168] in, Indicates the optimal pruning ratio; Pruning rate for locally trained FDRL models, in real-time priority case When bandwidth efficiency is prioritized, When precision and stability are prioritized, .

[0169] Based on the optimal pruning ratio Structural compression is performed on the locally trained FDRL model (e.g., pruning some low-importance neurons or channels) to generate lightweight local policy parameter vectors. ; It is passed to the federal coordination server for global aggregation, and then leads the next iteration.

[0170] The formula is:

[0171]

[0172] in, For lightweight local policy parameter vectors; Prune represents pruning operations; Used to control the degree of sampling bias; For personalized local initialization strategies; The parameter vector for a uniformly defined deep reinforcement learning model.

[0173] Because mobile devices i may be in different network environments (e.g., some support 5G high-speed access, while others can only connect to Wi-Fi or low-Earth orbit satellites) in multi-heterogeneous and mixed network scenarios, a heterogeneous link adaptation mechanism is designed. This mechanism calculates the utility function of different links. Mobile devices can autonomously select a suitable upload path to achieve asynchronous upload;

[0174] Utility function The calculation formula is:

[0175]

[0176] in, This indicates that the utility function value is calculated under different links, and the link with the largest result value is taken as the optimal link; k represents the link type. This represents the utility weighting coefficient, used to adjust the importance of latency, bandwidth, energy consumption, and packet loss rate in decision-making; This represents the communication delay of link k; This indicates the available bandwidth of link k; This represents the energy consumption for transmitting data on link k; This represents the packet loss rate of link k (between 0 and 1), and a smaller value is better;

[0177] Example of asynchronous upload:

[0178] 1) With a latency of <10ms and a bandwidth of >100Mbps, tasks with high real-time performance and low latency are prioritized for uploading via the 5G link to ensure rapid synchronization of control commands and key model parameters;

[0179] 2) For tasks with a latency of 10-50ms and a bandwidth of 30-100Mbps that are considered medium bandwidth tolerance, Wi-Fi networks are used for transmission to balance communication costs and timeliness.

[0180] 3) Tasks with latency > 100ms and bandwidth < 20Mbps have a high latency tolerance and can be uploaded asynchronously via satellite link to alleviate the competition pressure in congested networks.

[0181] Subsequently, the mobile device selects the optimal link (such as 5G, Wi-Fi, or satellite network) based on the current network conditions and then uses the lightweight local policy parameter vector. Personalized prompt vectors Uploaded to the federal coordination server to achieve optimal utilization of heterogeneous network links.

[0182] This invention can autonomously perform model compression and upload path selection, achieving flexible resource backhaul and communication optimization in hybrid multi-network architectures. Upload overhead can be reduced by pruning the local FDRL training model, and transmission efficiency and robustness can be improved through a heterogeneous link adaptation mechanism. This fully unleashes the participation capabilities of resource-constrained mobile devices, enhancing the adaptability and scalability of the entire system under bandwidth-constrained or multi-network heterogeneous conditions.

[0183] Step 5: Global model aggregation, hint distillation, and joint optimization

[0184] The federated coordination server periodically aggregates lightweight local policy parameter vectors uploaded from various network domains of multiple heterogeneous and promiscuous network modules. ; Integrate personalized prompt vectors from various mobile devices Complete the cue distillation and generate a unified global cue vector. In addition, the federated coordination server also includes various performance metrics (training latency). Training energy consumption Transmission delay With transmission energy consumption (a) multi-objective reward function.

[0185] The specific steps are as follows:

[0186] (5.1) The federated coordination server is based on the uploaded lightweight local policy parameter vector. Aggregate the vectors to obtain the global policy parameter vector for round t+1. :

[0187]

[0188] in, This is the sum of the local sample sizes of all mobile devices, which is the total global sample size. Let be the number of data samples used locally by the i-th mobile device. Let be the lightweight local policy parameter vector for mobile device i in round t, which is generated locally by the mobile device based on... (Or the aggregation parameters from the previous round) are obtained through training.

[0189] (5.2) The federated coordination server, as the leader, performs global policy aggregation and constructs a system-level resource scheduling policy model, that is, for personalized prompt vectors Distillation and fusion are performed to obtain the global cue vector for the next round. :

[0190]

[0191] in, This indicates the feature extraction function. This represents the global hint vector for the next round, used for issuing the next round of policies.

[0192] (5.3) The federated coordination server collects model energy constraint parameters, communication latency parameters, and performance metrics (such as training latency) uploaded by all mobile devices. Training energy consumption Transmission delay With transmission energy consumption Design a multi-objective reward function, which sets or adjusts the importance weight parameters of training latency, communication latency, training energy consumption, and communication energy consumption according to the application scenario. Adjusted Instant rewards in step 3 The calculation, by increasing instant rewards This can incentivize the local experience pool. The purpose of sample quality optimization; the adjusted This also provides feedback on the optimal pruning ratio in step 4. The calculations are used to meet the different needs of mobile devices for real-time performance, bandwidth, and accuracy.

[0193] The formula for the multi-objective reward function is:

[0194]

[0195] Where U represents the multi-objective reward function, which is mathematically a comprehensive cost function; the smaller the value, the better. Indicates whether mobile device i participates in the current round of federated training (if it does, then...). (Otherwise, it is 0); M represents the number of mobile devices currently participating in the federated learning task, which is usually the number of online terminals in the system that have completed initial synchronization. These parameters, corresponding to the importance weights of training latency, communication latency, training energy consumption, and communication energy consumption respectively, can be set according to the application scenario. For example, in tasks with high real-time requirements, the importance weights can be adjusted accordingly. Set to a higher value to prioritize latency optimization; in energy-sensitive scenarios, this can improve latency. The proportion of [something] is used to control the overall power consumption level.

[0196] This step serves as the "global scheduling hub" of the entire system. By combining federated parameter aggregation with a prompt distillation mechanism, it enhances knowledge transfer and sharing between models while preserving the individuality of local policies. This addresses the issues of weak generalization ability of personalized models and poor robustness of global models in existing solutions, resulting in a more adaptable system. By setting a multi-objective reward function with multiple performance indicators, it unifies the modeling of factors such as training latency, energy consumption, and transmission costs. Furthermore, by guiding the model's learning direction through a prompt mechanism, it effectively reduces energy consumption and communication rounds during policy convergence while improving policy accuracy.

[0197] Step 6: Strategy feedback and iterative closed-loop restart

[0198] The federated coordination server will send the global policy parameter vector for the next round. The next round of global cue vectors and importance weight parameters The data is then distributed to each mobile device, which continues to complete the next round of local FDRL training, forming a policy optimization closed loop. This continues until the system iterates until the utility function converges or the expected performance of resource scheduling is achieved, such as meeting the requirements for task throughput, average latency, and power consumption constraints.

[0199] Experimental Example: In the constructed hybrid network simulation platform, 100 mobile devices were deployed, connecting to three types of networks: 5G, WiFi 6, and satellite. Compared with a single-network federated learning strategy, the system of this invention exhibits the following performance improvements, as detailed in the appendix. Figure 2 To be continued Figure 6 .

[0200] (1) By employing offline DTR, pruning strategies, and multi-granularity mechanisms, the accuracy of this invention reaches 97.23%. The offline DTR of this invention utilizes historical task trajectories to preload and adapt local initialization strategies for deep reinforcement learning models. To avoid the blindness of random initialization, this invention employs a multi-granularity resource mapping module to input fine-grained states such as cache utilization and single-task computation latency into the model, improving the comprehensiveness of feature capture. This invention achieves leading accuracy by precisely optimizing parameter convergence and feature quality. CN120066765A (A method and system for optimizing resource allocation in federated learning for edge intelligent networks) uses a Stackelberg game model, but its unoptimized parameter initialization and feature input make it prone to adaptation failure in complex scenarios; CN116669067A (A method and system for optimizing resource allocation in learning) uses a data similarity strategy, but its accuracy is limited in dynamic scenarios due to its lack of refined feature dimensions.

[0201] (2) The communication volume of this invention is 4.65MB. The core originates from the granularity adaptation capability of the multi-granularity service-heterogeneous link adaptation mechanism and the high-pruning model strategy. For example, high-definition video services are allocated to high-bandwidth satellite links, and low-power IoT tasks are set as high-pruning models, which saves costs, improves efficiency, enhances security, and can optimize system performance in resource-constrained scenarios (such as IoT or mobile networks). CN120066765A uses full parameter transmission, which does not adapt to the characteristics of multiple networks and has a high proportion of redundant data; CN116669067A only optimizes bandwidth allocation, lacks model compression, and the use of relay forwarding will increase the transmission burden.

[0202] (3) This invention uses asynchronous uploading, keeping the end-to-end latency of critical services around 0.93s. Through a heterogeneous link adaptation mechanism, tolerable tasks are transferred to the satellite link, freeing up 5G core resources for uRLLC tasks and avoiding resource contention. CN120066765A uses a centralized game theory decision-making approach, which requires waiting for feedback from all devices and is prone to delays due to device differences; CN116669067A uses a user relay mechanism, and multi-stage forwarding increases processing time.

[0203] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning (FDRL), characterized in that, Includes the following steps: Step 1: System initialization; Step 2: The mobile device perceives the local resource status, performs fine modeling of heterogeneous resources, and divides the resource status into three structural levels: device level, network level, and service level. Each structural level is further subdivided into coarse-grained and fine-grained levels, which are then mapped into structured state vectors by the multi-granularity resource mapping module. Step 3: Local FDRL training and global cue vector generation, the specific steps are as follows: (3.1) The federated deep reinforcement learning controller is based on the local experience pool collected in real time on each mobile device i. Global cue vectors uploaded by the local FDRL training module An offline trajectory-guided mechanism is used to perform local FDRL training initialized by a federated coordination server, and a personalized local initialization strategy is built. ; Personalized local initialization strategy The formula is as follows: ； in, For strategy fusion weights, For localized, personalized prior strategies; The initial global policy issued by the federal coordination server; (3.2) Construct personalized prompt vectors , Generated by fusing state trajectories, action trajectories, and resource metrics recorded by the mobile device during local FDRL training: ； in, This indicates the feature extraction function; This represents the historical state sequence of mobile device i. Represents a historical sequence of actions; Represents the historical reward sequence; Each mobile device records the training latency during local FDRL training. Training energy consumption pruning rate, transmission delay With transmission energy consumption And send it to the multi-heterogeneous hybrid network module; Step 4: Multiple heterogeneous hybrid network modules based on , , , and Calculate the optimal pruning ratio Based on the optimal pruning ratio Structural compression is performed on the locally trained FDRL model to generate a lightweight local policy parameter vector. By calculating the utility functions of different links Mobile devices can independently choose the appropriate upload path; Step 5: The federated coordination server periodically aggregates lightweight local policy parameter vectors. ; Integrate personalized prompt vectors from various mobile devices Complete the cue distillation and generate a unified global cue vector. ; Constructing includes , , and A multi-objective reward function; Step 6: The federated coordination server will send the global policy parameter vector for the next round. The next round of global cue vectors and importance weight parameters Distribute to various mobile devices.

2. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 1, characterized in that, Step 1 specifically involves the initialization phase, where the federated coordination server will uniformly set the deep reinforcement learning model parameter vector. The local FDRL training module is distributed to each mobile device i to initialize the local FDRL training model of the local FDRL training module, and is based on the initial cue vector of each mobile device i. The global hint vector is obtained by weighted aggregation according to certain weights. : ； in, is the aggregation weight of mobile device i; M represents the number of mobile devices currently participating in the federated learning task; This represents the initial cue vector for each mobile device i.

3. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 2, characterized in that, coarse-grained resource state vector This includes, but is not limited to: link bandwidth utilization, device CPU computing power load, and network congestion level; fine-grained resource state vectors. This includes, but is not limited to: cache utilization, task queue length, single-task computation latency, and remaining energy level.

4. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 3, characterized in that, The multi-granularity resource mapping module maps coarse-grained resource state vectors for each structural level of the mobile device. With fine-grained resource state vector After being normalized and encoded according to preset rules, the data are concatenated to generate a unified structured state vector. : ； Here, || represents the concatenation operation. Indicates the device-level structural hierarchy. Indicates network-level structural hierarchy. This indicates the business-level structural hierarchy.

5. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 4, characterized in that, Local experience pool The construction method is as follows: (1) The local FDRL training module of each mobile device i uploads the action vector. For the federated deep learning controller; Action vectors Represented as: ； in, For participation, The pruning rate for training a local FDRL model. Select the action vector for the current upload channel. This indicates the action performed by the mobile device i; (2) Each mobile device i records the local training trajectory. And cached in the built local experience pool. middle; in, A structured state vector representing the current state. A structured state vector representing the new state entered after an action is performed. Represents the action vector of mobile device i The structured state vector in the current state Instant rewards obtained; in, The calculation formula is: ； in, These are the importance weight parameters for training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

6. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 5, characterized in that, During local FDRL training, from the local experience pool When extracting samples, priority is assigned based on the temporal difference error of the samples: ； in, Let represent the temporal difference error of the i-th trajectory sample, and let represent the prediction bias of the current strategy for that trajectory sample. Discount factor; The new action vector represents the structured state vector of the mobile device in the new state. The next step is to select a new action vector to execute; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward; The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula: ； Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

7. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 6, characterized in that, Optimal pruning ratio The calculation formula is: ； in, Indicates the optimal pruning ratio; The pruning rate for training the local FDRL model; The formula is: ； in, For lightweight local policy parameter vectors; Prune represents pruning operations; Used to control the degree of sampling bias; For personalized local initialization strategies; The parameter vector for a uniformly defined deep reinforcement learning model.

8. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 7, characterized in that, Utility function The calculation formula is: ； in, This indicates that the utility function value is calculated under different links, and the link with the largest result value is taken as the optimal link; k represents the link type. This represents the utility weighting coefficient; Indicates training delay; This indicates the available bandwidth of link k; Indicates training energy consumption; This represents the packet loss rate of link k.

9. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 8, characterized in that, Step 5 consists of the following steps: (5.1) The federated coordination server is based on the uploaded lightweight local policy parameter vector. Aggregate the vectors to obtain the global policy parameter vector for round t+1. : ； in, This is the sum of the local sample sizes across all mobile devices. Let be the number of data samples used locally by the i-th mobile device. Let be the lightweight local policy parameter vector of mobile device i in round t; (5.2) Federated Coordination Server for Personalized Prompt Vectors Distillation and fusion are performed to obtain the global cue vector for the next round. : ； in, This indicates the feature extraction function. This represents the global hint vector for the next round; (5.3) Federal coordination server collects training latency Training energy consumption Transmission delay With transmission energy consumption Design a multi-objective reward function: ； Where U represents the multi-objective reward function, Indicates whether mobile device i participates in the current round of federated training; M represents the number of mobile devices currently participating in the federated learning task. These are the importance weight parameters corresponding to training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning as described in any one of claims 1 to 9.

Images