Multi-network multi-granularity resource optimization method based on federal deep reinforcement learning

Abstract

The invention discloses a multi-network multi-granularity resource optimization method based on federal deep reinforcement learning, and belongs to the technical field of edge computing and artificial intelligence, and the method comprises the steps: a mobile device senses a local resource state, and maps the local resource state into a structured state vector through a multi-granularity resource mapping module; local FDRL training and global prompt vector generation are carried out; performing pruning optimization on the local FDRL training model, and realizing asynchronous uploading by the multiple heterogeneous hybrid network modules based on a heterogeneous link adaptation mechanism; performing global model aggregation, prompting distillation and joint optimization; and strategy return and iterative closed-loop restarting are carried out. According to the invention, a three-level and multi-granularity state modeling method is introduced to enhance the expression ability; an off-line track guiding mechanism is introduced to realize rapid convergence in a strategy cold start stage; generating acceleration strategy convergence by adopting experience-first sampling + prompt; the uploading overhead is reduced by pruning a local FDRL training model, and the transmission efficiency and robustness are improved by adopting a heterogeneous link adaptation mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing and artificial intelligence, and in particular to a multi-network multi-granularity resource optimization method based on federated deep reinforcement learning. BACKGROUND

[0002] With the wide deployment of 5G communication, industrial Internet of Things, edge computing and intelligent mobile devices, modern computing environments are rapidly moving towards a development stage of diversification, heterogeneity and distribution. Mobile devices are of various types and have significant differences in capabilities, and network architecture presents a "multi-network" form in which multiple communication methods such as Wi-Fi, 5G and satellite links coexist. This network heterogeneity brings great scheduling complexity, especially in the case of rapid changes in resource state and unstable link quality, making it difficult for traditional static scheduling models to effectively adapt to system operation requirements.

[0003] At the same time, the requirements of system tasks for resources are becoming increasingly complex: for example, factory line fault warning identification and real-time analysis of environmental status require millisecond-level response to prevent system abnormalities from spreading; complex industrial drawing recognition and device status big data modeling require higher computing resources and communication throughput; and intelligent sensing tasks require low latency and high reliability, while some computationally intensive tasks require large bandwidth and high computing resources. The diversified and dynamic task characteristics exhibit significant "multi-granularity resource requirements", i.e., resources exhibit multi-level and multi-dimensional scheduling indicators from the network level (bandwidth, load), device level (computing power, power) to service level (task latency, reliability). However, existing resource scheduling mechanisms often only consider a single granularity or average load, lacking the ability to accurately model and control the resource diversity in complex systems.

[0004] To address the above challenges, reinforcement learning (RL) is widely used in resource allocation problems in edge computing due to its ability to optimize long-term benefits. However, traditional reinforcement learning methods rely on centralized control architecture, requiring a large amount of raw data to be transmitted back to the central server for unified training, which not only brings high communication burden, but also poses a serious risk of privacy leakage.

[0005] Federated learning (FL) is a distributed learning paradigm proposed in recent years, which distributes model training tasks to local devices and only uploads model parameters, achieving a collaborative optimization strategy of "data not leaving the local", and showing significant advantages in protecting data privacy and reducing communication costs. The combination of federated learning and reinforcement learning forms the federated deep reinforcement learning (FDRL) system, which is a key path to improving the intelligence level of edge systems.

[0006] However, the existing federal deep reinforcement learning system still has the following technical bottlenecks when facing the actual deployment scene of "multi-network heterogeneity + multi-granularity resource constraints":

[0007] 1) Poor mixed multi-network adaptability and low transmission efficiency;

[0008] Existing methods generally assume a single communication link (such as only 5G or Wi-Fi), making it difficult to flexibly select an upload path in a multi-network heterogeneous access environment such as 5G, Wi-Fi, and satellite, resulting in increased communication delay and unstable system response in network congestion or interruption scenarios.

[0009] 2) Single resource modeling granularity and weak expression ability;

[0010] Most existing methods only focus on macro resource indicators (such as total computing power, bandwidth, etc.), lack modeling support for micro fine-grained resources such as device cache, queue length, and battery remaining capacity, and are difficult to accurately match diverse task requirements, resulting in low resource utilization.

[0011] 3) Difficult policy cold start and slow model convergence;

[0012] Most existing methods lack effective prior knowledge guidance in the policy initialization stage and rely on random initialization, resulting in unstable initial training and slow convergence, especially in device heterogeneous or data distribution uneven scenarios.

[0013] 4) Lack of personalized decision-making mechanism, rough scheduling results;

[0014] Some existing methods use a unified model and policy to schedule all mobile devices, without considering device differences and task demand diversity, which can lead to poor generalization of scheduling results and mismatched resource allocation.

[0015] 5) High communication and computing overhead, heavy system burden;

[0016] In communication resource-constrained or energy-sensitive scenarios, existing methods frequently perform full-model upload and synchronization, lack effective model compression and asynchronous upload mechanisms, and are prone to communication bottlenecks and energy waste.

[0017] Based on this, the present application designs a multi-network multi-granularity resource optimization method based on federal deep reinforcement learning to solve the above problems. SUMMARY

[0018] To overcome the above-mentioned shortcomings of the prior art, the present application provides a multi-network multi-granularity resource optimization method based on federal deep reinforcement learning.

[0019] To achieve the above purpose, the present application is implemented by the following technical solutions:

[0020] The multi-network multi-granularity resource optimization method based on federated deep reinforcement learning comprises the following steps:

[0021] Step 1: system initialization;

[0022] Step 2: the mobile device senses the local resource state, finely models the heterogeneous resources, divides the resource state into three structure levels of device level, network level and service level, and further divides each structure level into coarse granularity and fine granularity, and maps the structure state vector through the multi-granularity resource mapping module;

[0023] Step 3: local FDRL training and generation of global prompt vector, the specific steps are:

[0024] (3.1) The federated deep reinforcement learning controller is based on the local experience pool collected by each mobile device i in real time , the global prompt vector uploaded by the local FDRL training module , adopts an offline trajectory guiding mechanism to perform the local FDRL training initialized by the federated coordination server, and builds a personalized local initialization strategy ;

[0025] The personalized local initialization strategy The formula is as follows:

[0026]

[0027] Wherein, is the policy fusion weight, is the local personalized prior strategy; is the initial global strategy issued by the federated coordination server;

[0028] (3.2) Constructing a personalized prompt vector , is generated by fusing the state trajectory, action trajectory and resource index recorded by the mobile device in the local FDRL training process:

[0029]

[0030] Wherein, represents a prompt feature extraction function; represents the historical state sequence of the mobile device i, represents the historical action sequence; represents the historical reward sequence;

[0031] During the local FDRL training process, each mobile device records the training delay , the training energy consumption , the pruning rate, and the transmission delay with transmission energy consumption , and sends to the multi-heterogeneous hybrid network module;

[0032] Step 4: The multi-heterogeneous hybrid network module calculates the optimal pruning ratio , , , and , and performs structural compression on the local FDRL training model according to the optimal pruning ratio , to generate a lightweight local policy parameter vector ; by calculating the utility function of different links , the mobile device autonomously selects a suitable upload path;

[0033] Step 5: The federal coordination server periodically aggregates the lightweight local policy parameter vector ; fuses the individualized hint vector of each mobile device , completes hint distillation, and generates a unified global hint vector ; constructs a multi-objective reward function containing , , and ;

[0034] Step 6: The federal coordination server distributes the next round of global policy parameter vector , the next round of global hint vector , and the importance weight parameter to each mobile device.

[0035] Further, step 1 is specifically: in the initialization stage, the federal coordination server distributes the uniformly set deep reinforcement learning model parameter vector to the local local FDRL training module in each mobile device i, to initialize the local FDRL training model of the local local FDRL training module, and the initial hint vector of each mobile device i is weighted aggregated according to a certain weight to obtain the global hint vector :

[0036]

[0037] wherein, is the aggregation weight of the mobile device i; M represents the number of mobile devices currently participating in the federal learning task; represents the initial hint vector of each mobile device i.

[0038] Further, the coarse-grained resource state vector ​including but not limited to: link bandwidth occupancy, device CPU computing power load and network congestion degree; fine-grained resource state vector including but not limited to: cache utilization, task queue length, single task computing delay and remaining energy level.

[0039] Further, the multi-granularity resource mapping module splices the coarse-grained resource state vector of each type of structural level of the mobile device and the fine-grained resource state vector after normalization encoding according to a preset rule to generate a unified structured state vector :

[0040]

[0041] Wherein, || represents the splicing operation.

[0042] Further, the construction method of the local experience pool is as follows:

[0043] (1) The local FDRL training module of each mobile device i uploads the action vector to the federal deep learning controller; the action vector represents :

[0044]

[0045] Wherein, is a participation flag, is a pruning rate of the local FDRL training model, is a current upload channel selection, and the action vector represents the action performed by the mobile device i;

[0046] (2) Each mobile device i records the local training trajectory , and caches it in the constructed local experience pool ;

[0047] Wherein, represents the structured state vector of the current state, represents the structured state vector of the new state entered after the action is performed, represents the action vector of the mobile device i obtained under the structured state vector of the current state ;

[0048] Wherein, The calculation formula of is as follows:

[0049]

[0050] ​​in, These are the importance weight parameters for training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

[0051] Furthermore, during local FDRL training, the local experience pool is used... When extracting samples, priority is assigned based on the temporal difference error of the samples:

[0052]

[0053] in, Let represent the temporal difference error of the i-th trajectory sample, and let represent the prediction bias of the current strategy for that trajectory sample. Discount factor; The new action vector represents the structured state vector of the mobile device in the new state. The next step is to select a new action vector to execute; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward;

[0054] The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula:

[0055]

[0056] Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

[0057] Furthermore, the optimal pruning ratio The calculation formula is:

[0058]

[0059] in, Indicates the optimal pruning ratio; The pruning rate for training the local FDRL model;

[0060] The formula is:

[0061]

[0062] wherein, is the light-weighted local policy parameter vector; Prune represents the pruning operation; is used to control the sampling bias degree; is the personalized local initialization policy; is the unified set of deep reinforcement learning model parameter vector.

[0063] Further, the utility function is calculated by the following formula:

[0064]

[0065] wherein, represents the calculation of the utility function value under different links, and the link with the maximum result value is taken as the optimal link; k represents the link type; represents the utility weight coefficient; represents the communication delay of link k; represents the available bandwidth of link k; represents the energy consumption of transmitting data on link k; represents the packet loss rate of link k.

[0066] Further, the specific steps of step 5 are as follows:

[0067] (5.1) The federal coordination server aggregates the uploaded light-weighted local policy parameter vector to obtain the global policy parameter vector of the t+1th round:

[0068]

[0069] wherein, is the sum of the local sample sizes of all mobile devices; is the number of data samples used by the i th mobile device, is the light-weighted local policy parameter vector of the i th mobile device in the t th round;

[0070] (5.2) The federal coordination server distills and fuses the personalized prompt vector to obtain the global prompt vector of the next round:

[0071]

[0072] wherein, represents the prompt feature extraction function, represents the global prompt vector of the next round;

[0073] (5.3) Federal coordination server collects training latency Training energy consumption Transmission delay With transmission energy consumption Design a multi-objective reward function:

[0074]

[0075] Where U represents the multi-objective reward function, Indicates whether mobile device i participates in the current round of federated training; M represents the number of mobile devices currently participating in the federated learning task. These are the importance weight parameters corresponding to training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

[0076] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning.

[0077] Compared with the prior art, the beneficial effects of this invention are as follows:

[0078] This invention introduces a "three-level (network level, device level, service level) + multi-granularity (coarse-grained and fine-grained)" state modeling method to achieve multi-dimensional dynamic abstraction of resource states, providing a more accurate and structured perception basis for policy input, realizing the unity of the breadth and depth of resource perception, improving the generalization and robustness of policy optimization in actual deployment scenarios, and helping to improve resource utilization.

[0079] This invention introduces an offline trajectory guidance mechanism (DTR) during local FDRL training. By loading historical task trajectories on the device side, a local personalized prior policy is generated and initialized with the global policy. This enables rapid convergence and stable progress during the policy cold start phase, significantly improving the system's generalization and adaptability in multi-task and multi-environment scenarios.

[0080] By prioritizing learning from samples that are difficult to predict or have large policy biases, the local FDRL training module can adjust the training scope more quickly (e.g., due to the number of mobile devices from 0 to j). (This avoids situations where the cumulative reward is significantly higher than the average, leading to a large deviation from the overall target), thus preventing the system from getting trapped in local optima. Furthermore, this method exhibits stronger adaptability in environments with multiple network access points and significantly different tasks on heterogeneous devices, contributing to improved training efficiency and generalization ability of the entire system.

[0081] The DTR module not only improves sample utilization and reduces cold start time, but also significantly enhances the strategy convergence instability problem caused by cross-device differences and asynchronous communication in federated training, and is especially suitable for edge intelligent systems with severe delay fluctuations under multi-network access and inconsistent training task complexity.

[0082] The present application adopts an "empirical priority sampling + prompt generation" parallel mechanism, which on the one hand accelerates strategy convergence by enhancing training efficiency and stability, and on the other hand provides rich knowledge expression for personalized distillation through prompt construction.

[0083] The present application can autonomously complete model compression and upload channel selection, and realize flexible resource backhaul and communication optimization under a mixed multi-network architecture. Pruning the local FDRL training model can reduce upload overhead, and the heterogeneous link adaptation mechanism can improve transmission efficiency and robustness, fully releasing the participation ability of resource-constrained mobile devices and enhancing the adaptability and scalability of the entire system under bandwidth-intensive or multiple network heterogeneous conditions.

[0084] The present application combines federated parameter aggregation and prompt distillation mechanism, strengthens the knowledge transfer and sharing between models on the basis of preserving local strategy personality, solves the problem of weak generalization ability of personalized models and poor robustness of global models in existing solutions, and the system has stronger adaptability. By setting a multi-objective reward function of multiple performance indicators, factors such as training delay, energy consumption and transmission cost are modeled, and the model learning direction is guided through the prompt mechanism, which effectively reduces the energy consumption and communication rounds in the strategy convergence process while improving the strategy accuracy.

[0085] The comprehensive experimental results show that the accuracy of the present application is as high as 97.23%, the communication amount is 4.65MB, and the end-to-end delay of key services is maintained at about 0.93s, which has clear practical application prospect and industrial value. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0087] Figure 1 The flowchart of the present application of multi-network multi-granularity resource optimization method based on federated deep reinforcement learning.

[0088] Figure 2 The accuracy comparison results of the present application and other federated learning strategies.

[0089] Figure 3 Comparison results of the communication volume of the present application and other federated learning strategies.

[0090] Figure 4 Comparison results of the latency of the present application and other federated learning strategies.

[0091] Figure 5 Variation of the accuracy of the present application and other federated learning strategies with the training round.

[0092] Figure 6 Variation of the communication volume of the present application and other federated learning strategies with the training round. DETAILED DESCRIPTION

[0093] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0094] Embodiment one: please refer to the accompanying drawings of the specification Figure 1 , the multi-network multi-granularity resource optimization method based on federated deep reinforcement learning, the specific steps are as follows:

[0095] Step 1: system initialization;

[0096] In the initialization stage, the federated coordination server will uniformly set the deep reinforcement learning model parameter vector to each mobile device i in the local local FDRL training module to initialize the local FDRL training model of the local local FDRL training module, and the initial prompt vector of each mobile device i is aggregated by a certain weight to obtain the global prompt vector :

[0097]

[0098] Among them, is the aggregation weight of the mobile device i (such as the mobile device data volume, the training quality or the mobile device contribution degree); M represents the number of mobile devices currently participating in the federated learning task; represents the initial prompt vector of each mobile device i.

[0099] The local local FDRL training module will aggregate the global prompt vector obtained by weighting and the local experience pool of each mobile device i Further passed to the federated deep reinforcement learning controller.

[0100] Step 2: The mobile device perceives the local resource state and maps it into a structured state vector through a multi-granularity resource mapping module; specifically including the following steps:

[0101] (2.1) Resource collection and classification

[0102] Before each mobile device i performs a task, it needs to perceive the current resource state and pre-model the heterogeneous resources, dividing the resource state into three types of structural levels:

[0103] Device level (pruning rate, cache, power, etc.);

[0104] Network level (bandwidth, load, network type, link type, etc.);

[0105] Service level (task type, delay / reliability, etc. QoS requirements);

[0106] Further, each type of structural level is further divided into coarse-grained (stable indicators) and fine-grained (dynamic indicators), respectively constituting a coarse-grained resource state vector and a fine-grained resource state vector:

[0107] Coarse-grained resource state vector , including but not limited to: link bandwidth occupancy, device CPU computing power load, and network congestion level;

[0108] Fine-grained resource state vector , including but not limited to: cache utilization, task queue length, single task computing delay, and remaining energy level;

[0109] (2.2) Structured mapping

[0110] The multi-granularity resource mapping module normalizes and encodes each type of structural level of the mobile device and according to the preset rules, and then splices them to generate a unified structured state vector :

[0111]

[0112] Where || represents the splicing operation, and the structured state vector represents the input of the structured state of the environment state of the mobile device i at the current time (such as local cache state, channel condition, remaining power, etc.).

[0113] (2.3) Structured state vector Embedding input into the local FDRL training model of the federated deep reinforcement learning controller as a training parameter;

[0114] The application realizes the unification of the breadth and depth of resource perception by introducing a "three-level (network level, device level, and service level) + multi-granularity (coarse-granularity and fine-granularity)" state modeling method, providing a more accurate and structured perception basis for policy input, improving the generalization and robustness of policy optimization in actual deployment scenarios, and facilitating the improvement of resource utilization.

[0115] Step 3: Local FDRL training and generation of global hint vector

[0116] (3.1) In order to improve the stability of federated deep reinforcement learning and improve the utilization rate of policy samples, the federated deep reinforcement learning controller introduces an offline trajectory guiding mechanism (DTR) in the local FDRL training process

[0117] Under the federated learning framework, a local FDRL training module (including a local FDRL training model) is deployed in each network domain.

[0118] Specifically, the federated deep reinforcement learning controller uses the local experience pool collected by each mobile device i in real time , the global hint vector uploaded by the local FDRL training module , and the offline trajectory guiding mechanism (DTR) to perform local FDRL training initialized by the federated coordination server, and builds a personalized local initialization policy

[0119] Personalized local initialization policy The formula is as follows:

[0120]

[0121] Wherein, is the policy fusion weight, is the local personalized prior policy; is the initial global policy issued by the federated coordination server.

[0122] In the traditional federated deep reinforcement learning, if the mobile device directly starts learning from the global policy at system startup, a large number of interaction rounds are often required for convergence, and the performance in the early stage is poor. Due to the different hardware capabilities, network conditions, and task preferences of each mobile device i, if the unified global policy is completely relied on, it will lead to poor performance in the local environment.

[0123] In the system starting stage, in order to improve the model cold start efficiency and enhance the individualized adaptability, the federal deep reinforcement learning controller firstly constructs a local initialization strategy model. The local experience pool Rich offline trajectory samples are provided for the DTR, and the DTR learns and generates a guide strategy based on offline trajectory data; the global prompt vector Provides additional global information and guidance signals for the DTR, and the DTR can better align the trajectory data of different local models using the global prompt vector, improving the accuracy and consistency of trajectory guidance. Considering the interaction experience accumulated by the mobile device i in the historical task, each mobile device i loads the offline trajectory guidance mechanism (DTR) from the local and trains a local individualized prior strategy Subsequently, the federal deep reinforcement learning controller fuses the local individualized prior strategy With the global strategy issued by the federal coordination server To generate a local initialization strategy For subsequent training. The local initialization strategy Inherits the universality of the global and retains the adaptation characteristics of the device itself, so that the model has certain task adaptation ability at the beginning, which can greatly reduce the performance trough in the cold start stage and improve the completion efficiency of the local task.

[0124] The application introduces an offline trajectory guidance mechanism (DTR) in the local FDRL training process, loads the historical task trajectory on the device side to generate a local individualized prior strategy, and initializes the fusion with the global strategy, realizes the rapid convergence and stable promotion of the strategy cold start stage, and significantly improves the generalization ability and adaptability of the system in the multi-task and multi-environment scene.

[0125] The construction method of the local experience pool Is as follows:

[0126] The action vector Before the initial round of deep reinforcement learning model parameter vector And the local FDRL training model of the new round of federal deep reinforcement learning controller is subjected to a structured compression operation, that is, pruning, and the pruning rate The initial setting is 0.2.

[0127] The action vector Of each mobile device i local FDRL training module is given to the federal deep learning controller; the action vector Is expressed as:

[0128]

[0129] Among them, participation flag (0 or 1), pruning rate for local FDRL training model, current upload channel selection (5G, Wi-Fi, satellite), action vector denotes the action (such as pruning rate selection, participation decision, upload channel selection, etc.) performed by mobile device i.

[0130] Each mobile device i records a local training trajectory and caches in the constructed local experience pool ;

[0131] wherein, denotes the structured state vector of the current state, denotes the structured state vector of the new state entered after the action is performed, i.e., the structured state vector of the next time state; denotes the action vector of mobile device i obtained under the structured state vector of the current state immediate reward (used to update the policy when training) for feedback to itself of the local experience pool for sample quality optimization;

[0132] wherein, The calculation formula is:

[0133]

[0134] wherein, respectively, are importance weight parameters of training delay, communication delay, training energy consumption, and communication energy consumption, which are usually negative numbers (representing cost) calculated according to delay, energy consumption, etc.

[0135] (3.2) In the local FDRL training process, a weighted priority replay mechanism is introduced

[0136] When samples are extracted from the local experience pool , instead of equal-probability sampling, a priority is assigned according to the time series difference error (TD-error) of the sample:

[0137]

[0138] wherein, denotes the time series difference error of the i-th trajectory sample, and denotes the prediction deviation of the current policy for the trajectory sample; is a discount factor measuring the importance of future rewards; denotes a new action vector, which is the action performed by the mobile device in the structured state vector of the new state You can then choose to execute new action vectors, including actions such as pruning; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward.

[0139] The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula:

[0140]

[0141] Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

[0142] By prioritizing learning from samples that are difficult to predict or have large policy biases, the local FDRL training module can adjust the training scope more quickly (e.g., due to the number of mobile devices from 0 to j). (This avoids situations where the cumulative reward is significantly higher than the average, leading to a large deviation from the overall target), thus preventing the system from getting trapped in local optima. Furthermore, this method exhibits stronger adaptability in environments with multiple network access points and significantly different tasks on heterogeneous devices, contributing to improved training efficiency and generalization ability of the entire system.

[0143] This DTR module not only improves sample utilization and reduces cold start time, but also significantly enhances the policy convergence instability caused by cross-device differences and asynchronous communication in federated training. It is especially suitable for edge intelligent systems with severe latency fluctuations and inconsistent training task complexity under multiple network access.

[0144] (3.3) Construct personalized prompt vectors

[0145] Personalized hint vectors Capture the preferences and strategic characteristics of mobile devices. It is generated by fusing state trajectories, action trajectories, and resource metrics (such as latency, energy consumption, and pruning rate) recorded by the mobile device during local FDRL training.

[0146]

[0147] in, denotes the prompt feature extraction function; denotes the history state sequence of mobile device i (i.e., several ), denotes the resource state change trend; denotes the history action sequence, denotes the device preference behavior (e.g., bias to high pruning / bias to Wi-Fi upload, etc.); denotes the history reward sequence, denotes the policy effect, such as delay / energy consumption performance;

[0148] Each mobile device records the performance indicators of training delay , training energy consumption , pruning rate, transmission delay , and transmission energy consumption , etc. during the local FDRL training process;

[0149] Training delay , the calculation formula is as follows:

[0150]

[0151] wherein, is the participation flag (0 or 1), C represents the amount of computing resources required per bit, denotes the number of CPU cycles required for the data set , and is the device computing capability.

[0152] Training energy consumption , the calculation formula is as follows:

[0153]

[0154] wherein, denotes the CPU unit power consumption constant.

[0155] Transmission delay , the calculation formula is as follows:

[0156]

[0157] wherein, is the size of the local FDRL training model, is the transmission rate of the mobile device i, is the pruning rate of the local FDRL training model, satisfying , denotes the upper limit value of the pruning rate.

[0158] Transmission energy consumption , the calculation formula is as follows:

[0159]

[0160] in, This refers to the transmit power of the mobile device.

[0161] After several rounds of local FDRL training, , , , , and They are then sent together to the multi-heterogeneous hybrid network module.

[0162] This invention employs a parallel mechanism of "experience-first sampling + prompt generation". On the one hand, it accelerates strategy convergence by enhancing training efficiency and stability, and on the other hand, it provides rich knowledge representation for personalized distillation through prompt construction.

[0163] Step 4: Prune and optimize the local FDRL training model; implement asynchronous uploading of multiple heterogeneous hybrid network modules based on heterogeneous link adaptation mechanism.

[0164] Multi-heterogeneous hybrid network module: refers to a hybrid heterogeneous network environment with multiple communication protocols, multi-layer architecture, and multiple service QoS requirements, including but not limited to the following network domains: 5G communication network, Wi-Fi local area network, Bluetooth low power network, etc. Each network domain is configured with an independent resource agent unit to monitor local resource status and network performance indicators.

[0165] Pruning primarily addresses the issue of high computational and communication overhead in locally trained FDRL models within federated learning scenarios. It is a method to optimize the overall federated learning framework. Pruning can compress models while maintaining their accuracy, significantly reducing the overall model size. The appropriate pruning ratio can be calculated based on the compression requirements of different models, thereby reducing communication overhead and the communication load between clients and servers in federated learning. It is suitable for low-speed networks or low-power devices.

[0166] The multiple heterogeneous hybrid network modules receive the pruning rate from the federated deep reinforcement learning controller. Training delay Training energy consumption Transmission delay and transmission energy consumption Calculate the optimal pruning ratio :

[0167]

[0168] in, Indicates the optimal pruning ratio; Pruning rate for locally trained FDRL models, in real-time priority case When bandwidth efficiency is prioritized, When precision and stability are prioritized, .

[0169] According to the optimal pruning ratio Structural compression is performed on the local FDRL training model (such as pruning part of the low importance neurons or channels), to generate a lightweight local policy parameter vector ; is passed to the federal coordination server for global aggregation, and then leads the next round of iteration.

[0170] The formula is:

[0171]

[0172] wherein, is a lightweight local policy parameter vector; Prune represents a pruning operation; is used to control the sampling bias degree; is a personalized local initialization strategy; is a uniformly set deep reinforcement learning model parameter vector.

[0173] Since in the case of multiple heterogeneous hybrid networks, each mobile device i may be in different network environments (such as some supporting 5G high-speed access, and some only being able to connect to Wi-Fi or low-orbit satellite), a heterogeneous link adaptation mechanism is designed, which calculates the utility function of different links

[0174] Utility function The calculation formula is:

[0175]

[0176] wherein, represents calculating the utility function value under different links, and taking the link with the maximum result value as the optimal link; k represents the link type; represents the utility weight coefficient, which is used to adjust the importance of delay, bandwidth, energy consumption and packet loss rate in decision-making; represents the communication delay of link k; represents the available bandwidth of link k; represents the energy consumption of transmitting data on link k; represents the packet loss rate of link k (between 0 and 1), the smaller the value is, the better;

[0177] Asynchronous upload example:

[0178] 1) Delay <10ms, bandwidth >100Mbps to achieve high real-time, low delay task priority through 5G link upload, to ensure fast synchronization of control instructions and key model parameters;

[0179] 2) Delay 10-50ms, bandwidth 30-100Mbps for medium bandwidth tolerant tasks, transmitted through Wi-Fi network, balancing communication cost and timeliness;

[0180] 3) Delay >100ms, bandwidth <20Mbps for higher delay tolerant tasks, transmitted through satellite link asynchronously, relieving the competition pressure in congested network.

[0181] Subsequently, the mobile device selects the optimal link (such as 5G, Wi-Fi or satellite network) according to the current network condition to upload the lightweight local policy parameter vector , personalized prompt vector to the federal coordination server, realizing the optimal utilization of heterogeneous network links.

[0182] The present application can autonomously complete model compression and upload channel selection, realizing flexible resource backhaul and communication optimization under mixed multi-network architecture. Pruning the local FDRL training model can reduce upload overhead, and the heterogeneous link adaptation mechanism can improve transmission efficiency and robustness, fully releasing the participation ability of resource-constrained mobile devices and enhancing the adaptability and scalability of the entire system under bandwidth-intensive or multiple network heterogeneous conditions.

[0183] Step 5: Global model aggregation, prompt distillation and joint optimization

[0184] The federal coordination server periodically aggregates the lightweight local policy parameter vectors uploaded by multiple heterogeneous mixed network modules in each network domain ; fuses the personalized prompt vectors of each mobile device , completes prompt distillation, and generates a unified global prompt vector ; In addition, the federal coordination server also constructs a multi-objective reward function containing multiple performance indicators (training delay , training energy consumption , transmission delay and transmission energy consumption .

[0185] The specific steps are as follows:

[0186] (5.1) The federal coordination server aggregates the uploaded lightweight local policy parameter vectors to obtain the global policy parameter vector of the t+1th round:

[0187]

[0188] Wherein, is the sum of the local sample sizes of all mobile devices, i.e. the global total sample size; the number of data samples used locally by the i-th mobile device, the light-weight local policy parameter vector of the i-th mobile device in the t-th round, which is trained by the mobile device locally based on (or the aggregation parameter of the previous round).

[0189] (5.2) The federal coordination server acts as a leader to perform global policy aggregation and build a system-level resource scheduling policy model, that is, to perform distillation fusion on the personalized prompt vector to obtain the global prompt vector of the next round:

[0190]

[0191] wherein, represents a prompt feature extraction function, represents the global prompt vector of the next round, which is used for policy distribution in the next round.

[0192] (5.3) The federal coordination server collects model energy constraint parameters, communication delay parameters, and performance index data (such as training delay , training energy consumption , transmission delay , and transmission energy consumption ) uploaded by all mobile devices, and designs a multi-objective reward function. The multi-objective reward function sets or adjusts the importance weight parameters of training delay, communication delay, training energy consumption, and communication energy consumption according to the application scenario. The adjusted is fed back to the calculation of the immediate reward in step 3, so as to achieve the purpose of optimizing the quality of local experience pool samples by improving the immediate reward ; the adjusted is also fed back to the calculation of the optimal pruning ratio in step 4, so as to meet the different needs of mobile devices for real-time, bandwidth, and accuracy.

[0193] The formula of the multi-objective reward function is:

[0194]

[0195] wherein, U represents the multi-objective reward function, which is essentially a comprehensive cost function (Cost Function), and the smaller the value is, the better, represents whether the i-th mobile device participates in the current round of federated training (if it participates, then , otherwise 0); M represents the number of mobile devices participating in the current federated learning task, which is usually the number of terminals online and completing initialization synchronization in the system. The importance weight parameters corresponding to the training delay, the communication delay, the training energy consumption and the communication energy consumption respectively can be set according to the application scenario. For example, in a task with high real-time requirement, the value of the importance weight parameter can be set to be higher to prioritize the optimization of the time delay; in an energy consumption sensitive scenario, the proportion of the importance weight parameter can be increased to control the overall power consumption level.

[0196] This step is the "global scheduling center" of the whole system. By combining the federated parameter aggregation and the prompt distillation mechanism, the knowledge migration and sharing between models are strengthened on the basis of preserving the individuality of the local strategy, and the problems of weak generalization ability of the individualized model and poor robustness of the global model in the existing solutions are solved, and the system adaptability is stronger. By setting a multi-objective reward function of multiple performance indicators, the training delay, the energy consumption, the transmission cost and other factors are uniformly modeled, and the model learning direction is guided through the prompt mechanism, which effectively reduces the energy consumption and the communication rounds in the strategy convergence process while improving the strategy accuracy.

[0197] Step 6: Strategy feedback and iterative closed loop restart

[0198] The federated coordination server sends the global strategy parameter vector of the next round, the global prompt vector of the next round and the importance weight parameter to each mobile device, and the mobile device continues to complete the local FDRL training of the next round to form a strategy optimization closed loop. Until the system iteration converges to the utility function or reaches the expected performance of resource scheduling, such as the task throughput rate, the average delay and the power consumption constraint.

[0199] Experimental example: in the constructed hybrid network simulation platform, 100 mobile devices are deployed, and three types of networks of 5G / WiFi 6 / satellite are accessed. Compared with the single network federated learning strategy, the system of the present application shows the following performance improvement, and the specific results are shown in the attached Figure 2 to Figure 6 .

[0200] (1) The offline DTR, the pruning strategy and the multi-granularity mechanism are adopted, and the accuracy of the present application is as high as 97.23%. The offline DTR of the present application uses the historical task trajectory to pre-load the adaptive local initialization strategy for the deep reinforcement learning model ​​, avoid the blindness of random initialization; the application adopts a multi-granularity resource mapping module to input buffer utilization, single task calculation delay and other fine-grained states to the model, and improves the comprehensiveness of feature capture. The application realizes leading precision through precise optimization parameter convergence and feature quality. CN120066765A (a federated learning resource allocation optimization method and system for edge intelligent network) adopts a Stackelberg game model, which does not optimize parameter initialization and feature input, and is easy to adapt to failure in complex scenarios; CN116669067A (a learning resource allocation optimization method and system) adopts a data similarity strategy, and the accuracy is limited in a dynamic scene because the feature dimension is not refined.

[0201] (2) The communication volume of the application is 4.65MB. The core is the granularity adaptation capability of the multi-granularity service-heterogeneous link adaptation mechanism and the model high pruning strategy cooperation, such as allocating high-definition video service to high-bandwidth satellite link, and setting low-power IoT task as high pruning model, saving cost, improving efficiency, enhancing security, and optimizing system performance in resource-limited scenarios (such as Internet of Things or mobile network). CN120066765A adopts full-amount parameter transmission, which does not adapt to multi-network characteristics, and the proportion of redundant data is high; CN116669067A only optimizes bandwidth allocation, lacks model compression, and using relay forwarding will increase transmission burden.

[0202] (3) The application adopts asynchronous uploading, so that the end-to-end delay of key services is maintained at about 0.93s, tolerable tasks are transferred to satellite link through the heterogeneous link adaptation mechanism, 5G core resources are released for uRLLC task use, and resource competition is avoided. CN120066765A adopts centralized game decision, which needs to wait for feedback from all devices, and is easy to delay due to device differences; CN116669067A adopts a user relay mechanism, and multi-stage forwarding increases processing time consumption.

[0203] The above embodiments are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning, characterized in that, Includes the following steps: Step 1: System initialization; Step 2: The mobile device perceives the local resource status, performs fine modeling of heterogeneous resources, and divides the resource status into three structural levels: device level, network level, and service level. Each structural level is further subdivided into coarse-grained and fine-grained levels, which are then mapped into structured state vectors by the multi-granularity resource mapping module. Step 3: Local FDRL training and global cue vector generation, the specific steps are as follows: (3.1) The federated deep reinforcement learning controller is based on the local experience pool collected in real time on each mobile device i. Global cue vectors uploaded by the local FDRL training module An offline trajectory-guided mechanism is used to perform local FDRL training initialized by a federated coordination server, and a personalized local initialization strategy is built. ; Personalized local initialization strategy The formula is as follows: in, For strategy fusion weights, For localized, personalized prior strategies; The initial global policy issued by the federal coordination server; (3.2) Construct personalized prompt vectors , Generated by fusing state trajectories, action trajectories, and resource metrics recorded by the mobile device during local FDRL training: in, This indicates the feature extraction function; This represents the historical state sequence of mobile device i. Represents a historical sequence of actions; Represents the historical reward sequence; Each mobile device records the training latency during local FDRL training. Training energy consumption pruning rate, transmission delay With transmission energy consumption And send it to the multi-heterogeneous hybrid network module; Step 4: Multiple heterogeneous hybrid network modules based on , , , and Calculate the optimal pruning ratio Based on the optimal pruning ratio Structural compression is performed on the locally trained FDRL model to generate a lightweight local policy parameter vector. By calculating the utility functions of different links Mobile devices can independently choose the appropriate upload path; Step 5: The federated coordination server periodically aggregates lightweight local policy parameter vectors. ; Integrate personalized prompt vectors from various mobile devices Complete the cue distillation and generate a unified global cue vector. ; Constructing includes , , and A multi-objective reward function; Step 6: The federated coordination server will send the global policy parameter vector for the next round. The next round of global cue vectors and importance weight parameters Distribute to various mobile devices.

2. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 1, characterized in that, Step 1 specifically involves the initialization phase, where the federated coordination server will uniformly set the deep reinforcement learning model parameter vector. The local FDRL training module is distributed to each mobile device i to initialize the local FDRL training model of the local FDRL training module, and is based on the initial cue vector of each mobile device i. The global hint vector is obtained by weighted aggregation according to certain weights. : in, is the aggregation weight of mobile device i; M represents the number of mobile devices currently participating in the federated learning task; This represents the initial cue vector for each mobile device i.

3. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 2, characterized in that, coarse-grained resource state vector This includes, but is not limited to: link bandwidth utilization, device CPU computing power load, and network congestion level; fine-grained resource state vectors. This includes, but is not limited to: cache utilization, task queue length, single-task computation latency, and remaining energy level.

4. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 3, characterized in that, The multi-granularity resource mapping module maps coarse-grained resource state vectors for each structural level of the mobile device. With fine-grained resource state vector After being normalized and encoded according to preset rules, the data are concatenated to generate a unified structured state vector. : In this context, || represents the concatenation operation.

5. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 4, characterized in that, Local experience pool The construction method is as follows: (1) The local FDRL training module of each mobile device i uploads the action vector. For the federated deep learning controller; Action vectors Represented as: in, For participation, The pruning rate for training a local FDRL model. Select the action vector for the current upload channel. This indicates the action performed by the mobile device i; (2) Each mobile device i records the local training trajectory. And cached in the built local experience pool. middle; in, A structured state vector representing the current state. A structured state vector representing the new state entered after an action is performed. Represents the action vector of mobile device i The structured state vector in the current state Instant rewards obtained; in, The calculation formula is: in, These are the importance weight parameters for training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

6. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 5, characterized in that, During local FDRL training, from the local experience pool When extracting samples, priority is assigned based on the temporal difference error of the samples: in, Let represent the temporal difference error of the i-th trajectory sample, and let represent the prediction bias of the current strategy for that trajectory sample. Discount factor; The new action vector represents the structured state vector of the mobile device in the new state. The next step is to select a new action vector to execute; The structured state vector representing the new state Next, adopt a new action vector. Then, the expected cumulative rewards that can be obtained in the future; The structured state vector representing the current state of mobile device i. Next execution action vector Expected performance afterward; The larger the value, the greater the prediction bias of the current weighted priority replay mechanism on that state, and the more it should be learned. The sampling priority is calculated by the following formula: Where F(i) represents the local experience pool The priority of selecting sample i for learning; Positive numbers are used to avoid This results in a sampling probability of 0. Used to control the degree of sampling bias.

7. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 6, characterized in that, Optimal pruning ratio The calculation formula is: in, Indicates the optimal pruning ratio; The pruning rate for training the local FDRL model; The formula is: in, For lightweight local policy parameter vectors; Prune represents pruning operations; Used to control the degree of sampling bias; For personalized local initialization strategies; The parameter vector for a uniformly defined deep reinforcement learning model.

8. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 7, characterized in that, Utility function The calculation formula is: in, This indicates that the utility function value is calculated under different links, and the link with the largest result value is taken as the optimal link; k represents the link type. This represents the utility weighting coefficient; This represents the communication delay of link k; This indicates the available bandwidth of link k; This represents the energy consumption for transmitting data on link k; This represents the packet loss rate of link k.

9. The multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning according to claim 8, characterized in that, Step 5 consists of the following steps: (5.1) The federated coordination server is based on the uploaded lightweight local policy parameter vector. Aggregate the vectors to obtain the global policy parameter vector for round t+1. : in, This is the sum of the local sample sizes across all mobile devices. Let be the number of data samples used locally by the i-th mobile device. Let be the lightweight local policy parameter vector of mobile device i in round t; (5.2) Federated Coordination Server for Personalized Prompt Vectors Distillation and fusion are performed to obtain the global cue vector for the next round. : in, This indicates the feature extraction function. This represents the global hint vector for the next round; (5.3) Federal coordination server collects training latency Training energy consumption Transmission delay With transmission energy consumption Design a multi-objective reward function: Where U represents the multi-objective reward function, Indicates whether mobile device i participates in the current round of federated training; M represents the number of mobile devices currently participating in the federated learning task. These are the importance weight parameters corresponding to training latency, communication latency, training energy consumption, and communication energy consumption, respectively.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-network, multi-granularity resource optimization method based on federated deep reinforcement learning as described in any one of claims 1 to 9.

Images