Training delay optimization method and system based on cloud-edge-end three-layer reinforcement learning

Abstract

The invention discloses a training delay optimization method and system based on cloud-edge-end three-layer reinforcement learning, and belongs to the technical field of artificial intelligence and distributed computing. The method comprises the following steps: establishing a cloud-edge-end three-layer reinforcement learning framework, and initializing parameters; the edge agent collects a subnet state and outputs sequence division and local iteration times, the end equipment agent collects a local state and outputs a local training round and an uploading opportunity, and training and edge aggregation in the subnet are executed; the cloud agent collects a global state and outputs an aggregation period and a subnet priority, and edge model uploading and global model aggregation updating are executed every time the aggregation period is reached; calculating an agent reward of each layer, and updating a strategy network parameter of the corresponding agent; and repeating the steps until a total training round or global model convergence is achieved, and outputting the optimized global model. The problem of training delay caused by equipment heterogeneity, network fluctuation and task variability in a community scene is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and distributed computing technology, specifically relating to a training latency optimization method and system based on cloud-edge-device three-layer reinforcement learning. Background Technology

[0002] In smart community service scenarios, terminal devices such as smart sensors and video access control need to continuously collect diverse data such as residents' behavior trajectories and environmental monitoring. However, data processing in this scenario faces the dual demands of data privacy protection and efficient utilization. Therefore, hierarchical federated learning has become the mainstream technical solution.

[0003] However, the inherent characteristics of community scenarios exacerbate the challenges of training latency: due to device heterogeneity, the computing power of terminal devices varies significantly, and uneven distribution of training tasks can lead to some devices being overloaded or idle; community networks are highly dynamic and fluctuate, and the uncertainty of model update transmission latency is high, making fixed communication strategies unsuitable; different service scenarios have different model requirements, and static training parameters such as training rounds and aggregation cycles cannot meet the needs of dynamic switching between multiple tasks.

[0004] Existing federated learning latency optimization schemes have significant shortcomings. The application of reinforcement learning in federated learning is mostly limited to one or two layers, failing to fully leverage the collaborative decision-making advantages of the "cloud-edge-device" three-layer architecture. Therefore, there is an urgent need for a method that can adapt to dynamic community scenarios and optimize training latency through three-layer collaborative decision-making, minimizing end-to-end training latency while ensuring data privacy and model performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a training latency optimization method and system based on cloud-edge-device three-layer reinforcement learning, thereby solving the problems in existing technologies.

[0006] The objective of this invention can be achieved through the following technical solutions: A training latency optimization method and system based on cloud-edge-device three-layer reinforcement learning includes the following steps: S1. Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device. S2, the edge agent collects the subnet state and outputs the sequence division and local iteration number, the end device agent collects the local state and outputs the local training round and upload time, and then performs linear training and edge aggregation within the subnet to obtain the updated edge model; S3, the cloud-based intelligent agent collects the global status and outputs the aggregation period and subnet priority. When the aggregation period is reached, it performs edge model uploading and global model aggregation update based on the updated edge model. S4, based on the multi-objective reward function, combines the training latency within the subnet, device load, cloud aggregation latency, and model convergence progress indicators to calculate the rewards for each agent and update the policy network parameters of the corresponding agents. S5, repeat S2 to S4 until the total number of training rounds or global model convergence is reached, and output the optimized global model.

[0007] Furthermore, the cloud-edge-device three-layer reinforcement learning framework includes: The cloud module includes a global model server and a cloud agent. The global model server is used to initialize the global model, aggregate edge models, and distribute updated global models. The cloud agent is used to collect global status, decide the global aggregation cycle and subnet priority, calculate cloud rewards, and update its own policy network. Edge Module: One edge module is deployed in each subnet, which includes an edge server and an edge agent; the edge server is used for model aggregation within the subnet, updating edge models, and communicating with the cloud or end devices; the edge agent is used to collect subnet status, determine the maximum sequence length and local iteration number, calculate edge rewards, and update its own policy network. End device module: Each end device deploys one end device module, which includes a user device and an end device agent; the user device is used to perform local training and generate model updates; the end device agent is used to collect local state, decide on the local training rounds and upload timing, calculate end device rewards and update its own policy network; Iteration control module: Used to set the total number of training rounds and the initial learning rate, monitor the training process, determine whether the iteration termination condition has been met, and output the optimized global model.

[0008] Furthermore, the process by which the edge agent collects the subnet state and outputs the sequence partitioning and local iteration count includes: Edge agents collect subnet states ,in, For the computation delay of device i, For the transmission delay of device i, Average remaining power of devices within the subnet; Output action ,in The maximum sequence length, The number of local iterations is calculated based on the maximum sequence length. Divide the devices within the subnet into One training sequence; The process by which the end-device intelligent agent collects local status and outputs local training rounds and upload timing includes: End device intelligent agent collects local status ,in, The remaining battery power of device i. The length of the task queue for device i; output action. Among them, local training rounds and ≤ , Indicates the upload timing: 1 for immediate upload and 0 for delayed upload.

[0009] Furthermore, the process of linear training and edge aggregation within the subnet is as follows: Devices within each subnet are trained in a linear order. The first device is initialized using an edge model, and subsequent devices are updated using the model of the previous device. Local training is performed in rounds; edge servers aggregate all sequence models and update the edge models. .

[0010] Furthermore, the process by which the cloud-based intelligent agent collects the global state and outputs the aggregation period and subnet priority is as follows: Cloud-based intelligent agents collect global status ,in, The average training latency for each subnet, B is the average communication delay across subnets, and B is the cross-subnet communication bandwidth. The global model convergence progress is displayed; the action is output. Where q is the global aggregation period, The training priority for subnet d.

[0011] Furthermore, the global model aggregation update process is as follows: After each q-round of training within the network, each edge server... Priority order for uploading edge models Uploaded to the cloud; the cloud-based intelligent agent aggregates edge models according to the weight matrix H and updates the global model. And distribute them to the edge servers of each subnet.

[0012] Furthermore, the cloud-based intelligent agent reward The formula for calculation is: in, , , These are the weighting coefficients. Due to cloud aggregation delay, For the communication delay between subnet d and the cloud, Δ For model convergence increments; The edge agent reward The formula for calculation is: in, , These are the weighting coefficients. For training delay within subnet d, This represents the total communication latency between devices within subnet d and the edge server. The load of device i; The terminal device intelligent agent reward The formula for calculation is: ) in, , These are the weighting coefficients. For the computation delay of device i, For the transmission delay of device i, This represents the remaining battery power of device i.

[0013] A training latency optimization system based on a three-layer reinforcement learning model (cloud-edge-device) includes: Initialization Unit: Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device; Training unit within the subnet: The edge agent collects the subnet state and outputs the sequence division and local iteration count. The end device agent collects the local state and outputs the local training round and upload timing. Then, linear training and edge aggregation are performed within the subnet to obtain the updated edge model. Global Aggregation Unit: The cloud-based intelligent agent collects the global status and outputs the aggregation period and subnet priority. When the aggregation period is reached, it performs edge model uploading and global model aggregation update based on the updated edge model. The policy update unit calculates the agent rewards at each layer based on the multi-objective reward function, combined with the training latency within the subnet, device load, cloud aggregation latency, and model convergence progress indicators, and updates the policy network parameters of the corresponding agents. Iterative output unit: Repeat the training unit in the subnet to the policy update unit until the total number of training rounds or global model convergence is reached, and output the optimized global model.

[0014] A computer storage medium storing a readable program that, when executed, instructs a computing device to perform the training latency optimization method based on a cloud-edge-device three-layer reinforcement learning as described above.

[0015] An electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the training latency optimization method based on the cloud-edge-device three-layer reinforcement learning described above.

[0016] The beneficial effects of this invention are: 1. This invention addresses the challenges of heterogeneous devices, network fluctuations, and variable tasks in community scenarios by coordinating three layers of intelligent agents: cloud, edge, and device. It adjusts training parameters based on the dynamic states of the global, subnet, and local environments. Optimization of dynamic aggregation period, serialization, and upload timing significantly reduces overall training latency. Furthermore, the reward function designed in this invention simultaneously considers multiple objectives, including latency, accuracy, and energy consumption, avoiding the degradation of model accuracy or device battery depletion caused by optimizing latency alone, thus meeting the actual needs of smart community services.

[0017] 2. The cloud-edge-device three-layer reinforcement learning framework of this invention achieves hierarchical management and control of "global-local" through modular division of labor. The global model server in the cloud module ensures unified initialization and aggregated updates of the model, and the global state perception and decision-making capabilities of the cloud agent enable efficient scheduling of training resources across multiple subnets in the community. The subnet-level deployment of the edge module adapts to the physical area division of the community, the intra-subnet aggregation function of the edge server reduces cross-subnet communication overhead, and the sequence partitioning decision of the edge agent makes device training more collaborative. The device-level deployment of the device module conforms to the heterogeneous status of community terminals, and the local state perception and decision-making capabilities of the device agent enable devices to autonomously adapt to their own computing power and power consumption for training. This architecture design not only ensures the consistency of global training but also fully releases the local optimization potential of the edge and device sides, which is the foundation for the efficient execution of multi-scenario and multi-device training tasks in the community.

[0018] 3. The mechanism by which the edge agent collects subnet states and outputs sequence partitioning and local iteration counts in this invention is key to optimizing training latency within the subnet. By dynamically partitioning training sequences, devices with similar computing power and communication capabilities are grouped together for training, avoiding training rhythm discrepancies caused by differences in device performance. The adaptive decision-making of local iteration counts allows the subnet to flexibly adjust the training rounds based on its own data volume and model complexity, minimizing unnecessary training time while ensuring model convergence. This refined decision-making at the subnet level effectively solves the training coordination problem caused by heterogeneous devices within the same subnet in the community, improving training efficiency within the subnet and laying a local foundation for global training latency optimization.

[0019] 4. The mechanism of linear training within the subnet and edge aggregation in this invention is the core of achieving efficient collaborative training of models within the subnet. Devices train in a linear sequence, with model updates from one device directly providing initialization for the next, avoiding the communication overhead of frequent synchronization between devices in traditional parallel training. The edge server's aggregation and update of the sequential models allows the training results within the subnet to be quickly integrated into a unified edge model. This mechanism is particularly well-suited to the physical proximity of devices within a community subnet, utilizing the advantages of local communication between devices while minimizing training latency within the subnet through the linear training and aggregation process design. It is a key design for achieving breakthroughs in subnet-level training efficiency in community scenarios.

[0020] 5. The mechanism in this invention, where the cloud-based intelligent agent performs global aggregation updates based on the updated edge models of each subnet, is the core of achieving a balance between global training efficiency and model performance. Dynamic decision-making regarding the aggregation cycle allows the cloud to flexibly select aggregation timing based on global convergence progress and network bandwidth status, avoiding network congestion caused by frequent aggregation during periods of bandwidth shortage, or model accuracy stagnation due to insufficient aggregation during critical convergence periods. Subnet priority decision-making allows the cloud to prioritize aggregation of subnet models with high data value and fast training progress, accelerating global model convergence. This global-level intelligent scheduling ensures optimal global allocation of training resources across multiple subnets in the community, reducing overall training latency while guaranteeing the adaptability of the global model to various community tasks.

[0021] 6. The mechanism of calculating rewards for each layer of agents and updating the policy network based on a multi-objective reward function is key to achieving multi-objective collaborative optimization of latency, accuracy, and energy consumption. The multi-dimensional weighting of latency, accuracy, and energy consumption in the reward function ensures that agents at each layer do not solely pursue latency reduction during decision-making, but rather comprehensively consider the model's usability in real-world community scenarios. This multi-objective-oriented reinforcement learning training enables the final optimized model to be quickly trained and deployed, and to operate stably and reliably in scenarios such as smart security, health management, and community services, truly meeting the diverse needs of smart community services.

[0022] 7. The iterative termination mechanism based on the total number of training rounds and global model convergence in this invention ensures precise utilization of training resources. By simultaneously monitoring the number of training rounds and model convergence, it avoids resource waste or undertraining caused by fixed-round training. In community scenarios, this mechanism allows the training process to terminate in a timely manner when the model meets the accuracy requirements of intelligent services, ensuring model availability while maximizing the conservation of computing power and power resources of edge and terminal devices, thus providing crucial support for community training tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the training latency optimization method of the present invention; Figure 2 This is a schematic diagram of the cloud-edge-device three-layer reinforcement learning framework structure of the present invention; Figure 3 This is a closed-loop flowchart of the three-layer intelligent agent of the present invention: "state perception - decision execution - training aggregation - reward feedback". Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1 like Figure 1 As shown, the training latency optimization method based on cloud-edge-device three-layer reinforcement learning includes the following steps: S1. Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device. like Figure 2 As shown, the cloud-edge-device three-layer reinforcement learning framework includes: The cloud module includes a global model server and a cloud agent. The global model server is used to initialize the global model, aggregate edge models, and distribute updated global models. The cloud agent is used to collect global status (global latency, bandwidth, convergence progress), decide the global aggregation cycle and subnet priority, calculate cloud rewards, and update its own policy network. Edge Module: One edge module is deployed in each subnet, which includes an edge server and an edge agent; the edge server is used for model aggregation within the subnet, updating edge models, and communicating with the cloud or edge devices; the edge agent is used to collect subnet status (device computing power, transmission latency, remaining power), determine the maximum sequence length and local iteration count, calculate edge rewards, and update its own policy network; End device module: Each end device deploys one end device module, which includes a user device and an end device agent; the user device is used to perform local training and generate model updates; the end device agent is used to collect local status (calculation load, power consumption, task queue), decide the local training rounds and upload timing, calculate end device rewards and update its own policy network; Iteration control module: Used to set the total number of training rounds and the initial learning rate, monitor the training process, determine whether the iteration termination condition has been met (total number of training rounds or global model convergence), and output the optimized global model.

[0027] Each layer of intelligent agents possesses state awareness (collection) capabilities, with the following state awareness range: End-device intelligent agent: collects local hardware computing power, battery level, task queue length, and local training progress; Edge intelligent agent: Aggregates the status of all end devices within the subnet, and supplements the collection of edge server CPU / GPU load, channel bandwidth utilization within the subnet, communication latency with the cloud, and model parameter differences within the subnet; Cloud-based intelligent agent: Integrates the status of all edge subnets, collects cross-subnet communication bandwidth, global network latency, aggregation results of each subnet model, and global convergence progress.

[0028] In some embodiments, the terminal device may specifically include a community smart sensor, a video access control device, or a wearable health device; the edge module corresponds to the edge node of the community subnet; and the cloud module corresponds to the community smart service cloud platform.

[0029] In S1, the initialization process includes: 1) Initialize the global model Set the total number of training rounds K and the initial learning rate η; 2) Initialize the policy network parameters of the cloud-based intelligent agent. And the state space, which includes global latency, network bandwidth, and global model convergence progress.

[0030] 3) Initialize the policy network parameters of the edge agent. (d is the subnet number) and the state space, which includes the device's computational delay, transmission delay and remaining power.

[0031] 4) Initialize the policy network parameters of the terminal device's intelligent agent. (i is the device number) and a state space, which contains the device’s local computing load, battery level, and task queue length.

[0032] S2, the edge agent collects the subnet state and outputs the sequence division and local iteration number, the end device agent collects the local state and outputs the local training round and upload time, and then performs linear training and edge aggregation within the subnet; 1) Edge agent decision making Edge agents collect subnet state data: ( For the computation delay of device i, For the transmission delay of device i, (Average remaining power of devices within the subnet), output action. ( The maximum sequence length, (Number of local iterations), based on the maximum sequence length Divide the devices within the subnet into One training sequence; 2) Decision-making by intelligent agents on end devices End device intelligent agent collects local status ( The remaining battery power of device i. (where i is the length of the task queue for device i), output action. ( For local training rounds and ≤ , Indicates the upload timing (1 for immediate upload, 0 for delayed upload). The process of linear training and edge aggregation within the subnet is as follows: Devices within each subnet are trained in a linear order. The first device is initialized using an edge model, and subsequent devices are updated using the model of the previous device. Local training round (gradient update formula is) , Let i be the training parameters for device i in the k-th round. (For device i, the loss function in the k-th round); the edge server aggregates all sequence models and updates the edge model. .

[0033] S3, the cloud-based intelligent agent collects the global state and outputs the aggregation period and subnet priority. When the aggregation period is reached, based on the updated edge model obtained after training and aggregation of each subnet in S2, the edge model upload and global model aggregation update are performed. The process of cloud-based agent decision-making (where the cloud-based agent collects global state and outputs aggregation period and subnet priority) includes: Cloud-based intelligent agents collect global status ( The average training latency for each subnet, B is the average communication delay across subnets, and B is the cross-subnet communication bandwidth. (For global model convergence progress), output actions. (q is the global aggregation period, (Training priority for subnet d); The process of global model aggregation and update is as follows: After each q-round of training within the network, each edge server... Priority order for uploading edge models Uploaded to the cloud; cloud-based intelligent agents aggregate edge models according to weight matrix H (aggregation formula is...). , For the edge model of subnet d in the kth round, (For the aggregate weights of subnet d), update the global model. And distribute them to the edge servers of each subnet.

[0034] S4, based on the multi-objective reward function, combines the training latency within the subnet in S2, device load, and cloud aggregation latency and model convergence progress in S3 to calculate the rewards of agents at each layer, and updates the policy network parameters of the corresponding agents based on the rewards of agents at each layer. Cloud-based intelligent agent rewards The formula for calculation is: in, , , These are the weighting coefficients. Due to cloud aggregation delay, For the communication delay between subnet d and the cloud, For model convergence increments; Edge agent rewards The formula for calculation is: in, , These are the weighting coefficients. For training delay within subnet d, This represents the total communication latency between devices within subnet d and the edge server. The load of device i; End device intelligent agent rewards The formula for calculation is: ) in, , These are the weighting coefficients; For the computation delay of device i, For the transmission delay of device i, The remaining battery power of device i; Finally, based on the reward values ​​of each layer, the policy network parameters of the corresponding agent are updated using temporal difference learning or policy gradient method. ).

[0035] Total training latency It consists of three parts: .

[0036] like Figure 3 As shown, in this process, the three-layered intelligent agent forms a complete closed loop through "state perception - decision execution - training aggregation - reward feedback," and the layers form cross-layer closed loops through data and instruction interaction. Their collaborative logic can be achieved through... Figure 3 Clearly reflected.

[0037] S5, repeat S2 to S4 until the total number of training rounds or global model convergence is reached, and output the optimized global model.

[0038] Based on a similar inventive concept, embodiments of the present invention also provide a computer storage medium storing a readable program that, when run by a processor, can execute the above-described training latency optimization method based on a three-layer reinforcement learning architecture (cloud-edge-device).

[0039] Based on a similar inventive concept, this invention provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the training latency optimization method based on the cloud-edge-device three-layer reinforcement learning described above.

[0040] Based on a similar inventive concept, embodiments of the present invention also provide a computer program product, including computer instructions, which instruct a computing device to perform the operations corresponding to the above-described training latency optimization method based on cloud-edge-device three-layer reinforcement learning.

[0041] Example 2 The technical solution of the present invention will be described in detail through specific examples; firstly, the core terms involved in this embodiment will be defined to ensure consistent understanding: Subnetting is based on the physical areas of the smart community, such as "Building 1 Subnet" and "Building 2 Subnet". Each subnet contains the terminal devices within that area. For example, Building 1 Subnet contains 10 video access control devices and 5 human infrared sensors. Training tasks refer to the model training needs for different service scenarios in the community, such as "high-altitude object throwing detection model", "key population trajectory tracking model", and "resident health data prediction model". Latency metrics: This represents the cloud aggregation latency, which is the time it takes for the cloud to receive all the subnet edge models, perform weight aggregation, and update the global model. This represents the training latency within subnet d, including the local training time of all devices within the subnet and the aggregated training time of the edge server. This represents the communication latency between subnet d and the cloud, specifically the time it takes for the edge server to upload the edge model to the cloud. The computing power level of the devices is classified according to the types of devices in the smart community scenario, and the devices are divided into three categories: high computing power devices include video access control and facial recognition access control; medium computing power devices include China Telecom central screen and smart door magnets; low computing power devices include smart bracelets, smart badges and smart smoke detectors.

[0042] In this embodiment, the smart community "video surveillance scenario" needs to train a "high-altitude object throwing detection model" to automatically capture events of objects being thrown from high-rise buildings. This scenario involves the following equipment: Terminal equipment: 12 primary high-trajectory cameras and 2 backup high-trajectory cameras; the computing power of a single camera supports the training time for a single round. ; Edge module: The community is divided into 3 subnets based on 3 areas, and 1 edge server is deployed in each subnet; Cloud module: Community Smart Service Cloud Platform; In this scenario, the training requirement is to achieve a high model accuracy. The delay for a single training session should be ≤8 minutes to avoid untimely alarms for projectile events due to excessive delay.

[0043] like Figure 1 As shown, a training latency optimization method based on three-layer reinforcement learning includes the following steps: first, initializing the framework and setting parameters, and then loading pre-trained weights as the initial global model. Simultaneously initialize the policy network parameters and state space of the three-layer intelligent agents: cloud, edge, and terminal device. Then, subnet training and edge-device agent decision-making are carried out. The edge agent collects the subnet state output sequence length and iteration number and divides the training sequence. The device agent collects the local state output training round and upload time. Then, subnet linear training and edge aggregation are performed. Next, global aggregation and cloud-based agent decision-making are performed. The cloud-based agent collects the global state, outputs the aggregation period and subnet priority, uploads the edge model according to the period and priority, and completes cloud aggregation and model distribution. Then, by calculating the agent rewards at each layer, the REINFORCE algorithm is used to update the policy network parameters. Finally, repeat the above training, aggregation, and update steps until the model accuracy reaches the target or the total number of training rounds is reached, and output the optimized global model.

[0044] Specifically, the present invention will be further illustrated below through embodiments: The process of latency optimization for training using data collected from terminal devices is as follows: Perform global model initialization and load pre-trained weights as the initial model. Let the total number of training rounds be... Initial learning rate ; Initialize the parameters of the three-layer agent: Cloud-based intelligent agent policy network parameters A 3-layer fully connected neural network is used, with an input dimension of 4, an output dimension of 2, and a state space of... ,(initial minute, , , ; The three subnets of the edge agent correspond to respectively state space ,(initial , , ).

[0045] End-device intelligent agent: 14 high-angle cameras respectively corresponding to state space ,initial No backlog of tasks.

[0046] Training within the subnet and decision-making for edge-to-end device agents are performed. Taking the "Building 1 subnet" as an example, the specific implementation process consists of the following steps: (1) Edge agent decision-making: Collect the current status. With sufficient network bandwidth, the policy network outputs actions. Divide the four cameras into A sequence.

[0047] (2) Decision-making by terminal device intelligent agent: The terminal device performs status awareness, and camera 1 collects data. No backlog of tasks, output actions This means 15 rounds of local training, followed by immediate upload upon completion. The same applies to other cameras, as they are all high-angle cameras with similar architecture, and the output... .

[0048] (3) Training and aggregation within subnets: In Sequence 1, Camera 1 is initialized with an edge model and undergoes 15 rounds of local training, with the gradient update formula... After training, the model is updated and uploaded to the edge server; Camera 2 is initialized using the trained model from Camera 1, and the training and uploading are repeated. The edge server receives model updates from two sequences, using a weighted average with each camera's data volume weighted at 25%, to update the edge model. ,time consuming ( ) 56s.

[0049] Global aggregation and cloud-based agent decision-making are performed, comprising two stages: cloud-based agent decision-making and global model updates. The cloud-based agent performs state perception and collects data. Because the convergence progress is still 10% and bandwidth is sufficient, the output... .

[0050] After training the four subnets, the edge model of Building 2 is uploaded first, followed by Buildings 3 and 1 in sequence. The cloud then processes the data according to the weight matrix. Aggregation, Aggregation Formula ,time consuming ;Will Distributed to 3 edge servers, taking 15 seconds.

[0051] Reinforcement learning strategy updates, calculating rewards at each level: Cloud-based rewards: ; Calculate marginal reward: ; Computing device rewards: =−0.7×(15×0.8+2)+0.3×90=17.2; Update parameters using the REINFORCE algorithm, such as cloud-based intelligent agents: After adjustment, the next aggregation cycle q will remain at 4, and the priority sorting will be optimized to... .

[0052] Repeating the above steps, the model accuracy reached 92.3% on the 120th training round, satisfying the requirements. Stop training and output the optimized global model. .

[0053] In practical verification, after optimization using this method, the average latency of a single training round decreased from the initial 7.2 minutes to 4.8 minutes, of which... From 45s to 32s, It decreased from 0.93 minutes to 0.65 minutes. The time was reduced from 83 seconds to 55 seconds, meeting the requirement of "≤8 minutes"; the final model accuracy was 92.3%, the recall rate for parabolic event recognition was 91%, and the false negative rate was ≤3%, which meets the requirements of community security.

[0054] In addition to smart community services, this invention can also be applied to the following scenarios: Industrial Internet of Things (IIoT): Optimize the training latency of federated learning for sensors and machine tools within the factory to ensure the real-time performance of production prediction models; Smart cities: Coordinate the training tasks of traffic cameras and environmental monitoring stations to minimize the training latency of traffic flow prediction models; Telemedicine: Coordinate the training parameters of wearable medical devices and hospital edge nodes to accelerate the iteration of diagnostic models while protecting patient privacy.

[0055] In summary, this invention effectively solves the training latency problem in dynamic scenarios through collaborative decision-making of a three-layer reinforcement learning agent consisting of cloud, edge, and device, while balancing model performance and device energy consumption, providing technical support for the engineering application of distributed federated learning.

[0056] Example 3 In this embodiment, a training latency optimization system based on a three-layer cloud-edge-device reinforcement learning model is proposed, including: Initialization Unit: Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device; Training unit within the subnet: The edge agent collects the subnet state and outputs the sequence division and local iteration count. The end device agent collects the local state and outputs the local training round and upload timing. Then, linear training and edge aggregation are performed within the subnet to obtain the updated edge model. Global Aggregation Unit: The cloud-based intelligent agent collects the global status and outputs the aggregation period and subnet priority. When the aggregation period is reached, it performs edge model uploading and global model aggregation update based on the updated edge model. The policy update unit calculates the agent rewards at each layer based on the multi-objective reward function, combined with the training latency within the subnet, device load, cloud aggregation latency, and model convergence progress indicators, and updates the policy network parameters of the corresponding agents. Iterative output unit: Repeat the training unit in the subnet to the policy update unit until the total number of training rounds or global model convergence is reached, and output the optimized global model.

[0057] The methods of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses the code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0058] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A training latency optimization method and system based on cloud-edge-device three-layer reinforcement learning, characterized in that, Includes the following steps: S1. Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device. S2, the edge agent collects the subnet state and outputs the sequence division and local iteration number, the end device agent collects the local state and outputs the local training round and upload time, and then performs linear training and edge aggregation within the subnet to obtain the updated edge model; S3, the cloud-based intelligent agent collects the global status and outputs the aggregation period and subnet priority. When the aggregation period is reached, it performs edge model uploading and global model aggregation update based on the updated edge model. S4, based on the multi-objective reward function, combines the training latency within the subnet, device load, cloud aggregation latency, and model convergence progress indicators to calculate the rewards for each agent and update the policy network parameters of the corresponding agents. S5, repeat S2 to S4 until the total number of training rounds or global model convergence is reached, and output the optimized global model.

2. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 1, characterized in that, The cloud-edge-device three-layer reinforcement learning framework includes: The cloud module includes a global model server and a cloud agent. The global model server is used to initialize the global model, aggregate edge models, and distribute updated global models. The cloud agent is used to collect global status, decide the global aggregation cycle and subnet priority, calculate cloud rewards, and update its own policy network. Edge Module: One edge module is deployed in each subnet, which includes an edge server and an edge agent; the edge server is used for model aggregation within the subnet, updating edge models, and communicating with the cloud or end devices; the edge agent is used to collect subnet status, determine the maximum sequence length and local iteration number, calculate edge rewards, and update its own policy network. End device module: Each end device deploys one end device module, which includes a user device and an end device agent; the user device is used to perform local training and generate model updates; the end device agent is used to collect local state, decide on the local training rounds and upload timing, calculate end device rewards and update its own policy network; Iteration control module: Used to set the total number of training rounds and the initial learning rate, monitor the training process, determine whether the iteration termination condition has been met, and output the optimized global model.

3. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 1, characterized in that, The process by which the edge agent collects the subnet state and outputs the sequence partitioning and local iteration count includes: Edge agents collect subnet states ,in, For the computation delay of device i, For the transmission delay of device i, Average remaining power of devices within the subnet; Output action ,in The maximum sequence length, The number of local iterations is calculated based on the maximum sequence length. Divide the devices within the subnet into One training sequence; The process by which the end-device intelligent agent collects local status and outputs local training rounds and upload timing includes: End device intelligent agent collects local status ,in, The remaining battery power of device i. The length of the task queue for device i; output action. Among them, local training rounds and ≤ , Indicates the upload timing: 1 for immediate upload and 0 for delayed upload.

4. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 3, characterized in that, The process of linear training and edge aggregation within the subnet is as follows: Devices within each subnet are trained in a linear order. The first device is initialized using an edge model, and subsequent devices are updated using the model of the previous device. Rotate to local training; The edge server aggregates all sequence models and updates the edge model. .

5. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 3, characterized in that, The process by which a cloud-based intelligent agent collects global state and outputs aggregation period and subnet priority is as follows: Cloud-based intelligent agents collect global status ,in, The average training latency for each subnet, B is the average communication delay across subnets, and B is the cross-subnet communication bandwidth. The global model convergence progress is displayed; the action is output. Where q is the global aggregation period, The training priority for subnet d.

6. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 5, characterized in that, The process of global model aggregation and update is as follows: After each q-round of training within the network, each edge server... Priority order for uploading edge models Uploaded to the cloud; the cloud-based intelligent agent aggregates edge models according to the weight matrix H and updates the global model. And distribute them to the edge servers of each subnet.

7. The training latency optimization method based on cloud-edge-device three-layer reinforcement learning according to claim 1, characterized in that, The cloud-based intelligent agent reward The formula for calculation is: in, , , These are the weighting coefficients. Due to cloud aggregation delay, For the communication delay between subnet d and the cloud, Δ For model convergence increments; The edge agent reward The formula for calculation is: in, , These are the weighting coefficients. For training delay within subnet d, This represents the total communication latency between devices within subnet d and the edge server. The load of device i; The terminal device intelligent agent reward The formula for calculation is: ) in, , These are the weighting coefficients. For the computation delay of device i, For the transmission delay of device i, This represents the remaining battery power of device i.

8. A training latency optimization system based on a three-layer cloud-edge-device reinforcement learning model, characterized in that, include: Initialization Unit: Establish a three-layer reinforcement learning framework of cloud-edge-device, initialize the global model, total number of training rounds, initial learning rate, and initialize the policy network parameters and state space of the three-layer intelligent agents of cloud, edge and device; Training unit within the subnet: The edge agent collects the subnet state and outputs the sequence division and local iteration number, while the end device agent collects the local state and outputs the local training round and upload timing. Then, linear training and edge aggregation are performed within the subnet to obtain the updated edge model. Global Aggregation Unit: The cloud-based intelligent agent collects the global status and outputs the aggregation period and subnet priority. When the aggregation period is reached, it performs edge model uploading and global model aggregation update based on the updated edge model. The policy update unit calculates the agent rewards at each layer based on the multi-objective reward function, combined with the training latency within the subnet, device load, cloud aggregation latency, and model convergence progress indicators, and updates the policy network parameters of the corresponding agents. Iterative output unit: Repeat the training unit in the subnet to the policy update unit until the total number of training rounds or global model convergence is reached, and output the optimized global model.

9. A computer storage medium storing a readable program, characterized in that, When the program runs, it can instruct the computing device to perform the training latency optimization method based on cloud-edge-device three-layer reinforcement learning as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the training latency optimization method based on cloud-edge-device three-layer reinforcement learning as described in any one of claims 1-7.

Images