Distributed reinforcement learning optimization method and device for communication resource limited scene

Abstract

The invention provides a distributed reinforcement learning optimization method and device for a communication resource limited scene, and relates to the technical field of wireless communication. The method comprises the following steps: establishing a distributed reinforcement learning model comprising a central controller and a plurality of reinforcement learning nodes, and constructing a global objective function; a local gradient estimation vector is obtained through reinforcement learning nodes based on interactive calculation of global strategy parameters and a local environment, a weighted information index, an inter-node redundancy index and a communication resource overhead index are calculated, and a target function with sub-model characteristics is constructed; a reinforcement learning node set participating in uploading is selected under the condition that the maximum number of participating nodes and communication resource budget constraints are given; and performing aggregation through the central controller to obtain a global gradient estimation vector, and updating a global optimal strategy parameter. According to the method, while the convergence performance of distributed reinforcement learning and the task unloading service quality are ensured, the training communication burden is effectively reduced, and the utilization efficiency of communication resources is improved.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication technology and reinforcement learning technology, and in particular to a distributed reinforcement learning optimization method and apparatus for scenarios with limited communication resources. Background Technology

[0002] With the rapid development of mobile internet, IoT, and edge computing technologies, numerous services require automatic acquisition of decision-making strategies through reinforcement learning in complex and dynamic environments. To improve training efficiency and policy generalization capabilities, distributed reinforcement learning has gradually become an important development direction. This involves parallel sampling and updating across multiple learning nodes, followed by parameter or gradient interaction with a central controller. In practical deployment scenarios such as cellular networks, wireless edge networks, and drone networks, learning nodes are often located on the access side or edge side, interconnected with the central controller via wireless or wired links with limited bandwidth and latency. However, the coexistence of limited communication resources and high concurrency business demands means that frequent full-node model synchronization and gradient uploading not only increase latency and energy consumption but also consume bandwidth for business transmission, becoming a significant bottleneck restricting the overall performance improvement of the system.

[0003] Therefore, there is an urgent need to propose a method for optimizing the communication overhead of distributed reinforcement learning in scenarios with limited communication resources, which can effectively reduce training communication overhead and improve the overall operating efficiency of the system while ensuring the convergence performance of distributed reinforcement learning and the quality of business services. Summary of the Invention

[0004] To address the technical problems of heavy communication burden and competition for limited link resources between training and service communication during distributed reinforcement learning training in scenarios with limited communication resources, this invention provides a distributed reinforcement learning optimization method and apparatus for such scenarios. The technical solution is as follows:

[0005] On the one hand, a distributed reinforcement learning optimization method for communication resource-constrained scenarios is provided. This method is implemented by a distributed reinforcement learning communication overhead optimization device, and includes: S1. Establish a distributed reinforcement learning model including a central controller and multiple reinforcement learning nodes, determine the comprehensive reward function of each reinforcement learning node, construct a global objective function based on the comprehensive reward function of all reinforcement learning nodes, and initialize the global policy parameters based on the global objective function.

[0006] S2. Each reinforcement learning node interacts with the local environment based on global policy parameters to obtain a state-action-reward sequence. The local gradient estimation vector is calculated using the policy gradient method based on the state-action-reward sequence. The gradient information content index of the reinforcement learning node is calculated based on the norm of the local gradient estimation vector.

[0007] S3. Maintain the information age for each reinforcement learning node, construct a time-sensitive weight function based on the information age, and multiply the gradient information content index with the time-sensitive weight function to obtain the weighted information index.

[0008] S4. Calculate the difference between the local gradient estimation vectors of any two reinforcement learning nodes, define the gradient similarity coefficient based on the difference between the local gradient estimation vectors, and obtain the redundancy index between nodes based on the gradient similarity coefficient.

[0009] S5. Calculate the training communication latency and training communication energy consumption based on the link rate, transmission power, and gradient data volume between each reinforcement learning node and the central controller. Then, linearly combine the training communication latency and training communication energy consumption to obtain the communication resource overhead index.

[0010] S6. Construct a sub-model-like objective function based on the weighted information index, inter-node redundancy index, and communication resource overhead index. Under the given maximum number of participating nodes and communication resource budget constraints, select the set of reinforcement learning nodes to participate in uploading based on the sub-model-like objective function and sub-model optimization algorithm, and upload the local gradient estimation vector of the selected reinforcement learning node set to the central controller.

[0011] S7. The central controller aggregates the received local gradient estimation vectors to obtain the global gradient estimation vector, updates the global optimal policy parameters based on the global gradient estimation vector, and distributes the updated global optimal policy parameters to each reinforcement learning node to achieve distributed reinforcement learning communication overhead optimization for scenarios with limited communication resources.

[0012] Optionally, the establishment in S1 includes a distributed reinforcement learning model with a central controller and multiple reinforcement learning nodes, determining the comprehensive reward function for each reinforcement learning node, including: For a distributed reinforcement learning system that includes a central controller and multiple reinforcement learning nodes, each reinforcement learning node is modeled as a reinforcement learning agent with a parameterized policy function, and a comprehensive reward function for each reinforcement learning node is defined according to the policy parameters. The comprehensive reward function includes indicators such as node information content, information timeliness, information similarity, training communication latency, and training communication energy consumption.

[0013] Optionally, S3 maintains an informational age for each reinforcement learning node, including: Determine the position of each reinforcement learning node at the 1st epoch. Whether it is selected to participate in the local gradient estimation vector upload during the training round; if selected, then in the... During each training round, the information age of the reinforcement learning node is reset to 1; if it is not selected, then in the next training round... During each training round, the information age of the reinforcement learning node is updated to the information age plus 1.

[0014] Optionally, the gradient similarity coefficient in S4 is shown in equation (1) below: (1) In the formula, Indicates the first The reinforcement learning node and the first The reinforcement learning node at the _ Gradient similarity coefficients in rounds of training Indicates the first The reinforcement learning node at the _ The local gradient estimation vector during training rounds. Indicates the first The reinforcement learning node at the _ The local gradient estimation vector during training rounds. This represents the smoothing parameter.

[0015] Optionally, in S5, the training communication latency and training communication energy consumption are calculated based on the link rate, transmit power, and gradient data volume between each reinforcement learning node and the central controller, including: The training communication latency is calculated based on the link rate and gradient data volume between each reinforcement learning node and the central controller.

[0016] The training communication energy consumption is calculated based on the training communication latency and the transmit power between each reinforcement learning node and the central controller.

[0017] Alternatively, the objective function with sub-model characteristics in S6 is shown in equation (2) below: (2) In the formula, This represents the objective function with submodel characteristics. Represents any set of candidate nodes. Represents the set of reinforcement learning nodes. This represents a weighted information indicator. Indicates training rounds, Indicates a similarity penalty. Represents the gradient similarity coefficient. The communication cost weight parameter represents the communication cost. This represents the communication resource overhead indicator.

[0018] Optionally, in S6, given the maximum number of participating nodes and communication resource budget constraints, the set of reinforcement learning nodes participating in uploading is selected based on a sub-model-like objective function and a sub-model optimization algorithm, including: Given a maximum number of participating nodes, the set of reinforcement learning nodes participating in uploading is set to an empty set. The reinforcement learning node that maximizes the gain of the objective function is selected iteratively and added to the set of reinforcement learning nodes participating in uploading. This process continues until the addition of any unselected reinforcement learning node no longer increases the gain of the objective function or the given communication resource budget is exhausted, thus obtaining the set of reinforcement learning nodes participating in uploading.

[0019] On the other hand, a distributed reinforcement learning optimization device for communication resource-constrained scenarios is provided. This device is applied to a distributed reinforcement learning optimization method for communication resource-constrained scenarios, and the device includes: The model building module is used to build a distributed reinforcement learning model including a central controller and multiple reinforcement learning nodes, determine the comprehensive reward function of each reinforcement learning node, construct a global objective function based on the comprehensive reward function of all reinforcement learning nodes, and initialize global policy parameters based on the global objective function.

[0020] The gradient information content calculation module is used to obtain the state-action-reward sequence by interacting with the local environment based on the global policy parameters of each reinforcement learning node. The local gradient estimation vector is calculated using the policy gradient method based on the state-action-reward sequence. The gradient information content of the reinforcement learning node is calculated based on the norm of the local gradient estimation vector.

[0021] The weighted information index calculation module is used to maintain the information age for each reinforcement learning node, construct a time-sensitive weight function based on the information age, and multiply the gradient information content index by the time-sensitive weight function to obtain the weighted information index.

[0022] The node redundancy index calculation module is used to calculate the difference between the local gradient estimation vectors of any two reinforcement learning nodes, define the gradient similarity coefficient based on the difference between the local gradient estimation vectors, and obtain the node redundancy index based on the gradient similarity coefficient.

[0023] The communication overhead evaluation module is used to calculate the training communication latency and training communication energy consumption based on the link rate, transmission power and gradient data volume between each reinforcement learning node and the central controller, and to obtain the communication resource overhead index by linearly combining the training communication latency and training communication energy consumption.

[0024] The sub-model optimization module is used to construct an objective function with sub-model characteristics based on weighted information indicators, inter-node redundancy indicators, and communication resource overhead indicators. Under the given constraints of the maximum number of participating nodes and communication resource budget, it selects the set of reinforcement learning nodes to participate in uploading based on the objective function with sub-model characteristics and the sub-model optimization algorithm, and uploads the local gradient estimation vector of the selected reinforcement learning node set to the central controller.

[0025] The policy update module is used by the central controller to aggregate the received local gradient estimation vectors to obtain the global gradient estimation vector, update the global optimal policy parameters based on the global gradient estimation vector, and distribute the updated global optimal policy parameters to each reinforcement learning node, thereby optimizing the communication overhead of distributed reinforcement learning in scenarios with limited communication resources.

[0026] On the other hand, a distributed reinforcement learning communication overhead optimization device is provided, the distributed reinforcement learning communication overhead optimization device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for distributed reinforcement learning optimization methods for communication resource-constrained scenarios.

[0027] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described distributed reinforcement learning optimization methods for communication resource-constrained scenarios.

[0028] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this invention, multiple factors such as the gradient information content of learning nodes, information timeliness, redundancy between nodes, and communication resource overhead are considered simultaneously under a unified framework. Compared with the scheme that selects nodes based only on gradient size or a single communication indicator, it can more accurately depict the occupation of physical links by training communication and is more in line with the actual needs of scenarios with limited communication resources.

[0029] This invention constructs an objective function with sub-model characteristics and uses a low-complexity greedy algorithm for node selection. Under the premise of satisfying the constraints of the number of participating nodes or communication resource budget, it can obtain a near-optimal subset of nodes. While effectively reducing the number of training communications and the total communication overhead, it maintains full utilization of key gradient information.

[0030] The communication overhead optimization mechanism proposed in this invention can adaptively adjust the node selection results according to changes in network status, giving priority to learning nodes with rich and fresh information and representative gradients. This improves the efficiency of communication resource utilization while ensuring the convergence performance of distributed reinforcement learning and the quality of business services, and is applicable to various distributed intelligent control scenarios with limited communication resources. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a distributed reinforcement learning optimization method for communication resource-constrained scenarios provided by an embodiment of the present invention. Figure 2 This is a flowchart of a greedy algorithm for node selection / sub-module in a distributed reinforcement learning optimization method for communication resource-constrained scenarios, provided by an embodiment of the present invention. Figure 3 This is a diagram of the multi-task unloading model provided in an embodiment of the present invention; Figure 4 This is a block diagram of a distributed reinforcement learning optimization device for communication resource-constrained scenarios provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a distributed reinforcement learning communication overhead optimization device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0034] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0035] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0036] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0037] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0038] This invention provides a distributed reinforcement learning optimization method for scenarios with limited communication resources. This method can be implemented using a distributed reinforcement learning communication overhead optimization device, which can be a terminal or a server. Figure 1 The flowchart shown is for a distributed reinforcement learning optimization method for communication resource-constrained scenarios. The processing flow of this method may include the following steps:

[0039] S1. Establish a distributed reinforcement learning model including a central controller and multiple reinforcement learning nodes, determine the comprehensive reward function of each reinforcement learning node, construct a global objective function based on the comprehensive reward function of all reinforcement learning nodes, initialize the global policy parameters based on the global objective function, and iteratively update the global objective function during training to optimize the global objective function.

[0040] Optionally, the establishment in S1 includes a distributed reinforcement learning model with a central controller and multiple reinforcement learning nodes, determining the comprehensive reward function for each reinforcement learning node, including: For a distributed reinforcement learning system including a central controller and multiple reinforcement learning nodes, each reinforcement learning node is modeled as a reinforcement learning agent with a parameterized policy function, and a comprehensive reward function for each reinforcement learning node is defined according to the policy parameters. The comprehensive reward function considers indicators such as node information content, information timeliness, information similarity, training communication latency, and training communication energy consumption.

[0041] One feasible implementation method is, for example Figure 2 As shown, based on the business scenarios of communication networks or edge computing systems, a distributed reinforcement learning model consisting of several learning nodes and a central controller is established. The state space, action space, and global objective function with node information content, information timeliness, information similarity, training communication latency, and training communication energy consumption as the core are determined for each learning node.

[0042] Specifically, let the set of learning nodes be denoted as Each learning node is modeled as a reinforcement learning agent, using a parameterized policy function. Indicates the state Select action The probability of , denoted by the policy parameter as ; for the first The definition of a reinforcement learning node includes a single-step comprehensive reward metric that incorporates factors such as node information content, information timeliness, information similarity, training communication latency, and training communication energy consumption. Construct a long-term discount cumulative return and sum the long-term discount cumulative returns of all learning nodes as the global objective function. The central controller adjusts policy parameters using methods such as policy gradient. Perform iterative updates to maximize the global reward and determine the globally optimal policy parameters: (1) S2. Each reinforcement learning node interacts with the local environment based on global policy parameters to obtain a state-action-reward sequence. The local gradient estimation vector is calculated using the policy gradient method based on the state-action-reward sequence. The gradient information content index of the reinforcement learning node is calculated based on the norm of the local gradient estimation vector.

[0043] In one feasible implementation, in each training round, each learning node interacts with the local environment based on the current policy parameters, collects the state-action-reward sequence, estimates the corresponding policy gradient locally using the policy gradient method, obtains the local gradient estimation vector, and calculates the gradient information content index of the learning node based on the norm of the local gradient estimation vector.

[0044] Specifically, the calculation of the gradient information content index includes: for the first... The reinforcement learning node at the _ Local gradient estimation vector in round training The squared norm of gradient is used as an indicator of gradient information content. This characterizes the strength of the learning node's contribution to the global policy update in the current round: (2) S3. Maintain the information age for each reinforcement learning node, construct a time-sensitive weight function based on the information age, and multiply the gradient information content index with the time-sensitive weight function to obtain the weighted information index.

[0045] Optionally, S3 maintains an informational age for each reinforcement learning node, including: Determine the position of each reinforcement learning node at the 1st epoch. Whether it is selected to participate in the local gradient estimation vector upload during the training round; if selected, then in the... During each training round, the information age of the reinforcement learning node is reset to 1; if it is not selected, then in the next training round... During each training round, the information age of the reinforcement learning node is updated to the information age plus 1.

[0046] In one feasible implementation, to characterize the freshness of the information uploaded by the learning nodes, the number of training rounds since the last participation of each learning node in global aggregation is maintained as the information age. A timeliness weight function is constructed based on the information age, and the gradient information quantity is weighted with the information age to obtain a weighted information index that reflects the gradient strength and information timeliness.

[0047] Specifically, the calculation of information age and weighted information metric includes: for each learning node Maintain information age This represents the number of training rounds since the last time the learning node participated in global gradient aggregation. When the learning node... In the When a round is selected to participate in gradient upload, in the first round... In the first round, reset its information age to 1; otherwise, in the second round... The information age is updated in ascending order of round number. The timeliness weighting function is defined based on the information age. :

[0048] (3) In the formula, This represents the preset attenuation coefficient. Multiplying the gradient information content index by the time-dependent weighting function yields the weighted information index. This gives learning nodes with a large amount of information that have not participated in aggregation for a long time a higher priority for selection in subsequent rounds:

[0049] (4) S4. Calculate the difference between the local gradient estimation vectors of any two reinforcement learning nodes, define the gradient similarity coefficient based on the difference between the local gradient estimation vectors, and obtain the redundancy index between nodes based on the gradient similarity coefficient.

[0050] In one feasible implementation, in order to reduce information redundancy caused by multiple learning nodes uploading highly similar gradients, the gradient similarity or redundancy index between any two learning nodes is calculated based on the difference between the local gradient estimation vectors of different learning nodes.

[0051] Specifically, the calculation of the redundancy index between nodes includes: for any two learning nodes and Based on the difference in their local gradient estimation vectors, a gradient similarity coefficient is defined. The greater the similarity, the higher the redundancy of the uploaded information between the two entities. (5) In the formula, Indicates the first The reinforcement learning node and the first The reinforcement learning node at the _ Gradient similarity coefficients in rounds of training Indicates the first The reinforcement learning node at the _ The local gradient estimation vector during training rounds. Indicates the first The reinforcement learning node at the _ The local gradient estimation vector during training rounds. This indicates the preset smoothing parameters.

[0052] Gradient similarity coefficient is used to measure the degree of similarity in gradient directions between any two learning nodes. and When the differences are small, the similarity coefficient is large, indicating that the information between the two is highly redundant; when and When the differences are large, the similarity coefficient is small, which is beneficial to improving the diversity of the overall gradient direction.

[0053] S5. Calculate the training communication latency and training communication energy consumption based on the link rate, transmission power, and gradient data volume between each reinforcement learning node and the central controller. Then, linearly combine the training communication latency and training communication energy consumption to obtain the communication resource overhead index.

[0054] Optionally, in S5, the training communication latency and training communication energy consumption are calculated based on the link rate, transmit power, and gradient data volume between each reinforcement learning node and the central controller, including: The training communication latency is calculated based on the link rate and gradient data volume between each reinforcement learning node and the central controller.

[0055] The training communication energy consumption is calculated based on the training communication latency and the transmit power between each reinforcement learning node and the central controller.

[0056] In one feasible implementation, in scenarios with limited communication resources, a learning node's participation in a single gradient upload incurs uplink latency and communication energy consumption. Considering the characteristic that training communication and service communication share a physical link, the uplink latency and communication energy consumption introduced by the node in a single upload are determined based on the link rate, transmission power, and gradient data volume between the learning node and the central controller. These two factors are then linearly combined according to preset weights to obtain a communication resource overhead index. This index is then normalized or nonlinearly mapped to obtain a normalized communication cost index used for constructing the objective function.

[0057] Specifically, the calculation of the communication resource overhead index includes: assuming the first... In the first round of training The number of bits of gradient uploaded by each reinforcement learning node is The effective uplink transmission rate between the controller and the central controller is Then the training communication delay : (6) The transmission power during the upload process is Training communication energy consumption: (7) Latency and energy consumption are weighted according to preset coefficients. and By performing a linear combination, the communication resource overhead index is obtained. : (8) S6. Construct a sub-model-like objective function based on the weighted information index, inter-node redundancy index, and communication resource overhead index. Under the given maximum number of participating nodes and communication resource budget constraints, select the set of reinforcement learning nodes to participate in uploading based on the sub-model-like objective function and sub-model optimization algorithm, and upload the local gradient estimation vector of the selected reinforcement learning node set to the central controller.

[0058] In one feasible implementation, an objective function is constructed based on weighted information metrics, inter-node redundancy metrics, and communication resource overhead metrics, including an information quantity term, a redundancy penalty term, and a communication overhead penalty term. Under the communication budget constraint of the number of participating learning nodes, a submodular optimization algorithm is used to select the set of learning nodes participating in gradient upload in this round. For example, a greedy algorithm is used to iteratively solve the objective function of the set, selecting the set of learning nodes constituting the training communication in this round from all candidate learning nodes.

[0059] Specifically, for any set of candidate nodes Based on the aforementioned weighted information metrics, gradient redundancy, and communication cost, this invention constructs an objective function with sub-model characteristics: (9) In the formula, This represents the objective function with submodel characteristics. Represents any set of candidate nodes. Represents the set of reinforcement learning nodes. This represents a weighted information indicator. Indicates training rounds, Indicates a similarity penalty. Represents the gradient similarity coefficient. The communication cost weight parameter represents the communication cost. This represents the communication resource overhead metric. The first term measures the total amount of effective information provided by the learning nodes within the set; the second term penalizes the redundancy of the gradient directions of the learning nodes within the set; and the third term measures the total communication resource overhead of the set. and The similarity penalty and communication cost weight parameters can be configured according to different application scenarios. Through the above construction, the objective function exhibits diminishing returns as the node set expands, thus satisfying the requirements of the sub-modulus function.

[0060] Optionally, in S6, given the maximum number of participating nodes and communication resource budget constraints, the set of reinforcement learning nodes participating in uploading is selected based on a sub-model-like objective function and a sub-model optimization algorithm, including: In one feasible implementation, in each training round, the central controller determines the objective function... Given a maximum limit on the number of participating nodes, candidate learning nodes are iteratively selected starting from an empty set. A greedy submodular optimization algorithm is used to obtain the set of learning nodes participating in the upload in this round. Specifically, each time, the candidate node that maximizes the gain of the objective function of the set is added to the current set, until the addition of any unselected learning node no longer improves the objective function or the communication budget is exhausted, thus obtaining the target learning node set for this round that satisfies the constraints.

[0061] S7. The central controller aggregates the received local gradient estimation vectors to obtain the global gradient estimation vector, updates the global optimal policy parameters based on the global gradient estimation vector, and distributes the updated global optimal policy parameters to each reinforcement learning node to achieve distributed reinforcement learning communication overhead optimization for scenarios with limited communication resources.

[0062] In one feasible implementation, the selected learning node uploads its local gradient estimate to the central controller. The central controller aggregates the received local gradients to obtain a global gradient estimate, updates the global policy parameters based on the global gradient estimate, and distributes the updated global policy parameters to each learning node for the next round of local interaction and gradient calculation. This process is repeated iteratively until the algorithm converges or reaches a preset number of rounds.

[0063] Furthermore, in the above method, the objective function can be adjusted according to communication constraints, energy constraints, or latency constraints in different scenarios to adapt to various communication resource-constrained scenarios such as cellular networks, edge computing networks, and drone networks; the local environment of the learning node can be a mobile edge computing task scheduling environment, a wireless resource allocation environment, or other environments that require decision optimization through distributed reinforcement learning.

[0064] To address the problems of existing technologies, this invention combines distributed reinforcement learning with communication overhead optimization techniques, providing an effective solution for scenarios with limited communication resources. On one hand, by executing policy gradient reinforcement learning algorithms locally on each distributed node and measuring local gradient information content, information timeliness, and inter-node gradient redundancy, representative nodes contributing significantly to the current global policy can be identified while ensuring learning performance. On the other hand, through node selection, communication round control, and modeling communication resource constraints related to link latency and energy consumption, the number of nodes participating in uplink communication and the communication load can be limited in each training round, thereby reducing training communication overhead and alleviating link congestion pressure under limited bandwidth conditions. Furthermore, by utilizing selection mechanisms with diminishing marginal returns, such as sub-modulus optimization, multi-dimensional indicators such as gradient information content, information timeliness, and communication resource consumption are unified into a single objective framework, effectively improving communication resource utilization efficiency while ensuring the convergence performance of distributed reinforcement learning.

[0065] For example, the distributed reinforcement learning communication overhead optimization method for communication resource-constrained scenarios according to embodiments of the present invention is applied to, for example, Figure 3 The multi-user task offloading model is shown. The network includes a cloud and multiple edge computing domains. Each edge computing domain can consist of one or more satellites, relay drones and connected edge servers, or it can consist of a ground base station and its connected edge servers. The specific form is not limited.

[0066] In this embodiment, there are multiple ground user terminals in the network, which are used to generate tasks that require computation and processing. Each user task can be divided according to a partial offloading strategy: a portion of the task bits are computed on the local terminal, and another portion of the task bits are offloaded to the edge computing domain or cloud server via a wireless link for computation.

[0067] In this embodiment of the invention, to achieve intelligent decision-making regarding the aforementioned task offloading and communication transmission processes, each edge computing domain is treated as a learning node and configured with a local reinforcement learning controller. The local reinforcement learning node outputs task offloading and resource allocation strategies based on the observed network state.

[0068] In this embodiment, the state of each edge computing domain at the discrete decision time is represented as follows: The status includes the queue length, available computing resources, wireless link rate, link occupancy, and historical task latency and energy consumption statistics for this domain; task offloading and resource allocation strategies are represented as actions. This includes options for "local computing, offloading to a local edge server, or offloading to another edge domain or the cloud via a relay," along with the corresponding offloading ratio. The environment returns an immediate report based on the stated action. This is to reflect indicators such as overall task latency and energy consumption.

[0069] Let the strategy parameter be... The strategy is In this embodiment of the invention, each edge computing domain Corresponding to a local loss function The global objective function is referred to in equation (1) above.

[0070] During training, each edge computing domain collects interaction trajectories locally, calculates local gradient estimates based on the policy gradient method, and periodically synchronizes parameters with the central controller, thus forming a distributed reinforcement learning training process. In traditional distributed reinforcement learning schemes, each edge computing domain uploads its local gradient or policy parameters to the central controller in each training round. Training communication imposes additional overhead on the underlying wireless and backhaul links. Especially in scenarios with high task load and limited communication resources, this can compete with the offloading and transmission of business tasks for resources, leading to a deterioration in overall latency and energy consumption.

[0071] To this end, embodiments of the present invention introduce the following metrics during the training phase to characterize the training information value, information timeliness, and physical overhead of training communication in each edge computing domain: In the In the first round of training, Each edge computing domain calculates its local gradient estimate based on locally collected task offloading interaction data. In this embodiment, the gradient information content index can be defined as Equation (2) above, which is used to characterize the contribution intensity of the edge domain to the global policy update in the current round.

[0072] To reflect the freshness of information uploaded from different edge computing domains, this embodiment maintains an information age for each edge computing domain. , representing the number of training rounds since the domain last participated in global gradient aggregation. If the domain is selected to participate in the upload in the current round, the information age is reset to 1 in the next round; if it is not selected, the information age is incremented in the next round. The timeliness weight function is defined according to the information age, referring to the above formula (3). Thus, a weighted information index combining information content and timeliness is obtained.

[0073] In this embodiment, to avoid redundancy caused by uploading highly similar gradient vectors from multiple edge domains and consuming valuable training communication resources, a similarity coefficient is defined based on the difference between the local gradients of different edge domains. It is used to measure the similarity of the gradient directions of any two nodes.

[0074] In this embodiment of the invention, training communication and task offloading share the same or partially overlapping physical links. While providing transmission services for the task, the edge computing domain also needs to upload local gradients to the central controller. Therefore, the first step can be calculated based on parameters such as the uplink rate, bandwidth occupied, transmission power, and training message size between the edge computing domain and the central controller. In the first round of training The communication latency and energy consumption of each edge computing domain during a single training upload are combined into a communication resource overhead index. .

[0075] Assume that the number of bits of its uploaded gradient is a fixed value. The effective upload speed between the center and the data center is Then the additional latency caused by training communication can be expressed as equation (6) above. If the transmit power during uploading is The corresponding training communication energy consumption is given by equation (7) above. In order to make it easier to compare the training communication burden of different edge domains when selecting nodes, this embodiment converts latency and energy consumption into a unified communication overhead index.

[0076] In this embodiment, before each round of distributed reinforcement learning training, the central controller, based on the aforementioned metrics reported by each edge computing domain, targets the candidate set. Construct the objective function as shown in Equation (9) above. The first term in Equation (9) is used to measure the total amount of effective information provided by the selected edge computing domain set in the current training round; the second term is the gradient redundancy penalty term, which is used to encourage the diversity of gradient directions of the selected edge domains; the third term is the communication cost penalty term, which is used to control the degree of occupation of training communication on the underlying link and task offloading process.

[0077] Given the training communication resource budget constraint, this embodiment employs a greedy submodular optimization algorithm to optimize the objective function. By iteratively solving the problem, we obtain the set of distributed reinforcement learning nodes that participated in uploading during this round of training. The selected distributed reinforcement learning node will perform local gradient estimation. The gradients are uploaded to the central controller, which then performs weighted aggregation on the received gradients and updates the global policy parameters. Then, the updated strategy parameters are distributed to all edge computing domains for subsequent task offloading decisions.

[0078] Through the above process, the embodiments of the present invention achieve a comprehensive consideration of training communication and task offloading service communication: on the one hand, at the task level, the edge computing domain performs local computation and offloading decisions for user tasks based on the current policy parameters; on the other hand, at the training level, by selecting sub-modules for the edge computing domains participating in training communication, the occupation of the underlying link by training communication is effectively reduced, thereby reducing the overall task latency and energy consumption.

[0079] This invention constructs an environment for joint modeling of multi-edge domain task offloading and distributed reinforcement learning, representing the business processing latency and energy consumption of user tasks as a comprehensive objective function for task offloading. Each edge domain interacts with the environment based on current policy parameters, samples the local policy gradient, and obtains a weighted information index reflecting gradient strength and information timeliness. The redundancy between edge domains is measured through gradient differences. A training communication resource overhead index is introduced, constructing a sub-modular function that simultaneously reflects information content, information timeliness, gradient redundancy, and communication overhead. Under constraints on the number of participating nodes or communication budget, a greedy algorithm selects a subset of edge domains participating in gradient uploading in this round. The central controller aggregates the gradients uploaded from the selected edge domains and updates the global policy parameters.

[0080] In this embodiment of the invention, multiple factors such as the gradient information content of learning nodes, information timeliness, redundancy between nodes, and communication resource overhead are considered simultaneously under a unified framework. Compared with the scheme that selects nodes based only on gradient size or a single communication indicator, it can more accurately depict the occupation of physical links by training communication and is more in line with the actual needs of scenarios with limited communication resources.

[0081] This invention constructs an objective function with sub-model characteristics and uses a low-complexity greedy algorithm for node selection. Under the premise of satisfying the constraints of the number of participating nodes or communication resource budget, it can obtain a near-optimal subset of nodes. While effectively reducing the number of training communications and the total communication overhead, it maintains full utilization of key gradient information.

[0082] The communication overhead optimization mechanism proposed in this invention can adaptively adjust the node selection results according to changes in network status, giving priority to learning nodes with rich and fresh information and representative gradients. This improves the efficiency of communication resource utilization while ensuring the convergence performance of distributed reinforcement learning and the quality of business services, and is applicable to various distributed intelligent control scenarios with limited communication resources.

[0083] Figure 4 This is a block diagram illustrating a distributed reinforcement learning optimization apparatus for communication resource-constrained scenarios, according to an exemplary embodiment. The apparatus is used in a distributed reinforcement learning optimization method for communication resource-constrained scenarios. (Refer to...) Figure 4The device includes a model building module 310, a gradient information content calculation module 320, a weighted information content calculation module 330, a node redundancy calculation module 340, a communication overhead evaluation module 350, a sub-model optimization module 360, and a policy update module 370. Among them:

[0084] The model building module 310 is used to build a distributed reinforcement learning model including a central controller and multiple reinforcement learning nodes, determine the comprehensive reward function of each reinforcement learning node, construct a global objective function based on the comprehensive reward function of all reinforcement learning nodes, and initialize global policy parameters based on the global objective function.

[0085] The gradient information content calculation module 320 is used to obtain the state-action-reward sequence by each reinforcement learning node interacting with the local environment based on the global policy parameters, to calculate the local gradient estimation vector using the policy gradient method based on the state-action-reward sequence, and to calculate the gradient information content index of the reinforcement learning node based on the norm of the local gradient estimation vector.

[0086] The weighted information index calculation module 330 is used to maintain the information age for each reinforcement learning node, construct a time-sensitive weight function based on the information age, and multiply the gradient information content index by the time-sensitive weight function to obtain the weighted information index.

[0087] The node redundancy index calculation module 340 is used to calculate the difference between the local gradient estimation vectors of any two reinforcement learning nodes, define the gradient similarity coefficient based on the difference between the local gradient estimation vectors, and obtain the node redundancy index based on the gradient similarity coefficient.

[0088] The communication overhead evaluation module 350 is used to calculate the training communication latency and training communication energy consumption based on the link rate, transmission power and gradient data volume between each reinforcement learning node and the central controller, and to obtain the communication resource overhead index by linearly combining the training communication latency and training communication energy consumption.

[0089] The sub-model optimization module 360 ​​is used to construct a sub-model-like objective function based on weighted information indicators, inter-node redundancy indicators, and communication resource overhead indicators. Under the given constraints of the maximum number of participating nodes and communication resource budget, it selects a set of reinforcement learning nodes to participate in uploading based on the sub-model-like objective function and the sub-model optimization algorithm, and uploads the local gradient estimation vector of the selected reinforcement learning node set to the central controller.

[0090] The policy update module 370 is used by the central controller to aggregate the received local gradient estimation vectors to obtain the global gradient estimation vector, update the global optimal policy parameters according to the global gradient estimation vector, and distribute the updated global optimal policy parameters to each reinforcement learning node, thereby realizing the optimization of communication overhead in distributed reinforcement learning for scenarios with limited communication resources.

[0091] In this embodiment of the invention, multiple factors such as the gradient information content of learning nodes, information timeliness, redundancy between nodes, and communication resource overhead are considered simultaneously under a unified framework. Compared with the scheme that selects nodes based only on gradient size or a single communication indicator, it can more accurately depict the occupation of physical links by training communication and is more in line with the actual needs of scenarios with limited communication resources.

[0092] This invention constructs an objective function with sub-model characteristics and uses a low-complexity greedy algorithm for node selection. Under the premise of satisfying the constraints of the number of participating nodes or communication resource budget, it can obtain a near-optimal subset of nodes. While effectively reducing the number of training communications and the total communication overhead, it maintains full utilization of key gradient information.

[0093] The communication overhead optimization mechanism proposed in this invention can adaptively adjust the node selection results according to changes in network status, giving priority to learning nodes with rich and fresh information and representative gradients. This improves the efficiency of communication resource utilization while ensuring the convergence performance of distributed reinforcement learning and the quality of business services, and is applicable to various distributed intelligent control scenarios with limited communication resources.

[0094] Figure 5 This is a schematic diagram of the structure of a distributed reinforcement learning communication overhead optimization device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the distributed reinforcement learning communication overhead optimization device may include the above-mentioned Figure 4 The illustrated distributed reinforcement learning optimization device is designed for scenarios with limited communication resources. Optionally, the distributed reinforcement learning communication overhead optimization device 410 may include a first processor 2001.

[0095] Optionally, the distributed reinforcement learning communication overhead optimization device 410 may also include a memory 2002 and a transceiver 2003.

[0096] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0097] The following is combined Figure 5 The various components of the distributed reinforcement learning communication overhead optimization device 410 are described in detail below: The first processor 2001 is the control center of the distributed reinforcement learning communication overhead optimization device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0098] Optionally, the first processor 2001 can perform various functions of the distributed reinforcement learning communication overhead optimization device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0099] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0100] In a specific implementation, as one example, the distributed reinforcement learning communication overhead optimization device 410 may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0101] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0102] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and its interface circuitry may be optimized through distributed reinforcement learning communication overhead. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0103] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0104] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0105] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and the interface circuit of the device 410 can be optimized through distributed reinforcement learning communication overhead. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0106] It should be noted that, Figure 5 The structure of the distributed reinforcement learning communication overhead optimization device 410 shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0107] Furthermore, the technical effect of the distributed reinforcement learning communication overhead optimization device 410 can be referred to the technical effect of the distributed reinforcement learning communication overhead optimization method for communication resource-constrained scenarios described in the above method embodiments, and will not be repeated here.

[0108] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0109] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0110] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0111] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0112] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0113] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed reinforcement learning optimization method for a communication resource limited scenario, characterized in that, The method comprises: S1, establishing a distributed reinforcement learning model comprising a central controller and a plurality of reinforcement learning nodes, determining a comprehensive reward function of each reinforcement learning node, constructing a global objective function according to the comprehensive reward functions of all reinforcement learning nodes, and initializing global policy parameters according to the global objective function; S2, each reinforcement learning node interacts with the local environment based on the global policy parameters to obtain a state-action-reward sequence, and calculates a local gradient estimation vector by using a policy gradient method according to the state-action-reward sequence, and calculates a gradient information quantity index of the reinforcement learning node according to the norm of the local gradient estimation vector; S3, maintaining an information age for each reinforcement learning node, constructing an aging weight function according to the information age, multiplying the gradient information quantity index and the aging weight function to obtain a weighted information index; S4, calculating the difference between the local gradient estimation vectors of any two reinforcement learning nodes, defining a gradient similarity coefficient according to the difference between the local gradient estimation vectors, and obtaining an inter-node redundancy index according to the gradient similarity coefficient; S5, calculating the training communication delay and the training communication energy consumption according to the link rate, the transmission power and the gradient data volume between each reinforcement learning node and the central controller, and linearly combining the training communication delay and the training communication energy consumption to obtain a communication resource overhead index; S6, constructing a target function with a submodular characteristic according to the weighted information index, the inter-node redundancy index and the communication resource overhead index, selecting a reinforcement learning node set participating in uploading according to the target function with the submodular characteristic and a submodular optimization algorithm under the condition of given maximum number of participating nodes and communication resource budget constraint, and uploading the local gradient estimation vectors of the selected reinforcement learning node set to the central controller; S7, the central controller aggregates the received local gradient estimation vectors to obtain a global gradient estimation vector, updates the global optimal policy parameters according to the global gradient estimation vector, and distributes the updated global optimal policy parameters to each reinforcement learning node, so as to realize the distributed reinforcement learning communication overhead optimization for the communication resource limited scene.

2. The method of claim 1, wherein, The establishment of the distributed reinforcement learning model comprising the central controller and the plurality of reinforcement learning nodes in S1 comprises: For a distributed reinforcement learning system comprising a central controller and a plurality of reinforcement learning nodes, each reinforcement learning node is modeled as a reinforcement learning agent using a parameterized policy function, and a comprehensive reward function of each reinforcement learning node is defined according to the policy parameters; wherein the comprehensive reward function comprises node information quantity, information aging, information similarity, training communication delay and training communication energy consumption and the like.

3. The method of claim 1, wherein, The information age for each reinforcement learning node in S3 comprises: Determine the position of each reinforcement learning node at the 1st epoch. Whether it is selected to participate in the local gradient estimation vector upload during the training round; if selected, then in the... During each training round, the information age of the reinforcement learning node is reset to 1; if it is not selected, then in the next training round... During each training round, the information age of the reinforcement learning node is updated to the information age plus 1.

4. The method of claim 1, wherein, The gradient similarity coefficient in S4 is shown in the following formula (1): （1） wherein denotes the gradient similarity coefficient of the i-th reinforcement learning node and the j-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the local gradient estimation vector of the i-th reinforcement learning node in the t-th training round, denotes the smoothing parameter.

5. The method of claim 1, wherein, The calculation of the training communication delay and the training communication energy consumption according to the link rate and the gradient data volume between each reinforcement learning node and the central controller in S5 comprises: The training communication delay is calculated according to the link rate and the gradient data volume between each reinforcement learning node and the central controller; The training communication energy consumption is calculated according to the training communication time delay and the transmission power between each reinforcement learning node and the central controller.

6. The method of claim 1, wherein, The target function with the submodular property in S6 is shown in the following formula (2): （2） wherein, denotes a target function with submodular properties, denotes an arbitrary candidate node set, denotes a reinforcement learning node set, denotes a weighted information metric, denotes a training round, denotes a similarity penalty, denotes a gradient similarity coefficient, denotes a communication cost weight parameter, denotes a communication resource overhead indicator.

7. The method of claim 1, wherein, The target function with the submodular property in S6 is shown in the following formula (2): Under the condition of the given maximum number of participating nodes, the set of reinforcement learning nodes participating in uploading is set as an empty set, the reinforcement learning node that maximizes the gain of the target function is iteratively selected, the selected reinforcement learning node is added to the set of reinforcement learning nodes participating in uploading, until the addition of any unselected reinforcement learning node no longer improves the gain of the target function or the given communication resource budget is exhausted, and the set of reinforcement learning nodes participating in uploading is obtained.

8. A distributed reinforcement learning optimization apparatus for a communication resource limited scenario, the distributed reinforcement learning optimization apparatus is configured to implement the distributed reinforcement learning optimization method for a communication resource limited scenario according to any one of claims 1-7, wherein, The device comprises: The model establishing module is configured to establish a distributed reinforcement learning model comprising a central controller and a plurality of reinforcement learning nodes, determine a comprehensive reward function of each reinforcement learning node, construct a global target function according to the comprehensive reward functions of all reinforcement learning nodes, and initialize global policy parameters according to the global target function; The gradient information quantity index calculation module is configured to obtain a state-action-reward sequence by allowing each reinforcement learning node to interact with a local environment based on the global policy parameters, calculate a local gradient estimation vector by using a policy gradient method according to the state-action-reward sequence, and calculate a gradient information quantity index of the reinforcement learning node according to the norm of the local gradient estimation vector; The weighted information index calculation module is configured to maintain an information age for each reinforcement learning node, construct an aging weight function according to the information age, multiply the gradient information quantity index by the aging weight function to obtain a weighted information index, and upload the weighted information index to the central controller; The inter-node redundancy index calculation module is configured to calculate the difference between the local gradient estimation vectors of any two reinforcement learning nodes, define a gradient similarity coefficient according to the difference between the local gradient estimation vectors, and obtain an inter-node redundancy index according to the gradient similarity coefficient; The communication overhead evaluation module is configured to calculate a training communication time delay and a training communication energy consumption according to the link rate, the transmission power and the gradient data volume between each reinforcement learning node and the central controller, and perform linear combination on the training communication time delay and the training communication energy consumption to obtain a communication resource overhead index; The submodular optimization module is configured to construct a target function with a submodular property according to the weighted information index, the inter-node redundancy index and the communication resource overhead index, select a set of reinforcement learning nodes participating in uploading according to the target function with the submodular property and a submodular optimization algorithm under the condition of the given maximum number of participating nodes and the communication resource budget constraint, and upload the local gradient estimation vectors of the selected set of reinforcement learning nodes to the central controller. The policy updating module is configured to aggregate the received local gradient estimation vectors to obtain a global gradient estimation vector, update the global optimal policy parameters according to the global gradient estimation vector, and distribute the updated global optimal policy parameters to each reinforcement learning node, so as to realize distributed reinforcement learning communication overhead optimization for a communication resource limited scene.

9. A distributed reinforcement learning communication overhead optimization device, characterized in that, The distributed reinforcement learning communication overhead optimization device comprises: a processor; a memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to implement the method according to any one of claims 1 to 7.

Images