A Distributed Computing Power Cooperative Allocation Method Based on Multi-Agent Reinforcement Learning

Abstract

This invention discloses a distributed computing power collaborative allocation method based on multi-agent reinforcement learning, comprising the following steps: Step 1: Collect task queues, computing resources, and link status to generate node observations; Step 2: Output resource allocation decisions based on node observations and constrain them within the feasible region; Step 3: Map decision variables to dual variables and maintain shared copy and index relationships; Step 4: Calculate dual differences and compress them before sending them to neighboring nodes; Step 5: Receive compressed differences and cumulatively update the shared copies; Step 6: Iteratively update dual variables according to mixed weights and consistency; Step 7: Obtain updated decisions through inverse mapping while satisfying the feasible region; Step 8: Execute offloading and computing power allocation and update policy parameters with rewards. This invention reduces communication load and improves the stability of collaborative allocation.

Description

Technical Field

[0001] This invention relates to the field of distributed computing power resource management technology, and in particular to a distributed computing power collaborative allocation method based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous development of computing models such as cloud computing, edge computing, and fog computing, distributed computing networks have gradually become an important infrastructure supporting large-scale task processing and intelligent applications. In distributed computing networks, tasks usually arrive at each computing node dynamically. Different nodes have significant differences in computing resources, storage resources, and network link conditions. A single node often cannot complete all task processing independently on its own. Therefore, how to efficiently offload tasks and allocate computing power among multiple computing nodes, make full use of distributed computing resources, reduce task processing latency, and reduce resource waste has become a key research problem in the field of distributed computing resource management technology. In existing technologies, common distributed computing scheduling methods include heuristic rule-based scheduling methods, centralized or distributed scheduling methods based on optimization models, and adaptive scheduling methods based on machine learning.

[0003] In rule-based or traditional optimization model-based collaborative computing power allocation schemes, fixed scheduling rules or precise mathematical models are usually required in advance, and decisions are made based on global or near-global information. These methods are effective when the network scale is small or the environment changes slowly, but in scenarios where task arrival is highly random and the states of computing power nodes and links change frequently, it is difficult to adapt to environmental changes in a timely manner, which can easily lead to a mismatch between resource allocation results and actual operating status. At the same time, centralized optimization methods often require uploading the state information of each computing power node to the central node for unified calculation, which not only brings large communication overhead but also poses a single point of failure risk. While some distributed optimization methods reduce the pressure of centralized computing, under the constraint of multi-node collaboration, they still need to frequently exchange all states or decision variables, resulting in high communication load and making it difficult to operate stably for a long time in bandwidth-limited distributed computing power networks.

[0004] In recent years, reinforcement learning methods have been gradually introduced into the field of distributed computing power scheduling and task offloading. By learning the mapping relationship between the environment state and scheduling decisions, adaptive optimization of dynamic environments can be achieved. However, existing reinforcement learning-based schemes mostly adopt single-agent or weakly coupled multi-agent structures. Each computing node often updates its policy independently or only exchanges a small amount of state information, making it difficult to achieve stable and consistent collaborative decision-making under multiple coupled conditions such as resource capacity constraints, link capacity constraints, and task constraints. In addition, when introducing consistency or collaborative mechanisms, some multi-agent reinforcement learning methods usually need to frequently transmit all parameters, all dual variables, or gradient information between nodes, resulting in a significant increase in communication overhead. When using compressed communication, collaborative decision drift is easily caused by the accumulation of compression errors, thus affecting the feasibility and stability of resource allocation results.

[0005] Therefore, how to provide a distributed computing power collaborative allocation method based on multi-agent reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a distributed computing power collaborative allocation method based on multi-agent reinforcement learning. This invention introduces a multi-agent reinforcement learning decision-making mechanism and an improved CHOCO-Gossip consistency algorithm into a distributed computing power network to achieve collaborative resource allocation among computing power nodes under complex resource and link constraints. By mapping resource allocation decisions to a dual variable space and using differential compression and shared copy cumulative update methods for neighborhood consistency communication, the invention reduces communication overhead while maintaining the stability and feasibility of collaborative decision-making. It has the advantages of strong adaptability, low communication load, and high stability of collaborative allocation.

[0007] A distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to an embodiment of the present invention includes the following steps:

[0008] Step 1: Configure a reinforcement learning decision unit for each computing node in the distributed computing power network, collect task queue status, available computing power resource status, link status, and generate node observations;

[0009] Step 2: The reinforcement learning decision unit outputs resource allocation decision variables based on node observations, and restricts the resource allocation decision variables to the feasible region;

[0010] Step 3: Map resource allocation decision variables to dual variables, maintain a shared copy of the dual variables for computing nodes, and establish a relationship between adjacent computing node identifiers and shared copy indexes;

[0011] Step 4: The improved CHOCO-Gossip uniformity algorithm is used to calculate the dual variable and the dual variable shared copy difference vector. The compression operator is applied to the difference vector to generate compressed difference information, and the compressed difference information is sent to the adjacent computing power nodes.

[0012] Step 5: The computing power node receives the compressed differential information sent by the adjacent computing power node, and performs cumulative update on the shared copy of the dual variable according to the shared copy index relationship;

[0013] Step 6: Perform a consistent iterative update on the dual variables according to the preset mixed weights to obtain the updated dual variables;

[0014] Step 7: Inversely map the updated dual variable to the updated resource allocation decision variable, and update the resource allocation decision variable to satisfy the feasible region;

[0015] Step 8: Based on the updated resource allocation decision variables, perform task unloading and computing power allocation, calculate the reward, and update the reinforcement learning strategy parameters.

[0016] Optionally, step one specifically includes:

[0017] In a distributed computing power network, each computing power node is configured with a reinforcement learning decision unit and a sampling period is set;

[0018] At each sampling moment, the task queue status, available computing power resource status, and link status are collected. The task queue status includes the task queue length, task arrival rate, and average task deadline. The available computing power resource status includes the available computing power capacity and available memory capacity. The link status includes the available bandwidth, round-trip latency, and packet loss rate.

[0019] Perform normalization processing on the task queue status, available computing resources status, and link status;

[0020] The normalized task queue status, normalized available computing power resource status, and normalized link status are concatenated in a fixed concatenation order to generate node observations, which are then stored in association with the computing power nodes.

[0021] Optionally, step two specifically includes:

[0022] The reinforcement learning decision-making unit calculates resource allocation decision variables based on node observations and reinforcement learning strategy parameters;

[0023] The resource allocation decision variables consist of task unloading decision variables and computing power allocation decision variables. The task unloading decision variables are used to determine the unloading ratio of tasks between computing power nodes and adjacent computing power nodes, while the computing power allocation decision variables are used to determine the computing power capacity allocation ratio of tasks on computing power nodes.

[0024] Construct a feasible region, which includes resource capacity constraints, link capacity constraints, and task constraints. Resource capacity constraints limit the upper bound of computing power allocation decision variables based on the available computing power resource status. Link capacity constraints limit the upper bound of bandwidth occupancy of task unloading decision variables based on link status. Task constraints limit the value range of task unloading decision variables based on task queue status.

[0025] The feasible region constraint processing is performed on the resource allocation decision variables. The feasible region constraint processing includes boundary pruning and normalization processing. Boundary pruning restricts the components of the resource allocation decision variables to a range of values, while normalization ensures that the resource allocation decision variables meet the resource capacity constraints and link capacity constraints, thus obtaining resource allocation decision variables that satisfy the feasible region.

[0026] Optionally, step three specifically includes:

[0027] Based on the resource allocation decision variables and the feasible region, dual variables are constructed. The dual variables include the dual components corresponding to the resource capacity constraints, the dual components corresponding to the link capacity constraints, and the dual components corresponding to the task constraints.

[0028] A shared copy of dual variables is maintained for each computing power node. The shared copy of dual variables includes the shared copy of dual variables corresponding to the computing power node and the shared copy of dual variables corresponding to the identifiers of adjacent computing power nodes. The shared copy of dual variables has the same dimension as the dual variables.

[0029] Establish a relationship between adjacent computing node identifiers and shared copy indexes. The shared copy index relationship records the mapping relationship between adjacent computing node identifiers and the storage location of the shared copy of the dual variable.

[0030] Optionally, step four specifically includes:

[0031] An improved CHOCO-Gossip unification algorithm is used to generate compressed differential information. The improved CHOCO-Gossip unification algorithm maintains a shared copy of the dual variables on the computing node side and uses the shared copy of the dual variables as the differential benchmark.

[0032] At the sampling time, the computing node reads the shared copy of the dual variable and the dual variable corresponding to the computing node, and calculates the difference vector according to the dimension of the dual variable;

[0033] A compression operator is applied to the difference vector to generate compressed difference information. The compression operator performs sparsification on the difference vector and retains the non-zero components. The compressed difference information includes a set of non-zero component values ​​and a set of non-zero component indices. The set of non-zero component indices identifies the position of the set of non-zero component values ​​in the dual variable dimension.

[0034] The data transmission unit is encapsulated based on the neighboring computing power node identifier and the shared copy index relationship. The data transmission unit includes the computing power node identifier, the sampling time identifier, the non-zero component index set, and the non-zero component value set. The computing power node sends the data transmission unit to the neighboring computing power node.

[0035] Optionally, step five specifically includes:

[0036] The computing power node obtains the computing power node identifier, sampling time identifier, non-zero component index set, and non-zero component value set based on the transmission data units sent by neighboring computing power nodes;

[0037] The computing power node determines the shared copy storage location of the dual variable corresponding to the adjacent computing power node identifier based on the relationship between the adjacent computing power node identifier and the shared copy index;

[0038] The computing node traverses the non-zero component index set in the index order of the non-zero component index set, uses the non-zero component index as the dual variable dimension position index, and uses the non-zero component value with the same index in the non-zero component value set as the accumulation amount to perform the accumulation update on the dimension component corresponding to the dual variable shared copy in the dual variable shared copy storage location.

[0039] The computing nodes will associate the accumulated and updated shared copies of the dual variables with the sampling time identifier and store them in the shared copy storage area of ​​the dual variables.

[0040] Optionally, step six specifically includes:

[0041] At the sampling time, the dual variable is obtained, and the dual variable shared copy corresponding to the computing power node and the dual variable shared copy corresponding to the adjacent computing power node identifier are obtained from the dual variable shared copy storage area. The dual variable shared copy corresponding to the adjacent computing power node identifier is determined according to the index relationship between the adjacent computing power node identifier and the shared copy.

[0042] Obtain a preset mixed weight, which includes a self weight and adjacent weights. The self weight corresponds to the computing power node, and the adjacent weights are arranged in the order of the adjacent computing power node identifier index. The preset mixed weight satisfies that the sum of the self weight and the adjacent weight is 1.

[0043] The dual variable is updated using a consistent iterative update based on a preset mixed weight. The consistent iterative update includes a weighted summation of the dual variable dimensions one by one. The component of the updated dual variable at any dimension is equal to the sum of its own weight multiplied by the component of the dual variable at the same dimension, plus the sum of the adjacent weights multiplied by the sum of the components of the shared copy of the dual variable corresponding to the adjacent computing node identifier at the same dimension. This results in the updated dual variable.

[0044] Optionally, step seven specifically includes:

[0045] The inverse mapping is performed based on the updated dual variable. The inverse mapping establishes a field correspondence based on the mapping relationship between the resource allocation decision variable and the dual variable, and transforms the updated dual variable into the updated resource allocation decision variable.

[0046] The updated resource allocation decision variables consist of updated task unloading decision variables and updated computing power allocation decision variables. The updated task unloading decision variables are arranged in the order of adjacent computing power node identifier index and are used to determine the unloading ratio of tasks between computing power nodes and adjacent computing power node sets. The updated computing power allocation decision variables are arranged in the computing power capacity dimension of available computing power resource status and are used to determine the computing power capacity allocation ratio of tasks on computing power nodes.

[0047] The updated resource allocation decision variables are subjected to feasible region restriction processing, which includes boundary pruning and normalization. Boundary pruning restricts the components of the updated resource allocation decision variables to the value range defined by the task constraints. Normalization ensures that the updated task unloading decision variables meet the link capacity constraints and the updated computing power allocation decision variables meet the resource capacity constraints, thus obtaining updated resource allocation decision variables that satisfy the feasible region.

[0048] Optionally, step eight specifically includes:

[0049] The task unloading execution volume and computing power allocation execution volume are determined based on the updated resource allocation decision variables. The task unloading execution volume is determined by the updated task unloading decision variables and the task queue status, while the computing power allocation execution volume is determined by the updated computing power allocation decision variables and the available computing power resource status.

[0050] Task unloading is performed based on the task unloading execution volume. Task unloading distributes tasks among computing nodes and sets of adjacent computing nodes according to the index order of adjacent computing node identifiers.

[0051] Computing power allocation is performed based on the amount of computing power allocated. Computing power allocation is performed on the task according to the computing power capacity dimension of available computing power resources.

[0052] Based on task unloading and computing power allocation, the task execution results are obtained and the reward is calculated. The reward is stored in association with the sampling time identifier. The reward is determined by the weighted sum of at least two of the following: task latency, resource consumption, and link occupation cost.

[0053] The reinforcement learning policy parameters are updated based on the node observations and rewards associated with the sampling time identifier. The updated reinforcement learning policy parameters are used to adjust the policy mapping relationship of the reinforcement learning decision unit in calculating resource allocation decision variables based on node observations.

[0054] The beneficial effects of this invention are:

[0055] Compared with existing distributed computing power scheduling and task offloading technologies, this invention configures reinforcement learning decision units for each computing node in the distributed computing power network and uses these decision units as intelligent agents to learn collaboratively. This enables the resource allocation process to adaptively adjust according to the dynamic changes in task queue status, computing power resource status, and link status. By constructing a feasible region that includes resource capacity constraints, link capacity constraints, and task constraints, and by continuously applying feasible region constraints during the generation and updating of resource allocation decision variables, this invention can effectively avoid the constraint violation problem caused by policy exploration in traditional reinforcement learning methods. Thus, it maintains the feasibility and stability of task offloading and computing power allocation results in complex multi-constraint scenarios, improves the overall utilization efficiency of distributed computing power resources, and reduces task processing latency.

[0056] Furthermore, this invention introduces an improved CHOCO-Gossip consistency mechanism into the dual variable space. By using a shared copy of the dual variables as the difference benchmark, compressing the difference vector, and implementing neighborhood collaboration through cumulative updates at the receiving end, it effectively reduces the amount of communication data between computing power nodes, lowers bandwidth consumption and synchronization overhead. Simultaneously, by combining preset mixed weights to perform consistent iterative updates on the dual variables, it enables multiple computing power nodes to achieve stable consistent collaboration with only local information exchange, suppresses the accumulation and propagation of compression errors, and enhances the convergence stability and robustness of multi-agent reinforcement learning strategies in distributed computing power networks, thereby improving the reliability and engineering practicality of overall computing power collaborative allocation. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart of a distributed computing power collaborative allocation method based on multi-agent reinforcement learning proposed in this invention;

[0059] Figure 2 This is a schematic diagram of the dual variable consistency collaborative communication of a distributed computing power collaborative allocation method based on multi-agent reinforcement learning proposed in this invention. Detailed Implementation

[0060] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0061] refer to Figures 1-2 A distributed computing power collaborative allocation method based on multi-agent reinforcement learning includes the following steps:

[0062] Step 1: Configure a reinforcement learning decision unit for each computing node in the distributed computing power network, collect task queue status, available computing power resource status, link status, and generate node observations;

[0063] Step 2: The reinforcement learning decision unit outputs resource allocation decision variables based on node observations, and restricts the resource allocation decision variables to the feasible region;

[0064] Step 3: Map resource allocation decision variables to dual variables, maintain a shared copy of the dual variables for computing nodes, and establish a relationship between adjacent computing node identifiers and shared copy indexes;

[0065] Step 4: The improved CHOCO-Gossip uniformity algorithm is used to calculate the dual variable and the dual variable shared copy difference vector. The compression operator is applied to the difference vector to generate compressed difference information, and the compressed difference information is sent to the adjacent computing power nodes.

[0066] Step 5: The computing power node receives the compressed differential information sent by the adjacent computing power node, and performs cumulative update on the shared copy of the dual variable according to the shared copy index relationship;

[0067] Step 6: Perform a consistent iterative update on the dual variables according to the preset mixed weights to obtain the updated dual variables;

[0068] Step 7: Inversely map the updated dual variable to the updated resource allocation decision variable, and update the resource allocation decision variable to satisfy the feasible region;

[0069] Step 8: Based on the updated resource allocation decision variables, perform task unloading and computing power allocation, calculate the reward, and update the reinforcement learning strategy parameters.

[0070] In this embodiment, step one specifically includes:

[0071] In a distributed computing power network, a reinforcement learning decision unit is configured for each computing power node and a sampling period is set. The reinforcement learning decision unit runs on the computing power node side and corresponds one-to-one with the computing power node. The sampling period is used to determine the sampling time sequence. The sampling time is triggered at equal intervals on the time axis by the sampling period. The reinforcement learning decision unit starts the state acquisition process and records the sampling timestamp at each sampling time.

[0072] At each sampling moment, the task queue status, available computing power resource status, and link status are collected. The task queue status is read from the computing power node task scheduling queue to form the task queue length, task arrival rate, and average task deadline time. The available computing power resource status is read from the computing power node resource management module to form the available computing power capacity and available memory capacity. The link status is read from the computing power node network monitoring module to form the available bandwidth, round-trip latency, and packet loss rate. The status collection results are written to the status buffer according to the sampling timestamp.

[0073] Normalization processing is performed on the task queue status, available computing power resource status, and link status. The normalization processing adopts a linear mapping method for each status component. The linear mapping method determines the mapping interval with a preset lower bound and a preset upper bound. When the value of the status component is lower than the preset lower bound, it participates in the mapping according to the preset lower bound. When the value of the status component is higher than the preset upper bound, it participates in the mapping according to the preset upper bound. The mapping result is written to the normalized status buffer and the sampling timestamp is retained.

[0074] The normalized task queue status, normalized available computing power resource status, and normalized link status are spliced ​​together in a fixed splicing order to generate node observations. The fixed splicing order is written into the configuration item during the initialization phase and remains consistent at each sampling time. The link status is arranged in the index order of adjacent computing power node identifiers during splicing. The node observations are associated with the computing power node identifiers and sampling timestamps and stored in the node observation storage area.

[0075] In this embodiment, step two specifically includes:

[0076] The reinforcement learning decision unit calculates resource allocation decision variables based on node observations and reinforcement learning policy parameters. Node observations are read at the sampling time and used as state variables for policy calculation. Reinforcement learning policy parameters are stored on the computing node side and used to determine policy mapping relationships. Policy calculation is completed at the sampling time and generates numerical results of resource allocation decision variables. Resource allocation decision variables are written to the decision cache area when policy calculation is completed.

[0077] The resource allocation decision variables consist of task unloading decision variables and computing power allocation decision variables. The task unloading decision variables are arranged according to the adjacent computing power node identifier index of the adjacent computing power node set and correspond to the unloading ratio of each adjacent computing power node. The computing power allocation decision variables are arranged according to the computing power capacity dimension of the available computing power resources of the computing power node and correspond to the allocation ratio of each computing power capacity dimension. The task unloading decision variables and computing power allocation decision variables are stored in the decision cache area using fixed fields in sequence.

[0078] Construct a feasible region, which includes resource capacity constraints, link capacity constraints, and task constraints. Resource capacity constraints are given by the available computing power resource status and are used to limit the upper bound of computing power allocation decision variables. Link capacity constraints are given by the link status and are used to limit the upper bound of task unloading decision variables at the link bandwidth occupancy level. Task constraints are given by the task queue status and are used to limit the value range of task unloading decision variables. The feasible region parameters are updated by the status acquisition results and written to the constraint parameter cache at the sampling time.

[0079] The feasible region constraint processing is performed on the resource allocation decision variables. The feasible region constraint processing includes boundary pruning and normalization processing. The boundary pruning processing performs lower bound pruning and upper bound pruning on each component of the task unloading decision variable and the computing power allocation decision variable respectively, and writes the pruned values ​​back to the decision cache area. The normalization processing is performed after the boundary pruning processing and performs proportional normalization on the task unloading decision variable to meet the link capacity constraint and on the computing power allocation decision variable to meet the resource capacity constraint. After the normalization processing is completed, the resource allocation decision variables that meet the feasible region are generated and written to the resource allocation decision variable storage area.

[0080] In this embodiment, step three specifically includes:

[0081] Based on the resource allocation decision variables and the feasible region, dual variables are constructed. The dual variables adopt a component representation method that corresponds one-to-one with the feasible region constraints. The dual components corresponding to the resource capacity constraints are arranged according to the computing power capacity dimension of the available computing power resource status. The dual components corresponding to the link capacity constraints are arranged according to the adjacent computing power node identifier index and correspond to the bandwidth dimension of the link status. The dual components corresponding to the task constraints are arranged according to the constraint dimension of the task queue status. The dual variables are jointly determined by the resource allocation decision variables and the feasible region constraint parameters at each sampling time and written into the dual variable storage area.

[0082] To maintain a shared copy of dual variables for computing power nodes, a shared copy storage area is established on the computing power node side and the storage partition is divided according to the identifier of adjacent computing power nodes. The shared copy of dual variables corresponding to the computing power node is stored in the local node partition, and the shared copy of dual variables corresponding to the identifier of adjacent computing power nodes is stored in the adjacent node partition. The shared copy of dual variables adopts the same dimension arrangement and field order as the dual variables. During initialization, the shared copy of dual variables corresponding to the computing power node is set as the initial value of the dual variables, and the shared copy of dual variables corresponding to the identifier of adjacent computing power nodes is set as the zero vector or the preset initial value.

[0083] Establish a relationship between adjacent computing power node identifiers and shared copy indexes. The shared copy index relationship is established on the computing power node side by creating an index table and recording the mapping relationship between adjacent computing power node identifiers and shared copy storage area partition numbers. The index table records the starting position and length of the dual variable dimension. The index table is used to locate the shared copy of the dual variable corresponding to the adjacent computing power node identifier and complete the read, write and update when receiving compressed differential information.

[0084] In this embodiment, step four specifically includes:

[0085] An improved CHOCO-Gossip uniformity algorithm is used to generate compressed differential information. The improved CHOCO-Gossip uniformity algorithm sets up a dual variable shared copy storage area on the computing node side and reads and writes the dual variable shared copy under the constraint of the shared copy index relationship. The dual variable shared copy is written with a preset initial value during the initialization phase and reads the dual variable shared copy as the differential reference at the sampling time.

[0086] At the sampling time, the computing power node reads the shared copy of the dual variable corresponding to the computing power node, calculates the difference vector components in sequence according to the dimensions of the dual variable, takes the dual variable component minus the shared copy component of the dual variable, writes the difference vector into the difference vector cache area and associates it with the computing power node identifier and the sampling time identifier.

[0087] The compression operator is applied to the difference vector to generate compressed difference information. The compression operator determines the non-zero components of the difference vector one by one and forms a set of non-zero component indices. The compression operator extracts the corresponding non-zero component values ​​according to the set of non-zero component indices and forms a set of non-zero component values. The set of non-zero component indices is represented by the dimension index of the dual variable. The set of non-zero component values ​​is arranged in the order of the index of the non-zero component indices and written into the compressed difference information buffer.

[0088] The data transmission unit is encapsulated based on the neighboring computing power node identifier and the shared copy index relationship. The data transmission unit includes the computing power node identifier, the sampling time identifier, the non-zero component index set, and the non-zero component value set. The computing power node determines the transmission target according to the neighboring computing power node identifier and sends the data transmission unit one by one.

[0089] This invention introduces an improved CHOCO-Gossip consistency algorithm into the dual variable space, employing a shared copy memory mechanism, a differential compression transmission mechanism, a shared copy accumulation update mechanism, and a hybrid weight consistency iteration mechanism. Combined with the relationship between adjacent computing node identifiers and shared copy indexes, it achieves stable consistency synchronization of dual variables within the neighborhood. This allows computing nodes to transmit only the compressed differential information of the differential vector in bandwidth-constrained scenarios and accumulate and recover the shared copy at the receiving end. Compared to the unimproved consistency scheme, this reduces the overhead of transmitting the full dual variable and repeated synchronization, lowers communication load, and suppresses synchronization drift caused by the direct accumulation of compression errors. Simultaneously, it maintains the traceability and consistency of the dual variable update process, thereby supporting the iterative update of resource allocation decision variables within the feasible domain through mapping and inverse mapping, and obtaining more stable collaborative allocation inputs during task unloading and computing power allocation execution.

[0090] In this embodiment, step five specifically includes:

[0091] The computing power node obtains the computing power node identifier, sampling time identifier, non-zero component index set, and non-zero component value set based on the data unit sent by the neighboring computing power nodes. The computing power node identifier is used to determine the neighboring computing power nodes of the data unit sent, the sampling time identifier is used to determine the sampling time corresponding to the non-zero component index set and the non-zero component value set, the non-zero component index set is used to identify the position of the non-zero component of the difference vector in the dual variable dimension, and the non-zero component value set is used to give the value of the non-zero component of the difference vector at the corresponding position of the non-zero component index set. The non-zero component value set and the non-zero component index set correspond one-to-one in terms of sequence number.

[0092] The computing power node determines the storage location of the shared copy of the dual variable corresponding to the adjacent computing power node identifier based on the relationship between the adjacent computing power node identifier and the shared copy index. The shared copy index records the mapping relationship between the adjacent computing power node identifier and the partition number of the shared copy storage area of ​​the dual variable in the index table. The partition number is retrieved from the index table through the adjacent computing power node identifier and the storage location of the shared copy storage area of ​​the dual variable is located. The storage location points to the shared copy vector of the dual variable corresponding to the adjacent computing power node identifier.

[0093] The computing node traverses the non-zero component index set in index order, using the non-zero component index as the dual variable dimension position index and the non-zero component value with the same index in the non-zero component value set as the accumulation amount. It performs accumulation update on the corresponding dimension component of the dual variable shared copy in the storage location of the dual variable shared copy. The accumulation update is performed once for each non-zero component index traversed. The calculation of the accumulation update is to add the accumulation amount to the component of the dual variable shared copy at the dual variable dimension position index and write the calculation result back to the dual variable shared copy vector as the updated component value. After the traversal is completed, the dual variable shared copy with the accumulation update is obtained.

[0094] The computing node associates the accumulated updated dual variable shared copy with the sampling time identifier and stores it in the dual variable shared copy storage area. The associated storage writes the sampling time identifier into the timestamp field of the dual variable shared copy storage area and keeps it in the same storage record as the dual variable shared copy vector. This is used to match the dual variable shared copy vector according to the sampling time identifier during the consistent iteration update.

[0095] In this embodiment, step six specifically includes:

[0096] At the sampling time, dual variables are obtained, and the shared copies of dual variables corresponding to the computing power node and the shared copies of dual variables corresponding to the identifiers of adjacent computing power nodes are obtained from the shared copy storage area of ​​dual variables. The shared copies of dual variables corresponding to the identifiers of adjacent computing power nodes are determined according to the index relationship between the identifiers of adjacent computing power nodes and the shared copy. The dual variables are arranged according to the dual variable dimension, and the shared copies of dual variables are arranged in the same dimension as the dual variables. The shared copy of dual variables corresponding to the computing power node is used to characterize the base vector of the shared copy memory mechanism on the computing power node side. The shared copy of dual variables corresponding to the identifiers of adjacent computing power nodes is used to characterize the shared copy vector maintained by the adjacent computing power nodes on the computing power node side. The determination based on the shared copy index relationship includes retrieving the partition number of the shared copy storage area of ​​dual variables by the identifier of adjacent computing power nodes in the index table and locating the shared copy vector of dual variables corresponding to the identifier of adjacent computing power nodes.

[0097] A preset mixed weight is obtained, which includes self-weight and adjacent weight. The self-weight corresponds to the computing power node, and the adjacent weight is arranged in the index order of the adjacent computing power node identifier. The preset mixed weight satisfies that the sum of the self-weight and the adjacent weight is 1. The self-weight is used to determine the retention ratio of the dual variable in the uniformized iterative update, and the adjacent weight is used to determine the fusion ratio of the shared copy of the dual variable corresponding to the adjacent computing power node identifier in the uniformized iterative update. The one-to-one correspondence between the adjacent weight and the index order of the adjacent computing power node identifier is used to ensure that the shared copy of the dual variable corresponding to the adjacent computing power node identifier and the adjacent weight maintain a consistent mapping relationship when performing dimension splicing and weighting operations. The sum of the self-weight and the adjacent weight is 1 to make the uniformized iterative update form a convex combination and keep the numerical scale of the updated dual variable stable.

[0098] The dual variable is updated using a consistent iterative update based on a preset mixed weight. The consistent iterative update includes a weighted summation of each dimension of the dual variable. The component of the updated dual variable at any dimension position is equal to the sum of the self weight multiplied by the component of the dual variable at the same dimension position, plus the sum of the adjacent weights multiplied by the component of the shared copy of the dual variable corresponding to the adjacent computing power node identifier at the same dimension position. The weighted summation is performed sequentially along the dimension position index of the dual variable. At any dimension position index, the product term of the self weight and the dual variable component is calculated first. Then, the product term of each adjacent weight and the corresponding shared copy component of the dual variable is calculated according to the adjacent computing power node identifier index order. The self weight product term is summed with all adjacent weight product terms to obtain the component value of the updated dual variable at the dimension position index. After all dimension position indices are calculated, the updated dual variable vector is formed and arranged according to the dual variable dimension.

[0099] In this embodiment, step seven specifically includes:

[0100] The inverse mapping is performed based on the updated dual variable. The inverse mapping establishes a field correspondence based on the mapping relationship between the resource allocation decision variable and the dual variable, and transforms the updated dual variable into the updated resource allocation decision variable. The field correspondence is used to limit the corresponding positions of the components of the updated dual variable and the components of the updated resource allocation decision variable. The field correspondence maintains the consistency between the dimensional arrangement of the dual variable and the field order of the resource allocation decision variable. The inverse mapping process locates the corresponding field of the updated resource allocation decision variable for each component of the updated dual variable according to the field correspondence and writes it into the corresponding field position, thereby forming a numerical vector representation of the updated resource allocation decision variable.

[0101] The updated resource allocation decision variables consist of updated task unloading decision variables and updated computing power allocation decision variables. The updated task unloading decision variables are arranged in the order of adjacent computing power node identifier index and are used to determine the unloading ratio of tasks between the computing power node and the set of adjacent computing power nodes. The updated computing power allocation decision variables are arranged in the computing power capacity dimension of available computing power resource status and are used to determine the computing power capacity allocation ratio of tasks on the computing power node. Each component of the updated task unloading decision variable corresponds one-to-one with an adjacent computing power node identifier index position, and the component value represents the unloading ratio towards the neighbor node corresponding to the adjacent computing power node identifier index position. Each component of the updated computing power allocation decision variable corresponds one-to-one with the computing power capacity dimension, and the component value represents the allocation ratio on the corresponding resource in the computing power capacity dimension. The updated resource allocation decision variables maintain a fixed arrangement order of the task unloading decision variable fields and the computing power allocation decision variable fields in the structure to ensure the matching relationship of the constraint components in the feasible domain restriction processing.

[0102] The updated resource allocation decision variables undergo feasible region constraint processing, which includes boundary pruning and normalization. Boundary pruning restricts the components of the updated resource allocation decision variables to a value range defined by the task constraints. Normalization ensures that the updated task unloading decision variables meet the link capacity constraints and the updated computing power allocation decision variables meet the resource capacity constraints, resulting in updated resource allocation decision variables that satisfy the feasible region. Specifically, boundary pruning performs lower and upper bound pruning on the components of the updated task unloading and updated computing power allocation decision variables, respectively, keeping the pruned components within the value range defined by the task constraints. Normalization is performed after boundary pruning. Normalization proportionally adjusts the updated task unloading decision variables according to the adjacent computing power node identifier index order, ensuring that the link bandwidth occupancy corresponding to the updated task unloading decision variables meets the link capacity constraints, and proportionally adjusts the updated computing power allocation decision variables according to the computing power capacity dimension, ensuring that the resource occupancy corresponding to the updated computing power allocation decision variables meets the resource capacity constraints. After normalization, updated resource allocation decision variables that satisfy the feasible region are formed.

[0103] In this embodiment, step eight specifically includes:

[0104] The task unloading execution volume and computing power allocation execution volume are determined based on the updated resource allocation decision variables. The task unloading execution volume is determined by the updated task unloading decision variables and the task queue status, while the computing power allocation execution volume is determined by the updated computing power allocation decision variables and the available computing power resource status. Specifically, the updated task unloading decision variables provide the unloading ratio in the order of adjacent computing power node identifier indexes, the task queue status provides the task queue length, task arrival rate, and average task deadline time, and the task unloading execution volume is calculated by combining the unprocessed task size represented by the task queue status and the unloading ratio represented by the updated task unloading decision variables. The updated computing power allocation decision variables provide the allocation ratio according to the computing power capacity dimension of the available computing power resource status, and the available computing power resource status provides the available computing power capacity and available memory capacity. The computing power allocation execution volume is calculated by combining the upper bound of available resources represented by the available computing power resource status and the allocation ratio represented by the updated computing power allocation decision variables. The task unloading execution volume and the computing power allocation execution volume are written into the execution volume cache and associated with the sampling time identifier.

[0105] Task unloading is performed based on the task unloading execution volume. Task unloading distributes tasks between computing nodes and sets of adjacent computing nodes according to the adjacent computing node identifier index order. The task unloading execution volume forms an allocation vector for the set of adjacent computing nodes according to the adjacent computing node identifier index order. Each component of the allocation vector corresponds to an adjacent computing node identifier index position and represents the unloading task volume for that adjacent computing node identifier index position. Task unloading execution separates the corresponding task volume from the computing node task queue according to the allocation vector components in sequence and forms an unloading task set for adjacent computing nodes. The task volume not allocated to the adjacent computing node identifier index position is retained in the computing node for local execution.

[0106] Computing power allocation is performed based on the execution amount of computing power allocation. The computing power allocation allocates computing power capacity to tasks according to the computing power capacity dimension of the available computing power resource status. The execution amount of computing power allocation forms a resource allocation vector on the computing power capacity dimension. Each component of the resource allocation vector corresponds to a computing power capacity dimension and represents the amount of resources allocated to that computing power capacity dimension. The computing power allocation execution allocates computing power capacity to the task execution set retained on the computing power node according to the resource allocation vector. The computing power capacity allocation result is used to constrain the computing resource occupation of tasks on the computing power node side.

[0107] The task execution result is obtained based on task unloading and computing power allocation, and the reward is calculated. The reward is stored in association with the sampling time identifier. The reward is determined by the weighted sum of at least two of the following: task latency, resource consumption, and link occupancy cost. The task execution result includes the local task completion latency and the unloading task transmission and execution completion latency. The resource consumption is formed by the computing power capacity occupancy and available memory capacity occupancy corresponding to the computing power allocation execution volume. The link occupancy cost is formed by the link bandwidth occupancy corresponding to the task unloading execution volume. The weighted summation calculation multiplies the task latency item, resource consumption item, and link occupancy cost item by preset weights and sums them to obtain the reward value. The reward value is written to the reward cache and stored together with the sampling time identifier.

[0108] The reinforcement learning policy parameters are updated based on the node observations and rewards associated with the sampling time identifier. The updated reinforcement learning policy parameters are used to adjust the policy mapping relationship of the reinforcement learning decision unit to calculate the resource allocation decision variables based on the node observations. The node observations are stored in association with the sampling time identifier, and the rewards are stored in association with the sampling time identifier. The reinforcement learning decision unit uses the sampling time identifier to match the node observations and rewards to form the training input. The reinforcement learning policy parameter update is executed on the computing node side and the updated reinforcement learning policy parameters are written back to the policy parameter storage area, which is used to output the resource allocation decision variables based on the node observations at the next sampling time.

[0109] Example 1: To verify the feasibility of the present invention in practice, it was applied to a distributed computing network scenario consisting of multiple computing nodes. This distributed computing network is used to carry continuously arriving computing tasks with different data sizes, computational complexities, and latency sensitivities. Each computing node has differences in available computing power capacity, available memory capacity, link bandwidth, round-trip latency, and packet loss rate with adjacent nodes. In this scenario, computing nodes need to achieve efficient utilization of overall computing resources through task offloading and computing power allocation while ensuring timely task completion. Existing scheduling methods are prone to problems such as task backlog, link congestion, or idle resources of local nodes when facing task load fluctuations and link status changes, making it difficult to achieve stable and efficient collaborative scheduling under multiple constraints.

[0110] In this scenario, each computing node deploys a reinforcement learning decision unit. The decision unit periodically collects the task queue status, available computing resources status, and link status between its node and adjacent computing nodes. After normalizing the above statuses, the units are concatenated to form node observations. Based on the node observations, the reinforcement learning decision unit outputs resource allocation decision variables consisting of task offloading ratio and computing power allocation ratio. Through boundary pruning and normalization, the resource allocation decision variables are restricted to a feasible domain that satisfies resource capacity constraints, link capacity constraints, and task constraints. Subsequently, the resource allocation decision variables are mapped to dual variables that correspond one-to-one with each type of constraint. Each computing node maintains a shared copy of its own and its adjacent computing nodes' dual variables locally. Through the improved CHOCO-Gossip consensus algorithm, each node calculates the difference vector of the dual variable using the shared copy as the difference benchmark. After sparse compression of the difference vector, only the compressed difference information is sent. After receiving it, the adjacent nodes accumulate and update the shared copy according to the index relationship. After the shared copy update is completed, each node performs a consensus iterative update of the dual variable according to the preset mixed weight, and inversely maps the updated dual variable to a new resource allocation decision variable for the next round of task unloading and computing power allocation execution. At the same time, the reward is calculated based on the task execution result and the reinforcement learning policy parameters are updated, thereby achieving continuous adaptive optimization.

[0111] In this embodiment, the method of the present invention is compared with an independent reinforcement learning scheduling method that does not employ a dual consistency mechanism and a consistent scheduling method that employs full dual variable synchronization. By running multiple scheduling cycles under the same task load conditions, statistical analysis is performed on the average task completion latency, computing resource utilization, and inter-node communication data volume. Experimental results show that the method of the present invention significantly reduces communication overhead and improves overall scheduling performance while ensuring scheduling stability. Specific comparison data are shown in Table 1:

[0112] Table 1 Performance Comparison of Different Scheduling Methods

[0113] index Independent reinforcement learning methods Full consistency method Method of the present invention Average task completion latency (milliseconds) 185 162 134 Task timeout rate (%) 6.8 4.5 2.1 Average utilization rate of computing resources (%) 72.4 78.9 86.3 Average communication data volume per node (KB / cycle) 18.6 96.4 31.2 Number of scheduling result constraints violated More less Very few

[0114] As can be seen from the table, after adopting the method of this invention, the average task completion latency is reduced by about 27.6% compared with the independent reinforcement learning method and by about 17.3% compared with the full consistency method, and the task timeout rate is significantly reduced. At the same time, in terms of communication data volume, the method of this invention only needs to transmit compressed differential information, which reduces the communication overhead by about 67.6% compared with the full consistency method, effectively alleviating the link pressure in the distributed computing network. In addition, since feasible region constraints are always applied in the resource allocation decision generation and update process, the invention has almost no resource capacity or link capacity violations during the experiment, and the scheduling results are stable and reliable.

[0115] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A distributed computing power collaborative allocation method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 1: Configure a reinforcement learning decision unit for each computing node in the distributed computing power network, collect task queue status, available computing power resource status, link status, and generate node observations; Step 2: The reinforcement learning decision unit outputs resource allocation decision variables based on node observations, and restricts the resource allocation decision variables to the feasible region; Step 3: Map resource allocation decision variables to dual variables, maintain a shared copy of the dual variables for computing nodes, and establish a relationship between adjacent computing node identifiers and shared copy indexes; Step 4: The improved CHOCO-Gossip uniformity algorithm is used to calculate the dual variable and the dual variable shared copy difference vector. The compression operator is applied to the difference vector to generate compressed difference information, and the compressed difference information is sent to the adjacent computing power nodes. Step 5: The computing power node receives the compressed differential information sent by the adjacent computing power node, and performs cumulative update on the shared copy of the dual variable according to the shared copy index relationship; Step 6: Perform a consistent iterative update on the dual variables according to the preset mixed weights to obtain the updated dual variables; Step 7: Inversely map the updated dual variable to the updated resource allocation decision variable, and update the resource allocation decision variable to satisfy the feasible region; Step 8: Based on the updated resource allocation decision variables, perform task unloading and computing power allocation, calculate the reward, and update the reinforcement learning strategy parameters.

2. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step one specifically involves: In a distributed computing power network, each computing power node is configured with a reinforcement learning decision unit and a sampling period is set; At each sampling moment, the task queue status, available computing power resource status, and link status are collected. The task queue status includes the task queue length, task arrival rate, and average task deadline. The available computing power resource status includes the available computing power capacity and available memory capacity. The link status includes the available bandwidth, round-trip latency, and packet loss rate. Perform normalization processing on the task queue status, available computing resources status, and link status; The normalized task queue status, normalized available computing power resource status, and normalized link status are concatenated in a fixed concatenation order to generate node observations, which are then stored in association with the computing power nodes.

3. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step two specifically involves: The reinforcement learning decision-making unit calculates resource allocation decision variables based on node observations and reinforcement learning strategy parameters; The resource allocation decision variables consist of task unloading decision variables and computing power allocation decision variables. The task unloading decision variables are used to determine the unloading ratio of tasks between computing power nodes and adjacent computing power nodes, while the computing power allocation decision variables are used to determine the computing power capacity allocation ratio of tasks on computing power nodes. Construct a feasible region, which includes resource capacity constraints, link capacity constraints, and task constraints. Resource capacity constraints limit the upper bound of computing power allocation decision variables based on the available computing power resource status. Link capacity constraints limit the upper bound of bandwidth occupancy of task unloading decision variables based on link status. Task constraints limit the value range of task unloading decision variables based on task queue status. The feasible region constraint processing is performed on the resource allocation decision variables. The feasible region constraint processing includes boundary pruning and normalization processing. Boundary pruning restricts the components of the resource allocation decision variables to a range of values, while normalization ensures that the resource allocation decision variables meet the resource capacity constraints and link capacity constraints, thus obtaining resource allocation decision variables that satisfy the feasible region.

4. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step three specifically involves: Based on the resource allocation decision variables and the feasible region, dual variables are constructed. The dual variables include the dual components corresponding to the resource capacity constraints, the dual components corresponding to the link capacity constraints, and the dual components corresponding to the task constraints. A shared copy of dual variables is maintained for each computing power node. The shared copy of dual variables includes the shared copy of dual variables corresponding to the computing power node and the shared copy of dual variables corresponding to the identifiers of adjacent computing power nodes. The shared copy of dual variables has the same dimension as the dual variables. Establish a relationship between adjacent computing node identifiers and shared copy indexes. The shared copy index relationship records the mapping relationship between adjacent computing node identifiers and the storage location of the shared copy of the dual variable.

5. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step four specifically involves: An improved CHOCO-Gossip unification algorithm is used to generate compressed differential information. The improved CHOCO-Gossip unification algorithm maintains a shared copy of the dual variables on the computing node side and uses the shared copy of the dual variables as the differential benchmark. At the sampling time, the computing node reads the shared copy of the dual variable and the dual variable corresponding to the computing node, and calculates the difference vector according to the dimension of the dual variable; A compression operator is applied to the difference vector to generate compressed difference information. The compression operator performs sparsification on the difference vector and retains the non-zero components. The compressed difference information includes a set of non-zero component values ​​and a set of non-zero component indices. The set of non-zero component indices identifies the position of the set of non-zero component values ​​in the dual variable dimension. The data transmission unit is encapsulated based on the neighboring computing power node identifier and the shared copy index relationship. The data transmission unit includes the computing power node identifier, the sampling time identifier, the non-zero component index set, and the non-zero component value set. The computing power node sends the data transmission unit to the neighboring computing power node.

6. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step five specifically involves: The computing power node obtains the computing power node identifier, sampling time identifier, non-zero component index set, and non-zero component value set based on the transmission data units sent by neighboring computing power nodes; The computing power node determines the shared copy storage location of the dual variable corresponding to the adjacent computing power node identifier based on the relationship between the adjacent computing power node identifier and the shared copy index; The computing node traverses the non-zero component index set in the index order of the non-zero component index set, uses the non-zero component index as the dual variable dimension position index, and uses the non-zero component value with the same index in the non-zero component value set as the accumulation amount to perform the accumulation update on the dimension component corresponding to the dual variable shared copy in the dual variable shared copy storage location. The computing nodes will associate the accumulated and updated shared copies of the dual variables with the sampling time identifier and store them in the shared copy storage area of ​​the dual variables.

7. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step six specifically involves: At the sampling time, the dual variable is obtained, and the dual variable shared copy corresponding to the computing power node and the dual variable shared copy corresponding to the adjacent computing power node identifier are obtained from the dual variable shared copy storage area. The dual variable shared copy corresponding to the adjacent computing power node identifier is determined according to the index relationship between the adjacent computing power node identifier and the shared copy. Obtain a preset mixed weight, which includes a self weight and adjacent weights. The self weight corresponds to the computing power node, and the adjacent weights are arranged in the order of the adjacent computing power node identifier index. The preset mixed weight satisfies that the sum of the self weight and the adjacent weight is 1. The dual variable is updated using a consistent iterative update based on a preset mixed weight. The consistent iterative update includes a weighted summation of the dual variable dimensions one by one. The component of the updated dual variable at any dimension is equal to the sum of its own weight multiplied by the component of the dual variable at the same dimension, plus the sum of the adjacent weights multiplied by the sum of the components of the shared copy of the dual variable corresponding to the adjacent computing node identifier at the same dimension. This results in the updated dual variable.

8. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step seven specifically involves: The inverse mapping is performed based on the updated dual variable. The inverse mapping establishes a field correspondence based on the mapping relationship between the resource allocation decision variable and the dual variable, and transforms the updated dual variable into the updated resource allocation decision variable. The updated resource allocation decision variables consist of updated task unloading decision variables and updated computing power allocation decision variables. The updated task unloading decision variables are arranged in the order of adjacent computing power node identifier index and are used to determine the unloading ratio of tasks between computing power nodes and adjacent computing power node sets. The updated computing power allocation decision variables are arranged in the computing power capacity dimension of available computing power resource status and are used to determine the computing power capacity allocation ratio of tasks on computing power nodes. The updated resource allocation decision variables are subjected to feasible region restriction processing, which includes boundary pruning and normalization. Boundary pruning restricts the components of the updated resource allocation decision variables to the value range defined by the task constraints. Normalization ensures that the updated task unloading decision variables meet the link capacity constraints and the updated computing power allocation decision variables meet the resource capacity constraints, thus obtaining updated resource allocation decision variables that satisfy the feasible region.

9. The distributed computing power collaborative allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, Step eight specifically involves: The task unloading execution volume and computing power allocation execution volume are determined based on the updated resource allocation decision variables. The task unloading execution volume is determined by the updated task unloading decision variables and the task queue status, while the computing power allocation execution volume is determined by the updated computing power allocation decision variables and the available computing power resource status. Task unloading is performed based on the task unloading execution volume. Task unloading distributes tasks among computing nodes and sets of adjacent computing nodes according to the index order of adjacent computing node identifiers. Computing power allocation is performed based on the amount of computing power allocated. Computing power allocation is performed on the task according to the computing power capacity dimension of available computing power resources. Based on task unloading and computing power allocation, the task execution results are obtained and the reward is calculated. The reward is stored in association with the sampling time identifier. The reward is determined by the weighted sum of at least two of the following: task latency, resource consumption, and link occupation cost. The reinforcement learning policy parameters are updated based on the node observations and rewards associated with the sampling time identifier. The updated reinforcement learning policy parameters are used to adjust the policy mapping relationship of the reinforcement learning decision unit in calculating resource allocation decision variables based on node observations.

Images