An end-to-end network slice resource allocation method based on machine learning

Abstract

The application discloses an end-to-end network slice resource allocation method based on machine learning. The reinforcement learning IPPO algorithm is mainly used for deciding an RB resource allocation scheme which satisfies various constraints of a network slice mathematical model and makes a target function maximum. The random forest algorithm is used as an auxiliary and is mainly used for quickly judging whether the RB resource allocation scheme given by an access network can successfully complete slice VNF mapping in a core network. The two complement each other, and the original end-to-end joint optimization problem is converted into separate optimization on the access network side and the core network side without affecting the optimal solution. Therefore, the calculation time complexity is greatly reduced, and the resource allocation efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, and in particular relates to an end-to-end network slicing resource allocation method based on machine learning. Background Technology

[0002] With the development of 5G technology and the emergence of diversified services, various quality of service (QoS) requirements have been placed on existing communication networks. As a key enabling technology and infrastructure in the digital economy era, 5G's services have expanded from simple mobile communication to ubiquitous connectivity and application scenarios. Therefore, in order to better meet the differentiated needs of users by utilizing existing communication networks, network slicing technology has emerged. The end-to-end network slicing resource allocation problem is generally divided into two parts: the allocation of radio spectrum resources in the access network (RAN) and the mapping of VNFs in the core network. The access network side is mainly responsible for allocating radio spectrum resource blocks (RBs) to users in the slice. The core network side is mainly responsible for mapping the VNFs of the slice to general-purpose servers in the underlying physical network. The network slicing resource allocation problem is usually modeled as a nonlinear integer programming mathematical model. Due to the numerous and complex constraints involved, it is difficult to solve using traditional optimization algorithms. In recent years, with the continuous emergence of artificial intelligence technology, many scholars have proposed using reinforcement learning methods to solve the problem; however, its time complexity is high, and it is usually only suitable for small-scale problems. Summary of the Invention

[0003] The purpose of this invention is to provide an end-to-end network slice resource allocation method based on machine learning, which integrates the random forest algorithm in machine learning with the reinforcement learning IPPO algorithm to solve the technical problem of resource allocation in network slicing.

[0004] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0005] An end-to-end network slice resource allocation method based on machine learning includes the following steps:

[0006] Step 1: First, build the corresponding simulation environment based on the network slicing mathematical model, and initialize the end-to-end network slicing access network and core network parameters; the access network is responsible for allocating RBs to users in the slice, where RB refers to Radio Spectrum Resource Block, while the core network is responsible for mapping the VNFs of the network slice to general servers that meet the model constraints, where VNF refers to Virtual Network Functions.

[0007] Step 2: Analyze the characteristics of the network slicing mathematical model. Calculate the elements in the Associated Tuple on the access network side and the core network side using the RB allocation scheme of the radio spectrum resources given on the access network side. This is used to determine whether the corresponding allocation scheme can be successfully deployed on the core network side. The Associated Tuple refers to the associated tuple.

[0008] Step 3: Use the DFS+pruning algorithm to determine whether the given Associated Tuple has a deployment method that satisfies all constraints on the core network side; where DFS+pruning algorithm refers to depth-first search + pruning algorithm; core network side constraints include: virtual machine (VM) number constraints on servers, server computing resource constraints, link bandwidth constraints, and link latency constraints.

[0009] Step 4: Randomly generate multiple sets of Associated Tuple datasets, use the DFS+pruning algorithm to obtain the corresponding labels, and then use the random forest algorithm to train a binary classifier to quickly determine whether a given Associated Tuple can complete VNF mapping in the core network.

[0010] Step 5: On the access network side, first transform the network slicing mathematical model optimization problem into a Markov Decision Process (MDP): in Representing the state space, Represents the action space. Represents the reward function;

[0011] Step 6: Train the agent using the reinforcement learning IPPO algorithm to make decisions on wireless spectrum resource (RB) allocation schemes that satisfy various constraints and maximize profits. The IPPO algorithm refers to Independent Near-End Policy Optimization Algorithm.

[0012] Furthermore, step 2 includes the following steps:

[0013] Step 2.1: First, calculate the rate of user u in slice s using Shannon's formula based on the RB resource allocation scheme given by the access network. in It is a 01 integer variable. This means assigning the k-th RB to user u in slice s. This indicates that the k-th RB will not be assigned to user u in slice s, W represents the bandwidth of an RB, and P d G represents downlink transmission power. s,u σ represents the channel gain between the user and the base station. 2 The power of the Gaussian white noise is then used to calculate the total rate for each slice. As a part of the Associated Tuple;

[0014] Step 2.2: Then, use the bisection method to find the maximum allowable delay of slice s that satisfies the core network constraints. As another element in the Associated Tuple, where m represents the core network latency, Describes the set of users in slice s. This represents the personalized latency requirements of user u in slice s. This represents the access network latency of user u in slice s. This represents the minimum user satisfaction threshold for slice s;

[0015] Step 2.3: Set there to a total of S slices, therefore the Associated Tuple is represented as:

[0016] Furthermore, step 3 includes the following steps:

[0017] Step 3.1, the service function chain (SFC) of slice s is represented as follows: Where t s This represents the total number of VNFs contained in the SFC of slice s. This indicates the type of the j-th VNF in the SFC of slice s. Based on the Associated Tuple obtained in step 2, the formula is used... Calculate the required link bandwidth resources between adjacent VNFs in slice s, where This represents the link bandwidth resources required between the j-th VNF and the (j+1)-th VNF in slice s. The type is The transmission rate conversion rate of the VNF is expressed by the formula. Calculate the server computing resources required for all VNFs in the slice, where Let α represent the computational resources required for the j-th VNF in slice s, and let α represent the correlation coefficient between the transmission rate and the required computational resources.

[0018] Step 3.2: Transform the VNF mapping problem of the network slice core network into a tree model. The number of child nodes of each node in the tree is equal to the number of server nodes. The depth of the tree corresponds to the total number of VNFs in all slices. The root node of the tree represents the starting mark of the allocation scheme. The path between the root node of the tree and the leaf node of the tree corresponds to a VNF mapping scheme.

[0019] Step 3.3: Then, use the DFS algorithm to traverse the tree model constructed in step 3.2 to find a mapping scheme that satisfies all the constraints of the core network.

[0020] Step 3.4: Start searching downwards from the root node of the tree, traversing all paths. Each path corresponds to a VNF mapping scheme. Each node in the path represents whether the current VNF ​​is deployed to the corresponding server. If the constraints are met, continue searching downwards along the current node; otherwise, backtrack. Continue until the leaf node of the tree is reached, indicating that the VNFs of all slices have been successfully deployed and a VNF deployment scheme that satisfies the constraints of the core network has been found.

[0021] Step 3.5: To reduce the time complexity of the search, a pruning strategy is added; Pruning strategy 1: Search from difficult to easy according to the difficulty of successfully deploying each VNF; Pruning strategy 2: If the current node does not meet a certain constraint, backtrack directly without searching for the complete deployment scheme and then making a unified judgment.

[0022] Furthermore, step 4 includes the following steps:

[0023] Step 4.1: First, randomly generate multiple sets of Associated Tuples, and then use the DFS+pruning algorithm in step 3 to generate the corresponding labels (0 / 1) of Associated Tuples. A label of 1 indicates that the current Associated Tuple has successfully completed VNF mapping in the core network, and a label of 0 indicates that the current Associated Tuple cannot complete VNF mapping in the core network. All Associated Tuples and their corresponding labels constitute the training set D.

[0024] Step 4.2: Use the Random Forest (RF) algorithm to train a binary classifier using the training set D obtained in Step 4.1;

[0025] Step 4.3: First, generate the training set for each decision tree through bootstrap sampling. Then, each base learner was trained independently using a machine learning decision tree algorithm;

[0026] Step 4.4: Finally, the classification results of the random forest are obtained by combining the outputs of all base learner decision trees through a voting method.

[0027] Furthermore, step 5 includes the following steps:

[0028] Step 5.1: First, assign an agent to each slice, numbered as: {1,2…,s,…,S};

[0029] Step 5.2, State Space Setting: For an agent numbered s, its state space Defined as: in Let G represent the set of users in slice s. s This represents the set of channel gains for users and base stations in slice s. N represents the set of personalized needs of users in slice s. s This represents the set of radio spectrum resources (RBs) allocated to users in slice s; therefore, the total state space... Represented as:

[0030] Step 5.3, Motion Space Setting: For an agent numbered s, its action space Defined as: Where K represents the total number of radio spectrum resources (RBs) in the resource pool. Let represent the set of users in slice s; the action space mentioned above essentially represents the set of specific schemes for allocating radio spectrum resources RB, and is also represented in row vector form: Among them, if This represents allocating RB resource k to user u in slice s; each agent is responsible for deciding the RB allocation scheme within its corresponding slice, therefore the total action space is represented by all slices. Cartesian product:

[0031] Step 5.4, Reward Function The setup is as follows: the given RB allocation scheme is transformed into an Associated Tuple according to step 2, and then input into the random forest classifier trained in step 4 to determine whether it meets the constraints of the core network. If the given RB allocation scheme meets the constraints, the reward function is set to the benefit obtained by the current allocation scheme; otherwise, a negative reward is obtained.

[0032] Furthermore, step 6 includes the following steps:

[0033] Step 6.1: First, initialize the policy network parameters of each agent in the IPPO algorithm. and value network parameters

[0034] Step 6.2: Each agent observes the current environmental state and, based on the current policy, moves from the action space... Choose one action a s ;

[0035] Step 6.3: The actions of all agents constitute a joint action: a = {a 1 ,a 2 ,...,a S};

[0036] Step 6.4: The combined action is applied to the current state, updating the current state from s to s′, and an immediate reward is obtained.

[0037] Step 6.5, then, the reward. This will be used as feedback to update the parameters of the policy neural network and the value neural network of all agents;

[0038] Step 6.6: Repeat steps 6.2-6.5 until the reward is obtained. Convergence indicates that the optimal RB resource allocation scheme has been found.

[0039] This invention presents a machine learning-based end-to-end network slicing resource allocation method with the following advantages: The reinforcement learning algorithm, as the core component, continuously learns better RB resource allocation schemes based on received reward feedback. The random forest algorithm, as an auxiliary component, helps reinforcement learning quickly determine whether its given allocation scheme satisfies the constraints of the core network. The two complement each other, transforming the original end-to-end joint optimization problem into separate optimizations on the access network and core network sides without affecting the optimal solution. This significantly reduces computational time complexity and improves resource allocation efficiency. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the training of the random forest algorithm of the present invention;

[0041] Figure 2 This is a schematic diagram of the reinforcement learning algorithm training of the present invention;

[0042] Figure 3 This is a flowchart of an end-to-end network slicing resource allocation method based on machine learning according to the present invention. Detailed Implementation

[0043] To better understand the purpose, structure, and function of this invention, the following detailed description of an end-to-end network slicing resource allocation method based on machine learning is provided in conjunction with the accompanying drawings.

[0044] This invention proposes an end-to-end network slice resource allocation method based on machine learning.

[0045] The following is an implementation example:

[0046] (1) First, build the corresponding simulation environment based on the network slicing mathematical model, and initialize the various parameters of the network slicing access network and core network. Among them, the access network is mainly responsible for allocating radio spectrum resources (RB) to users in the slice, while the core network is mainly responsible for mapping the VNF of the network slice to a suitable general server.

[0047] (2) Analyze the characteristics of the system model and identify the associated tuples between the access network side and the core network side. First, calculate the rate of user u in slice s using Shannon's formula based on the RB resource allocation scheme given by the access network. in It is a 0-1 integer variable indicating whether the k-th RB is assigned to user u in slice s, W represents the bandwidth of an RB, and P d G represents downlink transmission power. s,u σ represents the channel gain between the user and the base station. 2 The power of the Gaussian white noise is then used to calculate the total rate for each slice. As part of the associated tuple; then, the maximum allowable delay of slice s satisfying the core network constraints is calculated using a binary search method. As another element in the Associated Tuple, where m represents the core network latency, Describes the set of users in slice s. This represents the personalized latency requirements of user u in slice s. This represents the access network latency of user u in slice s. Let represent the minimum user satisfaction threshold for slice s; assuming there are a total of S slices, the Associated Tuple is represented as:

[0048] (3) The service function chain (SFC) of slice s is represented as follows: Where t s This represents the total number of VNFs contained in the SFC of slice s. This indicates the type of the j-th VNF in the SFC of slice s. Based on the Associated Tuple obtained in step 2, the formula is used... Calculate the required link bandwidth resources between adjacent VNFs in slice s, where This represents the link bandwidth resources required between the j-th VNF and the (j+1)-th VNF in slice s. The type is The transmission rate conversion rate of the VNF is expressed by the formula. Calculate the server computing resources required for all VNFs in the slice, where Let α represent the computational resources required for the j-th VNF in slice s, and let α represent the correlation coefficient between transmission rate and required computational resources. Then, the VNF mapping problem of the network slice core network is transformed into a tree model. The number of child nodes of each node in the tree equals the number of server nodes. The depth of the tree corresponds to the total number of VNFs in all slices. The root node of the tree represents the starting marker of the allocation scheme, and the path from the root node to a leaf node corresponds to a VNF mapping scheme. Then, a depth-first search + pruning algorithm is used to traverse the constructed tree model to find a mapping scheme that satisfies all constraints of the core network. Specifically, the DFS algorithm is used to traverse the tree model constructed in step 3.2 to find a mapping scheme that satisfies all constraints of the core network.

[0049] Step 3.4: Start searching downwards from the root node of the tree, traversing all paths. Each path corresponds to a VNF mapping scheme. Each node in the path represents whether the current VNF ​​is deployed to the corresponding server. If the constraints are met, continue searching downwards along the current node; otherwise, backtrack. Continue until the leaf node of the tree is reached, indicating that the VNFs of all slices have been successfully deployed, and a VNF deployment scheme that satisfies the constraints of the core network has been found.

[0050] To reduce the time complexity of the search, a pruning strategy is added. Pruning strategy 1: Search from hardest to easiest according to the difficulty of successfully deploying each VNF. Pruning strategy 2: If the current node does not meet a certain constraint, backtrack directly without searching for a complete deployment solution and then making a unified judgment.

[0051] (4) Randomly generate multiple sets of Associated Tuples, and then use the DFS+pruning algorithm in step (3) to generate labels (0 / 1) for the corresponding Associated Tuples. A label of 1 indicates that the current Associated Tuple can successfully complete VNF mapping in the core network, and a label of 0 indicates that the current Associated Tuple cannot complete VNF mapping in the core network. All Associated Tuples and their corresponding labels constitute the training set D. Then, the Random Forest (RF) algorithm is used to train a binary classifier using the training set D obtained above. Specifically, the training set for each decision tree is first generated through bootstrap sampling. Then, each base learner is trained independently using a machine learning decision tree algorithm. Finally, the outputs of all base learner decision trees are combined using a voting method to obtain the classification results of the random forest.

[0052] (5) On the access network side, the network slicing mathematical model is first transformed into a Markov decision process (MDP): in Representing the state space, Represents the action space. This represents the reward function.

[0053] Includes the following steps:

[0054] Step 5.1: First, assign an agent to each slice, numbered as: {1,2…,s,…,S};

[0055] Step 5.2, State Space Setting: For an agent numbered s, its state space Defined as: in Let G represent the set of users in slice s. s This represents the set of channel gains for users and base stations in slice s. N represents the set of personalized needs of users in slice s. s This represents the set of radio spectrum resources (RBs) allocated to users in slice s; therefore, the total state space... Represented as:

[0056] Step 5.3, Motion Space Setting: For an agent numbered s, its action space Defined as: Where K represents the total number of radio spectrum resources (RBs) in the resource pool. Let represent the set of users in slice s; the action space mentioned above essentially represents the set of specific schemes for allocating radio spectrum resources RB, and is also represented in row vector form: Among them, if This represents allocating RB resource k to user u in slice s; each agent is responsible for deciding the RB allocation scheme within its corresponding slice, therefore the total action space is represented by all slices. Cartesian product:

[0057] Step 5.4, Reward Function The setup is as follows: the given RB allocation scheme is transformed into an Associated Tuple according to step 2, and then input into the random forest classifier trained in step 4 to determine whether it meets the constraints of the core network. If the given RB allocation scheme meets the constraints, the reward function is set to the benefit obtained by the current allocation scheme; otherwise, a negative reward is obtained.

[0058] (6) Finally, the reinforcement learning IPPO algorithm is used to train the agents. The specific process is as follows: First, the policy network parameters of each agent in the IPPO algorithm are initialized. and value network parameters Each agent then observes the current environmental state and selects an action 'a' from the action space according to the current policy. s The actions of all agents constitute a joint action: a = {a 1 ,a 2 ,...,a S The combined action acts on the current state, updating it from s to s′, and simultaneously grants an immediate reward. After that, the reward This feedback is used to update the parameters of the policy neural network and the value neural network of all agents. This process is repeated until the reward converges, indicating that an RB resource allocation scheme has been found that satisfies all constraints and maximizes profit.

[0059] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A machine learning-based end-to-end network slice resource allocation method, characterized in that, Includes the following steps: Step 1: First, build the corresponding simulation environment based on the network slicing mathematical model, and initialize the end-to-end network slicing access network and core network parameters; the access network is responsible for allocating RBs to users in the slice, where RB refers to Radio Spectrum Resource Block, while the core network is responsible for mapping the VNFs of the network slice to general servers that meet the model constraints, where VNF refers to Virtual Network Functions. Step 2: Analyze the characteristics of the network slicing mathematical model. Calculate the elements in the Associated Tuple on the access network side and the core network side using the RB allocation scheme of the radio spectrum resources given on the access network side. This is used to determine whether the corresponding allocation scheme can be successfully deployed on the core network side. The Associated Tuple refers to the associated tuple. Step 3: Use the DFS+pruning algorithm to determine whether the given Associated Tuple has a deployment method that satisfies all constraints on the core network side; where DFS+pruning algorithm refers to depth-first search + pruning algorithm; core network side constraints include: virtual machine (VM) number constraints on servers, server computing resource constraints, link bandwidth constraints, and link latency constraints. Step 4: Randomly generate multiple sets of Associated Tuple datasets, use the DFS+pruning algorithm to obtain the corresponding labels, and then use the random forest algorithm to train a binary classifier to quickly determine whether a given Associated Tuple can complete VNF mapping in the core network. Step 5: On the access network side, first transform the network slicing mathematical model optimization problem into a Markov Decision Process (MDP): in Representing the state space, Represents the action space. Represents the reward function; Step 6: Train the agent using the reinforcement learning IPPO algorithm to make decisions on wireless spectrum resource (RB) allocation schemes that satisfy various constraints and maximize profits. The IPPO algorithm refers to Independent Near-End Policy Optimization Algorithm.

2. The end-to-end network slice resource allocation method based on machine learning according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: First, calculate the rate of user u in slice s using Shannon's formula based on the RB resource allocation scheme given by the access network. in It is a 01 integer variable. This means assigning the k-th RB to user u in slice s. This indicates that the k-th RB will not be assigned to user u in slice s, W represents the bandwidth of an RB, and P d G represents downlink transmission power. s,u σ represents the channel gain between the user and the base station. 2 The power of the Gaussian white noise is then used to calculate the total rate for each slice. As a part of the Associated Tuple; Step 2.2: Then, use the bisection method to find the maximum allowable delay of slice s that satisfies the core network constraints. As another element in the Associated Tuple, where m represents the core network latency, Describes the set of users in slice s. This represents the personalized latency requirements of user u in slice s. This represents the access network latency of user u in slice s. This represents the minimum user satisfaction threshold for slice s; Step 2.3: Set there to a total of S slices, therefore the Associated Tuple is represented as:

3. The end-to-end network slice resource allocation method based on machine learning according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1, the service function chain (SFC) of slice s is represented as follows: Where t s This represents the total number of VNFs contained in the SFC of slice s. This indicates the type of the j-th VNF in the SFC of slice s. Based on the Associated Tuple obtained in step 2, the formula is used... Calculate the required link bandwidth resources between adjacent VNFs in slice s, where This represents the link bandwidth resources required between the j-th VNF and the (j+1)-th VNF in slice s. The type is The transmission rate conversion rate of the VNF is expressed by the formula. Calculate the server computing resources required for all VNFs in the slice, where Let α represent the computational resources required for the j-th VNF in slice s, and let α represent the correlation coefficient between the transmission rate and the required computational resources. Step 3.2: Transform the VNF mapping problem of the network slice core network into a tree model. The number of child nodes of each node in the tree is equal to the number of server nodes. The depth of the tree corresponds to the total number of VNFs in all slices. The root node of the tree represents the starting mark of the allocation scheme. The path between the root node of the tree and the leaf node of the tree corresponds to a VNF mapping scheme. Step 3.3: Then, use the DFS algorithm to traverse the tree model constructed in step 3.2 to find a mapping scheme that satisfies all the constraints of the core network. Step 3.4: Start searching downwards from the root node of the tree, traversing all paths. Each path corresponds to a VNF mapping scheme. Each node in the path represents whether the current VNF ​​is deployed to the corresponding server. If the constraints are met, continue searching downwards along the current node; otherwise, backtrack. Continue until the leaf node of the tree is reached, indicating that the VNFs of all slices have been successfully deployed and a VNF deployment scheme that satisfies the constraints of the core network has been found. Step 3.5: To reduce the time complexity of the search, a pruning strategy is added; Pruning strategy 1: Search from difficult to easy according to the difficulty of successfully deploying each VNF; Pruning strategy 2: If the current node does not meet a certain constraint, backtrack directly without searching for the complete deployment scheme and then making a unified judgment.

4. The end-to-end network slice resource allocation method based on machine learning according to claim 1, characterized in that, Step 4 includes the following steps: Step 4.1: First, randomly generate multiple sets of Associated Tuples, and then use the DFS+pruning algorithm in step 3 to generate the corresponding labels (0 / 1) of Associated Tuples. A label of 1 indicates that the current Associated Tuple has successfully completed VNF mapping in the core network, and a label of 0 indicates that the current Associated Tuple cannot complete VNF mapping in the core network. All Associated Tuples and their corresponding labels constitute the training set D. Step 4.2: Use the Random Forest (RF) algorithm to train a binary classifier using the training set D obtained in Step 4.1; Step 4.3: First, generate the training set for each decision tree through bootstrap sampling. Then, each base learner was trained independently using a machine learning decision tree algorithm; Step 4.4: Finally, the classification results of the random forest are obtained by combining the outputs of all base learner decision trees through a voting method.

5. The end-to-end network slice resource allocation method based on machine learning according to claim 1, characterized in that, Step 5 includes the following steps: Step 5.1: First, assign an agent to each slice, numbered as: {1,2...,s,...,S}; Step 5.2, State Space Setting: For an agent numbered s, its state space Defined as: in Let G represent the set of users in slice s. s This represents the set of channel gains for users and base stations in slice s. N represents the set of personalized needs of users in slice s. s This represents the set of radio spectrum resources (RBs) allocated to users in slice s; therefore, the total state space... Represented as: Step 5.3, Motion Space Setting: For an agent numbered s, its action space Defined as: Where K represents the total number of radio spectrum resources (RBs) in the resource pool. Let represent the set of users in slice s; the above action space essentially represents the set of specific schemes for allocating radio spectrum resources RB, and is also represented as a row vector: Among them, if This represents allocating RB resource k to user u in slice s; each agent is responsible for deciding the RB allocation scheme within its corresponding slice, therefore the total action space is represented by all slices. Cartesian product: Step 5.4, Reward Function The setup is as follows: the given RB allocation scheme is transformed into an Associated Tuple according to step 2, and then input into the random forest classifier trained in step 4 to determine whether it meets the constraints of the core network. If the given RB allocation scheme meets the constraints, the reward function is set to the benefit obtained by the current allocation scheme; otherwise, a negative reward is obtained.

6. The end-to-end network slice resource allocation method based on machine learning according to claim 1, characterized in that, Step 6 includes the following steps: Step 6.1: First, initialize the policy network parameters of each agent in the IPPO algorithm. and value network parameters Step 6.2: Each agent observes the current environmental state and, based on the current policy, moves from the action space... Choose one action a s ; Step 6.3: The actions of all agents constitute a joint action: a = {a 1 ,a 2 ,...,a S }; Step 6.4: The combined action is applied to the current state, updating the current state from s to s′, and an immediate reward is obtained. Step 6.5, then, the reward. This will be used as feedback to update the parameters of the policy neural network and the value neural network of all agents; Step 6.6: Repeat steps 6.2-6.5 until the reward is obtained. Convergence indicates that the optimal RB resource allocation scheme has been found.

Images