Resource allocation method for NGMA IoE network supporting 6G

Abstract

The present invention relates to the technical field of NGMA IoE network supporting 6G, and specifically discloses a resource allocation method for NGMA IoE network supporting 6G. The method establishes a new conflict graph model for conflicts between TLs and converts it into a conflict hypergraph model. Then, based on the conflict hypergraph model, a resource allocation Markov decision network with sequential decision characteristics is established. In addition, in order to reduce the amount of calculation by distributing the calculation load throughout the network and realize distributed learning, a Federated Average Dominance Actor-Critic algorithm (FedAC) is proposed. Simulation results show that the scheme has a higher network throughput in the NGMA IoE network supporting 6G.

Description

Technical Field

[0001] The present invention relates to the technical field of NGMA IoE network supporting 6G, and in particular to a resource allocation method of an NGMA IoE network supporting 6G. Background Art

[0002] With the commercial deployment of 5G systems, attention has shifted to the development of 6G networks to support ubiquitous, massive connections in the Internet of Everything (IoE). According to the latest vendor report, there will be 13.2 billion device connections in the Internet of Everything by 2022, and will grow at an annual rate of 18% to reach 34.7 billion by 2028, which will create a series of huge challenges in the future, including high spectrum efficiency, network throughput, and massive Internet of Everything Device (IoED) access. Device-to-device (D2D) communication is a short-range technology that allows D2D IoED (DID) to communicate directly without relying on base stations by sharing the resources of cellular IoED (CID), which can improve network throughput and spectrum efficiency.

[0003] Next Generation Multiple Access (NGMA) technology is an advanced form of Non-Orthogonal Multiple Access (NOMA) technology that can connect a large number of IoEDs to a network on a given radio resource. By adopting NGMA, cell resources can be efficiently allocated between NOMA CID Groups (CIDGs), facilitating resource sharing and improving spectrum efficiency. Due to the spectrum efficiency advantages of NOMA and D2D technologies, some researchers combine them to achieve better network performance. However, integrating D2D communication and NOMA will lead to severe overlapping interference between IoEDs (e.g., DID and DID, DID and CID, CID and CID) if they are transmitted concurrently via the same spectrum. Therefore, how to design an effective resource management method to improve network performance and avoid resource interference is a key challenge. Existing work lacks a resource management method for integrating TLs of D2D communication and NOMA under resource conflicts. Therefore, how to effectively avoid resource conflicts between TLs in NGMA IoE supporting 6G remains an unresolved issue. Summary of the invention

[0004] The present invention provides a resource allocation method for an NGMA IoE network supporting 6G, and solves the technical problem of how to efficiently allocate wireless resources while avoiding conflicts between TLs in NGMAIoE supporting 6G.

[0005] In order to solve the above technical problems, the present invention provides a resource allocation method for a NGMAIoE network supporting 6G, comprising the steps of:

[0006] S1. Build a communication network architecture that supports 6G NGMA IoE network, where NGMA IoE network refers to the next generation multi-access Internet of Things.

[0007] The communication network architecture of the NGMA IoE network includes 1 access point, namely AP, M IoT devices, namely IoEDs, D resource blocks, namely RBs, and M transmission links, namely TLs, where the M IoEDs are divided into M′ CIDGs and M″ DID pairs. The CIDs sharing the same RB in the same AP constitute a CIDG, and two DIDs for D2D communication constitute a DID pair. CID refers to a shared cellular IoED, and DID refers to an IoED using D2D communication. D2D communication refers to device-to-device communication. Each DID pair is divided into a DID transmitter, namely a DID T and D2D receiver i.e. DID R ;AP allocates RBs to each CID and is responsible for communication between CIDs; The CDMA mode using cognitive radio allows DIDs to share resources with CIDG; When allocating RBs to TLs, only one TL is considered between CIDG and AP because CIDs in CIDG share the same RBs;

[0008] S2. Establish a conflict graph model of NGMA IoE network in resource allocation based on the communication network architecture of NGMA IoE network;

[0009] S3, establishing a conflict hypergraph model based on the conflict graph model;

[0010] S4. Establishing a conflict-free resource allocation problem of the NGMA IoE network based on the conflict hypergraph model;

[0011] S5, constructing a Markov decision network guided by a Markov decision process based on the conflict-free resource allocation problem;

[0012] S6, training the Markov decision network;

[0013] S7. The trained Markov decision network performs interference-free resource allocation according to the current state of the NGMAIoE network.

[0014] Furthermore, in step S4, the conflict-free resource allocation problem is constructed as follows:

[0015]

[0016]

[0017] Where O represents the network throughput, An indicator variable indicating that the d-th RB is allocated to the m-th IoED at time t, This means that the dth RB is allocated to the mth IoED. The dth RB is not allocated to the mth IoED; R m,d represents the data transmission rate of the mth IoED allocated with the dth RB; represents the RB allocation conflict degree between TLs at time t, D RB represents a set of D RBs, represents a set of M IoEDs; max represents maximization, and st represents what needs to be satisfied;

[0018] The definition is as follows:

[0019]

[0020] Among them, H T represents the transposed matrix of the hypergraph association matrix H of the conflict hypergraph model, max(·,1) means that each element of the matrix (·) is subjected to a max operation with 1, || ||1 represents the L1 norm, X t Represents the resource allocation matrix at time t:

[0021]

[0022] Furthermore, the state at time t is defined as:

[0023]

[0024] in, The hypergraph association matrix H representing the conflict hypergraph model and the set of conflict situations of all TLs The aggregate of all TL data transfer rates The collection of RB allocations of all TLs

[0025] Resource allocation action a at time t t It is expressed as:

[0026] a t ={0,1} M×D ;

[0027] The reward function r at time t t Defined as:

[0028]

[0029] Under strategy π, state s t Next, perform action a t The state action value function Q π (s t ,a t) is defined as:

[0030]

[0031] Under strategy π, state s t The state value function V under π (s t ) is defined as:

[0032]

[0033] Among them, Q π (s t+1 ,a t+1 ) indicates the state s t+1 Next, perform action a t+1 The obtained Q value is represents the expectation, V π (s t ) represents the state value at time t, π(a′|s t ) is represented as state s t The probability of executing action a', Q(s t ,a′) means in state s t Next, take action a' to obtain the Q value, and γ represents the discount factor.

[0034] Furthermore, the step S2 specifically includes the steps of:

[0035] S21. Using an unweighted undirected graph Represents the communication network architecture of the NGMAIoE network. is a vertex set representing N tuples, including AP, CIDG and DID, = {e1, e2, ..., e M} is the edge set representing the TL set; and the incidence matrix G = {0,1} N×M express The relationship between the edges and vertices of G(v n ,e m )=1 means that the nth tuple initiates the mth TL, otherwise G(v n ,e m )=0;

[0036] S22. Establish a conflict graph model based on the conflict relationship between TLs in the NGMA IoE network in is the set of vertices representing TL, is the set of edges representing the conflict relationship between TLs, The relationship between edges and vertices is represented by the adjacency matrix G C ={0,1} M×M Indicates that Indicates that the mth and mth in TL conflict, otherwise

[0037] Furthermore, in step S22, G C Obtained by the following steps:

[0038] 1) Transform the matrix G to obtain the matrix G CD =(G T G)⊙1, the operation⊙ is defined as:

[0039]

[0040] Where a represents the element in the matrix, b is a constant; G CD ={0,1} M×M , Indicates that the mth TL directly conflicts with the mth TL, otherwise A direct conflict means that the mth TL and the mth TL use the same RB at the same time and have the same send or receive tuple; G T is the transposed matrix of matrix G;

[0041] 2) Transform the matrix G to obtain the matrix I is the identity matrix;

[0042] 3) For the matrix Transform and get the matrix Representation Matrix The transposed matrix of Indicates that the mth TL has a hidden conflict with the mth TL, otherwise Hidden collision means that the mth TL and the mth TL use the same RB at the same time, and the send or receive tuple of one TL is within the communication range of the tuple in the other TL;

[0043] 4) For the matrix G CH Transform and get the matrix

[0044] 5) For the matrix and Transform to get the adjacency matrix

[0045] Furthermore, in step S3, the conflict hypergraph model is represented as is the vertex set, is a hyperedge set, a hyperedge set yes The association matrix of the conflict hypergraph model is given by express, Respectively The number of vertices and The number of hyperedges, any element H(v,e) in H takes the following value:

[0046]

[0047] Among them, H(v,e)=1 means that vertex v and hyperedge e are associated, that is, hyperedge e contains vertex v.

[0048] Furthermore, the step S6 specifically includes the steps of:

[0049] S61. Create multiple parallel client agents. The actor network and critic network of each client agent use different deep neural networks. The actor network and critic network are represented as and π θ , where w and θ represent the parameters of the actor network and the critic network respectively;

[0050] S62, each client agent first obtains the latest parameters of the global actor network and the global critic network {w, θ} from the global server, then calculates the gradient of its loss function on the replay buffer, and then updates the parameters of its local actor network and critic network based on the calculated gradient of the loss function;

[0051] S63. Each client agent sends its updated parameters to the global server. The global server performs weighted averaging on the parameters of all client agents, updates the global model with the weighted average parameters, and broadcasts the updated global model parameters to all client agents.

[0052] Further, in step S62, the loss functions of the actor network and the critic network of each client agent are defined as:

[0053]

[0054]

[0055] Among them, π θ (a t |s t ; θ) represents the state s t Next, perform action a t The probability of Indicates state s t Next, perform action a t compensation, Status t Next, perform action a t The Q value, Represents a parameter value, π θ (a′|st+1 ; θ) represents the state s t+1 The probability of executing action a' under Indicates state s t The Q value of executing action a'.

[0056] Further, in step S62, the loss function and The gradients are expressed as:

[0057]

[0058]

[0059] Among them, Q(s t ,a t ; w) π(a|s t+1 ; θ) represents the state s t+1 The probability of executing action a, Q(s t+1 ,a;w) represents the state s t+1 The Q value obtained by performing action a.

[0060] Furthermore, in step S62, the parameters of the actor network and critic network of the k-th client agent are updated using the following formulas:

[0061]

[0062]

[0063] Among them, η is the learning rate of parameter update, t represents the current moment, and t-1 represents the previous moment.

[0064] The resource allocation method for the NGMA IoE network supporting 6G provided by the present invention establishes a new conflict graph model for the conflict between TLs and converts it into a conflict hypergraph model. Then, based on the conflict hypergraph model, a resource allocation Markov decision network with sequential decision-making characteristics is established. In addition, in order to reduce the amount of calculation by distributing the calculation load throughout the network and realize distributed learning, a federated average advantage actor-critic algorithm (FedAC) is proposed. Simulation results show that the scheme has a higher network throughput in the NGMA IoE network supporting 6G. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a model example diagram of a NGMAIoE network supporting 6G provided by an embodiment of the present invention;

[0066] Figure 2 The embodiment of the present invention provides Figure 1 The corresponding graph model;

[0067] Figure 3 The embodiment of the present invention provides Figure 2 The corresponding conflict graph;

[0068] Figure 4 The embodiment of the present invention provides Figure 2 The corresponding conflict hypergraph;

[0069] Figure 5 is an architecture diagram of a FedAC-based Markov decision network provided by an embodiment of the present invention;

[0070] Figure 6 is a comparison diagram of convergence performance under different learning rates provided by an embodiment of the present invention;

[0071] Figure 7 is a comparison diagram of convergence performance under different discount factors provided by an embodiment of the present invention;

[0072] Figure 8 is a comparison chart of average SINRs of six methods provided by an embodiment of the present invention under different numbers of TLs;

[0073] Fig. 9 is a comparison chart of the maximum resource efficiency of the six methods provided by the embodiments of the present invention under different numbers of TLs;

[0074] Fig.10 is a comparison chart of average resource efficiency of six methods provided by the embodiments of the present invention under different numbers of TLs;

[0075] Fig.11 is a comparison chart of the maximum network throughput of the three methods provided by the embodiments of the present invention under different TL numbers;

[0076] Fig.12 This is a comparison chart of average network throughput under different numbers of TLs provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0078] The resource allocation method of the NGMA IoE network supporting 6G provided in an embodiment of the present invention comprises the steps of:

[0079] S1. Build a communication network architecture that supports 6G NGMA IoE network, where NGMA IoE network refers to the next generation multi-access Internet of Things.

[0080] S2. Establish a conflict graph model of NGMAIoE network during resource allocation based on the communication network architecture of NGMAIoE network;

[0081] S3, establishing a conflict hypergraph model based on the conflict graph model;

[0082] S4. Establishing a conflict-free resource allocation problem of NGMAIoE network based on the conflict hypergraph model;

[0083] S5, constructing a Markov decision network guided by a Markov decision process based on the conflict-free resource allocation problem;

[0084] S6, training the Markov decision network;

[0085] S7. The trained Markov decision network performs interference-free resource allocation according to the current state of the NGMA IoE network.

[0086] (1) System Model of NGMAIoE Network Supporting 6G

[0087] A. NGMA IoE network model supporting 6G

[0088] like Figure 1 As shown, this embodiment considers supporting 6G NGMA IoE, which includes an access point (AP) and M IoEDs (represented as a set The available D time-frequency resource blocks (RBs), which are the smallest unit of radio resources that can be allocated to the IoED, are considered to have the same value and are represented by the set In order to support large-scale connections in the network, the present invention adopts the power domain NOMA technology to allow CIDs to share RBs. CIDs that share the same RB in the same AP constitute a NOMA CIDG. AP allocates RBs to each CID and is responsible for communication between CIDs. In order to improve the resource utilization of the network, the CDMA mode of cognitive radio is adopted to allow DIDs to share resources with CIDGs. IoED is divided into M′ CIDGs and M″ DID pairs, respectively, which are composed of sets and In order to simplify the model, the present invention assumes that each CIDG includes two CIDs, denoted as CID1 and CID2, and the channel coefficient h1 of CID1 is less than the coefficient h2 of CID2. In addition, each DID pair is divided into a DID transmitter (DID T ) and D2D receiver (DID R). When allocating RBs to TLs, the present invention only considers one TL between CIDG and AP, because CIDs in CIDG share the same RBs. In the present invention, because IoED or AP belongs to different CIDG and DID pairs and is connected to different links when allocating RBs to TLs, AP, CIDG and DID are defined as tuples, and there are N tuples in total, including Tuple1 to Tuple2. N .

[0089] B. Communication Model

[0090] In NGMA IoE supporting 6G, the present invention takes into account large-scale fading and small-scale fading in the TL between IoEDs. These are simple and realistic approximations of real networks. The instantaneous channel coefficient of the transmission link on the dth RB at time t is:

[0091]

[0092] in, represents the frequency-dependent small-scale fading power component, which is exponentially distributed with unit mean. Represents large-scale fading effects, including shadow fading and frequency-independent path loss.

[0093] At time t, the signal to interference plus noise ratio (SINR) of the IoED in the m′th CIDG and the m″th D2D link receiver on the dth RB can be expressed as:

[0094]

[0095] in, and is a binary variable for RB allocation of the m'th CIDG and the m"th D2D link at time t. If This means that the dth RB is allocated to the m'th CIDG, otherwise In a similar way, This means that the dth RB is allocated to the m'th D2D link, otherwise P I Indicates the mth DID T The transmission power, P A Indicates the transmit power of the AP. i and β j are the powers allocated to the i-th and j-th CIDs in the CIDG respectively. and are the CIDs from AP to CIDG on the dth RB. i and CID j, from m” DIDs T to DID R , from DIDs T to DID R The channel gain of 2 Defined as the noise power.

[0096] Therefore, the transmission data rate of IoED can be given as follows:

[0097]

[0098] in, and is the transmission data rate of the CID in the m'th CIDG and the m"th D2D link on the dth RB.

[0099] C. Conflict Model

[0100] The present invention considers an unweighted undirected graph to represent the NGMA IoE supporting 6G, and analyzes the resource conflicts between TLs based on the graph. in is the set of vertices representing tuples (AP, CIDG, and DID), ε = {e1, e2, ..., e M} is an edge set representing a TL set, such as Figure 2 As shown. The correlation matrix G = {0,1} N×M Represents the relationship between edges and vertices, where G(v n ,e m )=1 means that the nth tuple initiates the mth TL, otherwise G(v n ,e m )=0.

[0101] In addition, in order to analyze the nature of resource conflicts between TLs, the present invention uses N(v n ) to represent the vertex v n The adjacent vertices (with v n vertices in the same edge associated with each other), d G (v n ) represents the degree v of the vertex n (that is, with vertex v n The number of associated edges), and define the mth TL and the mth The path between TLs is The specific definitions are as follows:

[0102] Path: The mth TL and the mth The path between two TLs is composed of a vertex sequence {v1,v2,...,v Y} represents, where v1 and TLem is associated, and v Y With TLe m is associated with, defined as If there is a path connecting them, then the mth TL and the mth TLs are considered accessible.

[0103] Simple path: If a vertex in the sequence appears only once, a path is considered simple and is denoted by Its length is expressed as

[0104] In NGMA IoE supporting 6G, there are connections between the mth and the There are various paths with different lengths for each TL, but the present invention only focuses on the shortest path Because it determines the accessibility between TLs. Based on the path definition, the mth TL and the The conflict conditions between TLs can be divided into two types as follows:

[0105] Direct conflict: The mth TL and the TLs use the same RB at the same time and have the same send or receive tuple, that is,

[0106] Hidden conflict: The mth TL and the TLs use the same RB at the same time, and the send or receive tuple of one TL is within the communication range of the tuple in another TL, that is,

[0107] It is worth mentioning that in NGMAIoE supported by 6G, there are two cases where resource conflicts do not occur, so they are not taken into account. The two cases are as follows:

[0108] Tuples that have stopped working and do not adopt any TL in NGMA IoE supporting 6G, which means that the vertices in the graph model are not associated with any edges, i.e., d G (v n )=0;

[0109] Tuples that communicate with each other are not in the communication area of ​​any other tuple in 6G NGMA IoE, which means that vertices are only associated with edges in the graph model, i.e., d G (v n )=1.

[0110] For conflicts between TLs in NGMA IoE supporting 6G, the direct conflict problem is essentially a typical edge coloring problem, which can be solved by the edge coloring algorithm in the graph. However, the hidden conflict problem deviates from the core essence of the typical edge coloring problem, so the typical edge coloring algorithm cannot solve it. Therefore, it is necessary to further establish a mechanism to analyze all conflicts between TLs.

[0111] (2) Hypergraph Model of NGMA IOE Supporting 6G with Interference

[0112] A. Conflict Graph Model

[0113] Suppose unweighted undirected graph Represents a conflict graph model, where is the set of vertices representing the TL, and is a set of edges that represent conflict relationships between TLs. The relationship between edges and vertices is represented by the adjacency matrix G C ={0,1} M×M Indicates that Indicates that the mth and mth in TL conflict, otherwise

[0114] Using the simple path length of direct conflict and hidden conflict, the present invention designs a conflict detection method through matrix operation and then constructs a conflict graph. Since the conflict graph is unweighted, the present invention first defines an element processing method in the matrix as follows:

[0115]

[0116] Where a represents the elements in the matrix and b is a constant.

[0117] To identify direct conflicts between TLs (i.e., the simple path length between TLs satisfies exist Figure 2 In , the operation is performed as follows:

[0118] G CD =(G T G)⊙1.(5)

[0119] Among them G CD ={0,1} M×M , Indicates that the mth TL directly conflicts with the mth TL, otherwise G is the incidence matrix of the conflict graph, G T is the transposed matrix of matrix G.

[0120] Considering that TL itself is detected as accessible in GCD, (i.e., the main diagonal elements are not 0), therefore, in order to obtain the direct conflict relationship between different TLs, the main diagonal elements are set to 0 as follows:

[0121]

[0122] where I is the identity matrix.

[0123] Similarly, to identify hidden conflicts between TLs (i.e., the simple path length between TLs satisfies Then similar operations are performed as follows:

[0124]

[0125]

[0126] Adjacency matrix G of the conflict graph C Including direct conflict and hidden conflict, based on equations (6) and (8), we get:

[0127]

[0128] Based on G C , we can get Figure 3 The conflict graph shown in Figure 1 presents the conflict relationship between TLs. For example, if the vertices connected to vertex v1 have the same coloring as vertex v1, they conflict with vertex v1. Due to the ultra-dense connection characteristics between vertices in the conflict graph, the network structure is highly complex. In addition, the edges in the conflict graph only connect two TLs, which means that the process of allocating conflict-free resources to TLs may be very cumbersome. Therefore, it is necessary to design a simplified conflict graph method so that the simplified conflict graph can analyze the relationship between multiple TLs at the same time while retaining the conflict information between them.

[0129] B. Conflict Hypergraph Model

[0130] Due to the ultra-densely connected nature of the conflict graph, there are many vertex subsets Make Every two vertices in are adjacent (i.e., ), therefore the present invention provides the following definition:

[0131] Therefore, the present invention provides the following definition:

[0132] Group: Subgraph is a clique if any two vertices are connected, i.e. N(v1)∩v2=v2, where N(v1)={v2|(v1,v2)∈ε sub} is the neighboring point set of v1.

[0133] Maximum group: One group is a maximal clique that satisfies the following conditions:

[0134] In order to express the relationship between multiple TLs, the present invention uses the concept of clique to introduce a conflict hypergraph. In this hypergraph, a hyperedge corresponds to a subset of vertices, where the vertices inside it are interconnected. The conflict hypergraph is denoted as where is the vertex set, is a hyperedge set, a hyperedge set yes A subset of . The hypergraph is represented by the incidence matrix express,

[0135]

[0136] Among them, H(v,e)=1 means that vertex v and hyperedge e are associated, that is, hyperedge e contains vertex v.

[0137] Since the connection relationships between vertices in a clique and a hyperedge are consistent, the conflict graph can be converted into a conflict hypergraph, such as Figure 4 As shown. In addition, the present invention converts the maximum group into a hyperedge to simplify the conflict graph as much as possible. The present invention uses a greedy algorithm to effectively determine the maximum group. In addition, the conflict-free resource allocation problem in NGMAIoE supported by 6G basically corresponds to the super-graph strong coloring problem, which means that a conflict-free resource allocation scheme can be obtained by solving the solution of the super-graph strong coloring problem, where the color of the vertex is equal to the RB of the TL.

[0138] For the super-graph coloring problem, the present invention defines the conflict degree at time t as To quantify the coloring conflict, it can intuitively reflect the degree of RB allocation conflict between TLs at time t. It can be defined as follows:

[0139]

[0140] Among them, H T represents the transpose of the hypergraph incidence matrix H, represents the RB matrix (or hypergraph coloring matrix) at time t, max(·,1) means that each element of the matrix (·) is maxed with 1, || ||1 represents the L1 norm, and X t The details are as follows:

[0141]

[0142] C. Problem Formulation

[0143] In solving the resource allocation problem (involving avoiding resource conflicts within NGMA IoE supporting 6G), the goal of the present invention is to design an effective resource allocation scheme while avoiding conflicts between TLs. Using the above definitions, the resource allocation optimization problem is constructed:

[0144]

[0145]

[0146] Where O represents the network throughput, An indicator variable indicating that the d-th RB is allocated to the m-th IoED at time t, This means that the dth RB is allocated to the mth IoED. The dth RB is not allocated to the mth IoED; R m,d represents the data transmission rate of the mth IoED allocated with the dth RB; represents the RB allocation conflict degree between TLs at time t, D RB represents a set of D RBs, represents a set of M IoEDs; max represents maximization, and st represents what needs to be satisfied;

[0147] Among them, the objective function (13a) is to maximize the network throughput, and the constraint (13b) ensures that the resources allocated to each TL are limited by not generating interference. Then, the constraint (13c) ensures that the resources allocated to each TL are limited by a single RB. Finally, the constraint (13d) ensures that the resources allocated to each TL are limited by the RB usage status. The proposed optimization problem is a non-convex optimization (CO) problem with NP-hardness, which is usually difficult to solve.

[0148] (3) Conflict-free resource allocation based on Federated Reinforcement Learning (FRL)

[0149] In order to solve the NP-hard CO problem in the optimization problem, the present invention first models the non-convex optimization problem of NGMA IoE supported by 6G as a discrete-time MDP (Markov decision model) problem. Then, the FRL method is studied to obtain the optimal solution of the optimization problem in NGMA IoE supported by 6G. These key elements of the MDP model, namely, the state space, action space, reward function and value function are defined as follows.

[0150] A. MDP model for NGMA IoE supporting 6G

[0151] In the proposed MDP model, the reward function r tIt is designed to help the FRL agent maximize the network throughput in the NGMAIoE supporting 6G. If there are packets reaching the target rate within the corresponding TL, the present invention considers giving positive feedback as a reward. When the NGMA IoE supporting 6G generates interference, the degree of conflict is considered as a negative feedback of the reward. Since the FRL agent can learn from the total reward during training and evaluate the action-state space, the reward function r for each time step t t It can be defined as:

[0152]

[0153] Therefore, the optimization goal of the FRL agent is to maximize the cumulative discounted reward from the rewards at time step t It is expressed as:

[0154]

[0155] in, represents the discount factor.

[0156] The goal of the proposed resource allocation scheme is to maximize the expected cumulative discounted reward To optimize the goal, obtain the optimal allocation strategy π * ,in Represents the expectation operator.

[0157] Therefore, the FRL algorithm can formulate the resource allocation problem (13) as an MDP problem, which can be formulated as:

[0158]

[0159] After each execution action of the FRL agent, the resulting conflicts, RB allocations, and transmission data rate changes can be abstracted as MDP state transitions at time t, including state s t 、Action a t , Reward t and the next state s t+1 The state of the environment can be observed by NGMA IoE supported by 6G and consists of several parts: the association matrix of the conflict hypergraph The set of all TL conflicts The aggregate of all TL data transfer rates The collection of RB allocations of all TLs In addition, the FRL agent can obtain the state information of NGMA IoE supported by 6G with limited signaling overhead. Although the state includes heterogeneous data, the deep neural network can effectively extract valuable information from such heterogeneous data to learn the best strategy. The present invention defines the state s tTo indicate whether NGMAIoE supporting 6G meets its network throughput requirements at time t, it can be expressed as:

[0160]

[0161] Then, after performing resource allocation in real time, NGMAIoE supported by 6G can be t Transition to the next state s t+1 Therefore, the FRL agent takes resource allocation action a based on policy π t , the policy π is from the state space at time step t Mapping of resource allocation actions in NGMA IoE supporting 6G t It is expressed as:

[0162] a t ={0,1} M×D . (18)

[0163] The FRL agent is dedicated to finding the optimal action for each state and introduces the action-value (Q-value) function To quantitatively evaluate the long-term value of each action. Action value function (Q value) Q π (s,a) is the state s under strategy π t The expected return can be decomposed into the Bellman equation, namely:

[0164]

[0165] γ represents the discount factor, Q π (s t ,a t ) indicates state s t Next, perform action a t The obtained Q value, Q π (s t+1 ,a t+1 ) indicates state s t+1 Next, perform action a t+1 The obtained Q value is Expressing expectation.

[0166] State value function V π (s) can be defined as follows:

[0167]

[0168] V π (s t ) represents the state value at time t, π(a′|s t ) is represented as state s t The probability of executing action a', Q(s t,a′) means in state s t Next, take action a' and obtain the Q value.

[0169] Therefore, the advantage function can be written as:

[0170]

[0171] T represents the total transmission time.

[0172] B. Parameter Learning

[0173] In the Federated Average Advantage Actor-Critic (FedAC) algorithm, the actor network and critic network of the client agent adopt different deep neural networks to solve the exploration and exploitation problems respectively.

[0174] The actor network and critic network are represented as and π θ , where w and θ represent the parameters of the actor network and the critic network, respectively. Therefore, the loss function of the actor network is defined as (22), and the loss function of the critic network can be defined as (23).

[0175]

[0176] and

[0177]

[0178] Among them, π θ (a t |s t ; θ) represents the state s t Next, perform action a t The probability of Indicates state s t Next, perform action a t compensation, Indicates state s t Next, perform action a t The Q value, Represents a parameter value, π θ (a′|s t+1 ; θ) represents the state s t+1 The probability of executing action a' under Indicates state s t The Q value of the next action a'.

[0179] In order to minimize the two loss functions, the neural network parameters use the FedAvg algorithm to increase the speed of parameter learning. Therefore, the global actor network and critic network of FedAC are obtained by aggregating the models of all client agents involved at the end of each learning round. Therefore, the minimized loss function of the global actor network and critic network can be expressed as:

[0180]

[0181]

[0182] Among them, p k Represents the weight coefficient of the k-th client agent. represents the loss function of the actor network of the k-th client agent, represents the loss function of the critic network for the k-th client agent.

[0183] Then, to obtain its local actor and critic network, each client agent of FedAC first obtains the latest parameters of the global actor and critic network {w,θ} from the global server. Therefore, the client agent calculates its gradient on the replay buffer as shown in Equations (25) and (26), and then updates the parameters of its local actor and critic network as shown in Equations (27a) and (27b).

[0184]

[0185]

[0186]

[0187]

[0188] η is the learning rate of parameter update, t represents the current moment, and t-1 represents the previous moment.

[0189] The present invention uses adaptive moment estimation (Adam) to provide a stochastic approximation of gradient descent to minimize the loss function of the actor network and the critic network. After the client agent trains the local actor network and the critic network, the global server of FedAC broadcasts the updated global model {w,θ} to all client agents. Assuming there are K agents, the server is responsible for receiving the neural network parameters of all client agents. 6G supports the FedAC-based algorithm of NGMA IoE as follows Figure 5 As shown, the specific steps include:

[0190] S61. Create multiple parallel client agents. The actor network and critic network of each client agent use different deep neural networks. The actor network and critic network are represented as and π θ , where w and θ represent the parameters of the actor network and the critic network respectively;

[0191] S62, each client agent first obtains the latest parameters of the global actor network and the global critic network {w, θ} from the global server, then calculates the gradient of its loss function on the replay buffer, and then updates the parameters of its local actor network and critic network based on the calculated gradient of the loss function;

[0192] S63. Each client agent sends its updated parameters to the global server. The global server performs weighted averaging on the parameters of all client agents, updates the global model with the weighted average parameters, and broadcasts the updated global model parameters to all client agents.

[0193] C. Computational complexity

[0194] The computational complexity of FedAC takes into account the local model training of the client agent and the local model aggregation of the server. Since a single client network model is trained using random samples from its own local buffer, the complexity of RL local update is Q((T value +T policy )×N lr ), depends on the value network complexity Tvalue, the strategy network complexity T policy and the number of local training Nlr. The complexity of the aggregation process is O(K) because it grows linearly with the number of client agents K. Therefore, the complexity of the FedAC method is Furthermore, the larger the number of client agents, the faster the FedAC method can be trained.

[0195] (4) Experimental simulation

[0196] A. Experimental Setup

[0197] The simulation experiments of the present invention are implemented using Python 3.9.13 and Pytorch 2.0.1 on a computing platform with an Intel(R) Xeon(R) Gold 6242R CPU with 3.1GHz and 64GBRAM and an Nvidia GeForce RTX 3080 Ti running Ubuntu 18.04 LTS OS. The detailed simulation parameters are summarized in Table 1. In addition, the deep Q network (DQN) and Q-Learning (Q learning) method are used for performance simulation to verify the efficiency of the proposed FedAC method (Proposed).

[0198] Table 1 Simulation parameters

[0199] parameter value TL quantity [20,30,40,50,60] Carrier frequency 2GHz bandwidth 1MHz IoED Antenna Height 1.5m IoED Antenna Gain 3dBi IoED receiver noise figure 9dB IoED transmission power 23dBm Noise Power -114dBm Correlation distance 100m Fast decline Rayleigh fading Fast fading update frequency Every 1ms Replay buffer size 10000 Mini-batch size 500 Optimal control Adam

[0200] B. Convergence of the FedAC Algorithm

[0201] The present invention first studies some hyperparameters that affect the convergence performance of the FedAC scheme. Specifically, the present invention analyzes the cumulative reward obtained by the method of the present invention as the number of iterations increases. Figure 6 and Figure 7 , where different learning rates and discount factors are applied during training.

[0202] Figure 6 The cumulative rewards with different learning rates are shown. In order to evaluate the performance of the proposed scheme under different learning rates while keeping other factors unchanged, the number of TLs is set to M = 20 and the learning rate α is set to 0.0001, 0.0003, 0.0005, 0.0007 and 0.009 respectively. Figure 6 As shown, when α = 0.0005, the proposed method has better cumulative reward.

[0203] Figure 7 The cumulative rewards with different discount factors are shown. Subsequently, the effects of various discount factors on the proposed method are studied. The discount factor γ ranges from 0.68 to 0.98, while the learning rate α is fixed to 0.0005. The convergence of different discount factors is shown in Figure 7 As shown. It is worth noting that when γ = 0.78, the cumulative reward exceeds the other cases. It is worth mentioning that the discount factor is a trade-off between future and immediate rewards. Therefore, the present invention sets the learning rate α to 0.0005 and the discount factor γ to 0.78.

[0204] C. Proposed Method

[0205] In the simulation, the present invention evaluates the average signal to interference and noise ratio, resource efficiency and network throughput obtained by the three schemes.

[0206] Figure 8 The average SINR of all TLs is plotted as a function of the number of TLs, and the proposed method is compared with the baseline method (-mean). Compared with the baseline method, the proposed method achieves the goals of high signal-to-noise ratio, high resource efficiency, and high network throughput by adjusting parameters. Figure 8 As shown, the average SINR is provided for NGMA IoE supporting 6G. A higher average SINR value means that the entire network is not severely degraded and experiences less interference. The proposed method is superior to the Q-Learning and DQN methods because it can allocate RBs more evenly. RBs can also be allocated by the FRL agent to avoid interference, thereby enhancing the average SINR of NGMA IoE supporting 6G. In contrast, the FedAC method focuses on avoiding interference.

[0207] Fig. 9 The maximum network resource efficiency of different schemes varies with the number of TLs. It is obvious that the method proposed in the present invention is always better than the two baseline methods. Compared with the other two baseline methods, the maximum network resource efficiency of the method proposed in the present invention is improved by 19.08% and 24.1% on average. The experimental results show that the scheme can effectively improve the resource utilization of NGMA IoE supporting 6G.

[0208] Fig.10 The figure shows how the network average resource efficiency of different schemes changes with the number of TLs. It is obvious that the scheme proposed in the present invention is always better than the other two baseline schemes. However, when the number of TLs increases, the resource efficiency begins to show differences (because the more TLs there are, the better resource allocation performance is required). Compared with the other two baseline schemes, the network average resource efficiency of the method proposed in the present invention is improved by 16.14% and 20.51% on average. In contrast, the FedAC method can effectively improve the resource efficiency of NGMA IoE supporting 6G.

[0209] Fig.11 The maximum network throughput (measured in Mbps) of different schemes is shown as a function of the number of TLs. For the three schemes, as the number of TLs increases, the maximum network throughput also increases, and the proposed method has the highest increase in maximum network throughput, and the proposed method is always better than the two baseline methods. These observations show that avoiding conflicts in NGMA IoE supporting 6G can effectively improve the maximum network throughput.

[0210] Fig.12 The average network throughput (measured in Mbps) of different schemes varies with the number of TLs. Fig.12It is observed that as the number of TLs increases, the average network throughput of the three schemes also increases. In addition, the method proposed in the present invention always has a higher average network throughput. Specifically, when the number of TLs is set to 60, the average network throughput of the method proposed in the present invention is improved by 170.17Mbps and 285.77Mbps compared with the other two methods. These observations show that avoiding conflicts in NGMA IoE supporting 6G has practical significance for the average network throughput.

[0211] In summary, the resource allocation method for the NGMA IoE network supporting 6G provided by the embodiment of the present invention has the following main contributions:

[0212] 1) In order to avoid resource conflicts between TLs in NGMA IoE under 6G, the present invention analyzes the resource conflict relationship and classifies the conflict types. Then, according to the characteristics of resource conflicts between TLs, a conflict graph model is established, and a matrix operation method is designed to construct a conflict graph, which intuitively represents the conflicts between TLs. In addition, the present invention uses clique and hypergraph theory to transform the conflict graph into a conflict hypergraph, and analyzes the resource conflicts between multiple TLs at the same time. The conflict-free resource allocation problem is transformed into a strong coloring problem of a hypergraph, and a method for calculating the color conflict degree is designed;

[0213] 2) As NGMA IoE resource allocation is a computationally intensive, non-convex NP-hard combinatorial optimization problem, a resource allocation Markov decision network with sequential decision-making characteristics is established. In particular, under the premise of avoiding resource conflicts between TLs, a reward function is designed based on the characteristics of high network throughput, so that the reinforcement learning (RL) agent can obtain efficient resource allocation solutions in large-scale dense interference network scenarios;

[0214] 3) In order to reduce the computational load and achieve distributed learning by distributing the computational workload throughout the network, a Federated Average Advantage Actor-Critic (FedAC) algorithm is proposed to handle the conflict-free resource allocation problem of NGMA IoE supporting 6G and accelerate the training process in large-scale dense interference network scenarios. A distributed training framework based on FedAvg is proposed, which consists of a multi-actor-critic local network and a local global network.

[0215] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A resource allocation method for a NGMA IoE network supporting 6G, characterized in that: Includes steps: S1. Build a communication network architecture that supports NGMA IoE network for 6G. NGMA IoE network refers to the next generation multi-access Internet of Things. The communication network architecture of the NGMA IoE network includes 1 access point, namely AP, M IoT devices, namely IoEDs, D resource blocks, namely RBs, and M transmission links, namely TLs, where the M IoEDs are divided into M′ CIDGs and M″ DID pairs. The CIDs sharing the same RB in the same AP constitute a CIDG, and two DIDs for D2D communication constitute a DID pair. CID refers to a shared cellular IoED, and DID refers to an IoED using D2D communication. D2D communication refers to device-to-device communication. Each DID pair is divided into a DID transmitter, namely a DID T and D2D receiver i.e. DID R ;AP allocates RBs to each CID and is responsible for communication between CIDs; The CDMA mode using cognitive radio allows DID to share resources with CIDG; when allocating RBs for TL, only one TL is considered between CIDG and AP because CIDs in CIDG share the same RBs; S2. Establish a conflict graph model of NGMA IoE network in resource allocation based on the communication network architecture of NGMA IoE network; The step S2 specifically includes the following steps: S21. Using an unweighted undirected graph Represents the communication network architecture of the NGMA IoE network. is a vertex set representing N tuples, including AP, CIDG and DID, ε={e1,e2,...,e M } is the edge set representing the TL set; and the incidence matrix G = {0,1} N×M express The relationship between the edges and vertices of G(v n ,e m )=1 means that the nth tuple initiates the mth TL, otherwise G(v n ,e m )=0; S22. Establish a conflict graph model based on the conflict relationship between TLs in the NGMA IoE network in is the set of vertices representing TL, is the set of edges representing the conflict relationship between TLs, The relationship between edges and vertices is represented by the adjacency matrix G C ={0,1} M×M Indicates that Indicates the mth and the first conflict, otherwise In step S22, G C Obtained by the following steps: 1) Transform the matrix G to obtain the matrix G CD =(G T G)⊙1, the operation⊙ is defined as: Where a represents the element in the matrix, b is a constant; G CD ={0,1} M×M , Indicates the mth TL and the TL directly conflict, otherwise Direct conflict refers to the mth TL and the TLs use the same RB at the same time and have the same send or receive tuple; G T is the transposed matrix of matrix G; 2) Transform the matrix G to obtain the matrix I is the identity matrix; 3) For the matrix Transform and get the matrix Representation Matrix The transposed matrix of Indicates the mth TL and the TL hides the conflict, otherwise Hidden conflicts refer to the mth TL and the TLs use the same RB at the same time, and the send or receive tuple of one TL is within the communication range of the tuple in another TL; 4) For the matrix G CH Transform and get the matrix 5) For the matrix and Transform to get the adjacency matrix S3, establishing a conflict hypergraph model based on the conflict graph model; In step S3, the conflict hypergraph model is represented as is the vertex set, is a hyperedge set, a hyperedge set yes The association matrix of the conflict hypergraph model is given by express, Respectively The number of vertices and The number of hyperedges, any element H(v,e) in H takes the following value: Where H(v,e) = 1 means that vertex v and hyperedge e are associated, that is, hyperedge e contains vertex v; S4. Establishing a conflict-free resource allocation problem of the NGMA IoE network based on the conflict hypergraph model; In step S4, the conflict-free resource allocation problem is constructed as follows: Where O represents the network throughput, An indicator variable indicating that the d-th RB is allocated to the m-th IoED at time t, This means that the dth RB is allocated to the mth IoED. The dth RB is not allocated to the mth IoED; R m,d represents the data transmission rate of the mth IoED allocated with the dth RB; represents the RB allocation conflict degree between TLs at time t, D RB represents a set of D RBs, represents a set of M IoEDs; max represents maximization, and st represents what needs to be satisfied; The definition is as follows: Among them, H T represents the transposed matrix of the hypergraph association matrix H of the conflict hypergraph model, max(·,1) means that each element of the matrix (·) is subjected to a max operation with 1, || ||1 represents the L1 norm, X t Represents the resource allocation matrix at time t: S5, constructing a Markov decision network guided by a Markov decision process based on the conflict-free resource allocation problem; The state at time t is defined as: in, The hypergraph association matrix H representing the conflict hypergraph model and the set of conflict situations of all TLs The aggregate of all TL data transfer rates The collection of RB allocations of all TLs Resource allocation action a at time t t It is expressed as: a t ＝{0,1} M×D ； The reward function r at time t t Defined as: Under strategy π, state s t Next, perform action a t The state action value function Q π (s t ,a t ) is defined as: Under strategy π, state s t The state value function V under π (s t ) is defined as: Among them, Q π (s t+1 ,a t+1 ) indicates state s t+1 Next, perform action a t+1 The obtained Q value is represents the expectation, V π (s t ) represents the state value at time t, π(a′|s t ) is represented as state s t The probability of executing action a', Q(s t ,a′) means in state s t Next, take action a' to obtain the Q value, γ represents the discount factor; S6, training the Markov decision network; S7. The trained Markov decision network performs interference-free resource allocation according to the current state of the NGMA IoE network.

2. The resource allocation method for the NGMA IoE network supporting 6G according to claim 1, characterized in that: The step S6 specifically comprises the following steps: S61. Create multiple parallel client agents. The actor network and critic network of each client agent use different deep neural networks. The actor network and critic network are represented as and π θ , where w and θ represent the parameters of the actor network and the critic network respectively; S62, each client agent first obtains the latest parameters of the global actor network and the global critic network {w, θ} from the global server, then calculates the gradient of its loss function on the replay buffer, and then updates the parameters of its local actor network and critic network based on the calculated gradient of the loss function; S63. Each client agent sends its updated parameters to the global server. The global server performs weighted averaging on the parameters of all client agents, updates the global model with the weighted average parameters, and broadcasts the updated global model parameters to all client agents.

3. The resource allocation method for the NGMA IoE network supporting 6G according to claim 2, characterized in that: In step S62, the loss functions of the actor network and critic network of each client agent are defined as: Among them, π θ (a t |s t ; θ) represents the state s t Next, perform action a t The probability of Indicates state s t Next, perform action a t compensation, Status t Next, perform action a t The Q value, Represents a parameter value, π θ (a′|s t+1 ; θ) represents the state s t+1 The probability of executing action a' under Indicates state s t The Q value of executing action a'.

4. The resource allocation method for the NGMA IoE network supporting 6G according to claim 3, characterized in that: In step S62, the loss function and The gradients are expressed as: Among them, Q(s t ,a t ; w) π(a|s t+1 ; θ) represents the state s t+1 The probability of executing action a, Q(s t+1 ,a;w) represents the state s t+1 The Q value obtained by performing action a.

5. The resource allocation method for the NGMA IoE network supporting 6G according to claim 4, characterized in that: In step S62, the parameters of the actor network and critic network of the kth client agent are updated using the following formulas: Among them, η is the learning rate of parameter update, t represents the current moment, and t-1 represents the previous moment.

Images