A large-scale relay selection and power control method for satellite-to-ground relay Internet of Things

Abstract

A method for large-scale relay selection and power control of satellite-to-ground relay Internet of Things belongs to the field of satellite communications. The satellite-to-ground relay Internet of Things scenario is analyzed, and the relay selection and power control problem of large-scale terminal node uplinks is defined; the decision-making process of each terminal node is modeled as a Markov game; the deep features of the terminal nodes are obtained through the autoencoder, so as to calculate the similarity between the terminal nodes, and the terminal node grouping results are obtained through clustering; the weighted average field deep reinforcement learning method is adopted to make each terminal node interact with the environment, and communicate its actions between neighboring nodes, and each intelligent agent makes decisions based on the environmental state, the weighted average action of the same group of intelligent agents, and the local features. The present invention uses the weighted average field mechanism of similarity between terminal nodes to simplify the mutual interaction between large-scale terminal nodes into the interaction of the weighted average action of each terminal node and its terminal nodes in the same group, which greatly reduces the dimension of joint action and improves the convergence effect and efficiency of multi-agent reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the field of satellite communications, and in particular relates to a large-scale relay selection and power control method for a satellite-to-ground relay Internet of Things based on weighted mean field reinforcement learning. Background Art

[0002] IoT terminal nodes, which are characterized by miniaturization, low power consumption, and low cost, often have difficulty meeting the requirements of direct communication with satellites. Therefore, for ground areas where terminal nodes are densely distributed, in order to lower the transmission power threshold of IoT terminal nodes and reduce the energy consumption of terminal nodes, ground relays can be deployed to assist in data transmission between IoT terminal nodes and satellites. The advantage of the relay mechanism is that it reduces the manufacturing and maintenance costs of IoT terminal nodes. On the one hand, considering that the solar cells at the terminal nodes have a rated number of cycles, the relay mechanism can effectively reduce their power consumption and thus extend the life of the terminal nodes; on the other hand, the terminal nodes do not need to have strong solar cells, and the failure rate is low. The operation and maintenance personnel only need to regularly check whether the solar power supply device of the relay node is working properly, which reduces the maintenance workload. The existing relay selection and resource allocation methods in the satellite-to-ground relay IoT scenario have the following deficiencies:

[0003] (1) Most studies are based on the assumption that the global channel state information is known and stable over a long period of time. However, in actual systems, the communication channel between the satellite and the ground is greatly affected by factors such as weather (rainfall, snowfall) and open space interference, and exhibits dynamic fluctuations. In addition, due to the long satellite communication link, the feedback CSI is prone to expire due to delays. Therefore, the premise of knowing the global, real-time and accurate CSI is difficult to meet in the satellite IoT scenario.

[0004] (2) It is difficult to efficiently coordinate the relay node selection and resource allocation decisions of large-scale terminal nodes. The coverage of ground networks is often at the kilometer level, but the coverage of satellite networks reaches thousands of kilometers. Due to the wide coverage, there are a large number of IoT nodes and different application types. The time required for solving existing research based on optimization or heuristic methods increases exponentially or even exponentially with the scale of terminal nodes. Therefore, the massive number of connected nodes poses a huge challenge to existing relay selection and resource allocation methods.

[0005] (3) Satellite propagation delay is large and the number of terminal nodes is huge, making it unsuitable to adopt a centralized control mechanism.

[0006] Multi-agent reinforcement learning regards each terminal node as a distributed decision-making agent. It does not rely on accurate channel quality information known in advance or accurate modeling of the channel model. Instead, it optimizes the strategy of sequential decision-making through environmental feedback. It is considered to be an effective way to deal with the dynamic communication environment problems in the satellite-to-ground relay Internet of Things. However, most of the existing methods based on multi-agent reinforcement learning adopt the mechanism of independent optimization strategy of agents, ignoring the mutual influence between agents. As the scale of terminal nodes increases, the mutual influence between agents increases significantly. Agents that do not consider the independent optimization strategy of other terminal nodes face serious environmental instability problems and are therefore difficult to converge. Summary of the invention

[0007] The present invention provides a large-scale relay selection and power control method for a satellite-to-ground relay Internet of Things. An autoencoder is used to learn terminal node features and calculate the similarity between terminal nodes. A similarity weighted average field mechanism is used to simplify the mutual interaction between large-scale terminal nodes into the interaction of the weighted average action of each terminal node and its terminal nodes in the same group, which greatly reduces the dimension of joint action and improves the convergence effect and efficiency of multi-agent reinforcement learning.

[0008] A satellite-to-ground relay Internet of Things large-scale relay selection and power control method comprises the following steps:

[0009] Step 1: Analyze the satellite-to-ground relay IoT scenario and define the relay selection and power control problems for large-scale terminal node uplinks;

[0010] Step 2: Model the relay selection and power control decision process of each terminal node as a Markov game;

[0011] Step 3: Obtain the deep features of the terminal nodes through the autoencoder, thereby calculating the similarity between the terminal nodes, and obtaining the terminal node grouping results through clustering;

[0012] Step 4: Use the weighted mean field deep reinforcement learning method to enable each terminal node to interact with the environment and communicate its actions between neighboring nodes. Each agent makes decisions based on the environment state, the weighted average action of the same group of agents, and local features.

[0013] The present invention adopts the above technical solution and has the following advantages compared with the prior art:

[0014] 1. Considering the large scale of terminal nodes in the satellite Internet of Things scenario, the autoencoder is used to learn the terminal node features and calculate the similarity between terminal nodes. The similarity weighted average field mechanism is used to simplify the mutual interaction between large-scale terminal nodes into the weighted average interaction of each terminal node with its group of terminal nodes, which greatly reduces the dimension of joint action and improves the convergence effect and efficiency of multi-agent reinforcement learning.

[0015] 2. Model the uplink access process of terminal nodes in the satellite-to-ground relay Internet of Things, including relay selection and power control. First, considering that IoT terminal nodes in remote areas are usually powered by batteries or solar energy mechanisms and have limited energy, the energy utilization of terminal nodes is taken as one of the optimization goals. Secondly, since switching relays requires coordination signaling overhead, the present invention considers the switching cost caused by relay switching. Finally, since different IoT applications have different QoS requirements, relay selection and power control strategies need to meet the QoS constraints of heterogeneous satellite IoT terminal nodes. This model helps to extend the life of IoT terminal nodes in the field while meeting diverse transmission requirements.

[0016] 3. Synchronize the global state with low communication volume to improve the coordination effect of each agent. On the one hand, each agent only reports the immediate benefits of the previous moment, and with the help of satellite broadcasting capabilities, the global state is synchronized between agents with low communication overhead, so that the agent can understand the information of other agents; on the other hand, in the decision-making process, each agent learns the decision-making rules of other agents contained in the temporal global state information through the Transformer module, further improving the coordination ability. Using these two mechanisms, the cooperation and coordination between agents is promoted at a low additional communication cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of an application scenario of the present invention, namely, a satellite-to-ground relay Internet of Things scenario;

[0018] Figure 2 It is a schematic diagram of the satellite-to-ground relay transmission process in the scenario of the present invention;

[0019] Figure 3 It is the autoencoder network structure for performing terminal node similarity calculation in the present invention;

[0020] Figure 4 It is the structure diagram of Q-value neural network. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below:

[0022] A satellite-to-ground relay Internet of Things large-scale relay selection and power control method comprises the following steps:

[0023] Step 1: Analyze the satellite-to-ground relay IoT scenario and define the relay selection and power control problems for large-scale terminal node uplinks;

[0024] Step 2: Model the relay selection and power control decision process of each terminal node as a Markov game;

[0025] Step 3: Obtain the deep features of the terminal nodes through the autoencoder, thereby calculating the similarity between the terminal nodes, and obtaining the terminal node grouping results through clustering;

[0026] Step 4: Use the weighted mean field deep reinforcement learning method to enable each terminal node to interact with the environment and communicate its actions between neighboring nodes. Each agent makes decisions based on the environment state, the weighted average action of the same group of agents, and local features.

[0027] In step 1, the satellite-to-ground relay IoT scenario is analyzed, and the relay selection and power control problem of large-scale terminal node uplinks is defined. The specific process is as follows:

[0028] like Figure 1 As shown, the LEO constellation provides IoT data backhaul services, and N IoT terminal nodes form a collection M relay nodes form a relay set Since IoT applications mainly use uplinks for data transmission, this paper focuses on uplinks. The system operation process can be discretized into a time sequence containing equal time slices.

[0029] The signal transmission power of the relay node to the satellite is constant, and different types of relay nodes have different transmission powers. Each terminal node has different QoS requirements for data transmission rate due to different business applications. Let the matrix W = [w m,n ], represents the relay selection matrix of the terminal node, where w m,n ∈{0,1},w m,n =1 indicates terminal node U n Select access relay node R m A terminal node can only access one relay node, while a relay node can serve multiple terminal nodes.

[0030] For the terminal node U n To solve the power control problem, the present invention discretizes the continuous transmission power variable into a series of power levels. That is, the terminal node can Select the appropriate transmission power from U for terminal n The maximum transmission power, N p is the number of gears for quantizing the transmission power. Let the matrix P = [p k,n ], represents the power control scheme of the terminal node, where p k,n ∈{0,1}, if p k,n =1 represents the terminal node U n Select the kth transmission power level, the terminal's transmission power is

[0031] Satellite channel is also a kind of wireless channel, so it is necessary to consider the large-scale fading and small-scale fading of the wireless channel. Its channel gain can be expressed as: in Represents satellite and relay node R m Small-scale fading on the link follows the shadow Rice distribution; GL m,S Including satellite and relay node R m The free space loss, transmission gain and reception gain between can be expressed as:

[0032]

[0033] where λ c is the carrier wavelength, Represents the relay node R at time t m The distance from the satellite, θ m,S Represents the relay node R m The angle between the link to the satellite and the center point of the satellite beam, G S (θ m,S ) represents the satellite receiving antenna gain, G m Represents the relay node R m The transmit antenna gain.

[0034] Terminal Node U n To relay node R m The ground wireless channel gain between

[0035]

[0036] Among them, GL n,m Includes free space loss, transmit antenna gain of the terminal node, and receive antenna gain of the relay node; Represents small-scale fading, which obeys the Rayleigh distribution.

[0037] The communication process from the ground IoT terminal node to the satellite is as follows: Figure 2 As shown, it can be divided into three stages.

[0038] The first stage is the decision-making stage, in which the relay node and terminal node transmission power are selected through the relay selection and power control algorithm; the second stage is the access relay stage. If the access is made for the first time or a relay switch occurs, it is necessary to establish data transmission with the relay communication. The third stage is the data transmission stage, which is divided into two time slots.

[0039] In the first time slot, the ground IoT terminal node U n Send data to the selected relay node R m . Relay node R mThe received terminal node U n The signal sent is

[0040]

[0041] Where P n Represents the terminal node U n The transmission power, n m (t) represents the relay node R m The Gaussian white noise at the receiving antenna has a mean of 0 and a variance of

[0042] In the second time slot of data transmission, the selected relay node R m The received signal is directly amplified and sent to the satellite. The amplification factor can be expressed as

[0043]

[0044] The satellite receives the signal from the relay node R m The signal can be expressed as

[0045]

[0046] Therefore, in the second time slot at time t, the satellite receives the signal from R m Relay forwarding U n The signal-to-noise ratio can be expressed as

[0047]

[0048]

[0049]

[0050] in Represents relay node R m The terminal node U received at n The signal-to-noise ratio of the signal; Represents only considering the relay node R m When sending a signal to a satellite, the signal-to-noise ratio received at the satellite.

[0051] Terminal Node U n The signal passes through the relay node R m The maximum transmission rate that can be achieved is:

[0052]

[0053] in and are the signal-to-noise ratios between the terminal node and the relay, and between the relay and the satellite, respectively. is the relay node R at time t m With terminal node U n Each relay node allocates its bandwidth to each terminal node according to the demand ratio of the connected terminal nodes, that is, Where D n Represents the terminal node U n QoS requirements for transmission rate, D i All D n The index when summing, B m Represents relay node R m The downlink has a total bandwidth of is the IoT terminal node U at time t i The relay selection result.

[0054] Each terminal node in the system needs to meet the minimum QoS requirement of its IoT application, which is expressed as

[0055]

[0056] The sum of the energy utilization rates of each terminal node is

[0057]

[0058] in For the terminal node U n The transmit power at time t.

[0059] At the same time, considering the protocol signaling cost required to switch the selected relay node, it will cause communication overhead. The number of switching occurrences can be expressed as Assuming that the communication overhead in a switch is η, the system switching cost from time t-1 to time t can be expressed as

[0060] Therefore, the overall long-term benefit of the system can be expressed as

[0061] In summary, the relay node selection and power control problem studied in this invention can be modeled as the following optimization problem:

[0062]

[0063]

[0064]

[0065]

[0066] The optimization variables W and P represent the relay selection and power control results of each terminal node respectively. The first constraint indicates that any terminal node can only access one relay node; the second constraint indicates that the user's transmission rate must meet its minimum QoS requirement. The third constraint indicates that in order to ensure that the terminal node U n The transmission quality, its signal-to-noise ratio should exceed the minimum threshold δ th , i.e. SINR n ≥δ th Otherwise, no resources will be allocated for this transmission and the terminal node request will be rejected.

[0067] Step 2: Model the relay selection and power control decision process of each terminal node as a Markov game, including the design of state characteristics, action space, immediate benefits and neural network structure, as follows:

[0068] 2.1 State Characteristics

[0069] The state of each IoT terminal node contains the environmental information it observes that is related to relay node selection and power control decisions. In the present invention, each terminal node is in a cooperative mode, and through collaboration, it selects a relay node that is more suitable for itself, and gives up the relay node that is not suitable for itself to the terminal node that needs it more. In order to achieve collaboration, interaction needs to be established between terminal nodes. The present invention allows each terminal node to report its own revenue to the satellite for aggregation in each time slice. After aggregation, the satellite broadcasts this revenue information to form the global state shared by each terminal node at the next moment. The advantage of this mechanism is that it realizes the shared state of each intelligent agent with a relatively small additional communication overhead. The interaction of this information is conducive to improving the collaborative effect between intelligent agents. Each terminal node is based on the global state information s and its own feature information Make decisions.

[0070] 2.1.1 Sharing state information

[0071] Shared state information s = [R t-T ,…,R t-1 ] includes the sequence of the instantaneous benefits of each agent at the previous T time. t-1 represents the instantaneous income of each terminal node at the previous moment. At time t-1, each terminal node agent will convert its instantaneous income into The uplink is aggregated to the satellite and the At the beginning of the next time slice, the satellite broadcasts the decision-making agent at each terminal node. Using the time series information of each agent's benefits as global state information helps to learn the time series rules contained therein without increasing the communication cost, potentially explore the decision-making rules of each agent, and to a certain extent reduce the problem of environmental instability faced by multi-agent reinforcement learning.

[0072] 2.1.2 Self-characteristic information

[0073] Each relay needs to consider its own data transmission requirements when performing relay selection and power control. Therefore, the characteristic information of agent n It must include the QoS requirements of the terminal node at the current moment The terminal node's last action and the immediate benefits achieved Right now

[0074] contains the relay selection action of this agent at time t-1 It helps the intelligent agent understand the relay node selected at the last moment, thereby keeping the relay node at the last moment unchanged and reducing the signaling cost required to switch the relay node.

[0075] The state space of the method proposed in the present invention does not include channel quality. The main reason is that in the satellite-ground collaborative relay IoT scenario of large-scale terminal nodes, there are a large number of links of three types: terminal node-satellite, terminal node-relay, and relay-satellite. Obtaining the CSI of the global link through information exchange or broadcasting will result in higher additional communication overhead, reducing the feasibility of the algorithm. The algorithm proposed in the present invention does not rely on CSI, but uses the environmental feedback optimization strategy of the instant benefit at the previous moment. In each time slice, the satellite only needs to broadcast the instant benefit vector R of each terminal node, which effectively reduces the required communication volume compared to reporting and broadcasting the CSI of each link.

[0076] 2.2 Action Space

[0077] For the distributed IoT terminal node relay selection problem, let Indicates the coverage terminal node U n The set of relay nodes, so the action space (i.e. the set of optional actions) selected by the relay is

[0078] For the power control problem, each terminal node can determine the transmission power under the limit of its maximum power. The optional power set is The action space of power control is

[0079] Therefore, the action space of agent n is To facilitate subsequent multi-agent training, the action a is represented as a one-hot encoding form, and the action dimensions of each agent are unified by zero padding, that is, the action positions corresponding to all unselectable relay nodes are filled with zero. After the zero padding operation, the action a of all agents n Unified to M×N p A vector of 0,1 elements of dimension .

[0080] 2.3 Immediate Profits

[0081] Since the optimization goal takes into account the energy utilization of data transmission and the relay switching cost, n These two parts also need to be considered when designing the instant benefit of the terminal node U n Facing status n Select action a n The benefit can be expressed as r n =EE(s n ,a n )-price(s n ,a n ), where EE(s n ,a n ) represents the terminal node U n Energy utilization rate, price(s n ,a n ) represents the switching cost required for the terminal node Un to change the relay node.

[0082] Terminal Node U n The energy utilization rate can be expressed as:

[0083]

[0084] in It is the minimum data transmission rate for a node to meet the QoS requirement. If the QoS requirement cannot be met, the data transmission benefit is 0.

[0085] Terminal Node U n The cost of switching relay nodes can be expressed as

[0086]

[0087] in Action for a node Action component in the relay selection problem.

[0088] Step 3: Obtain the deep features of the terminal through the autoencoder, thereby calculating the similarity between the terminals and obtaining the terminal grouping results through clustering. The specific process is as follows:

[0089] Nodes connected to the same relay need to allocate the bandwidth of the relay node on demand, so there is a certain competition relationship and the impact between them is more significant, while the impact between IoT terminal nodes connected to different relay nodes is smaller. Considering that intelligent agents facing similar conditions tend to choose the same relay node and have a greater impact on each other, the similarity of relay selection between nodes is used as the weight of the weighted average field.

[0090] The relay node selection problem of IoT terminal nodes is affected by their own data transmission QoS requirements. n , its distance to each relay node d n,m Influence and reflected in historical decisions Therefore, the above elements are selected as features, and the decision results of a period of historical time are used to form a feature vector to measure the similarity between nodes as the basis for their possibility of accessing the same relay node, thereby obtaining the weighted average weight. It can be expressed as: in Indicates that before T sim The average action selected by the relay nodes at time t.

[0091] The deep semantic features h of the terminal node are obtained through the autoencoder n , and use cosine similarity to calculate the similarity matrix between terminal nodes Furthermore, the neighbor division of each IoT terminal node is obtained by performing K-means clustering on the similarity matrix Sim. The overall process is as follows:

[0092]

[0093]

[0094] Figure 3 The schematic diagram of the autoencoder network structure for terminal node similarity calculation, where the sparse autoencoder contains two modules, encoding and decoding. The encoding module can be expressed as h = F (x) and is responsible for projecting the feature x into a low-dimensional space to obtain the deep feature h.

[0095]

[0096] The decoding module is o=G(h), which reconstructs the deep feature h into the original data, that is,

[0097]

[0098] in is the parameter of the autoencoder. In order to learn the nonlinear semantic features of the terminal nodes, the present invention uses a multi-layer neural network consisting of an encoding layer and a decoding layer, such as Figure 3 shown.

[0099] The loss function of network training is the reconstruction loss between input features and output, that is,

[0100]

[0101] After the neural network is trained to converge, the feature C of each terminal node is n Input the encoder of the neural network to obtain the deep feature hn , and use this deep feature to calculate the similarity matrix Sim between terminal nodes using cosine similarity = [sim i,j ]∈, in

[0102]

[0103] The Sim matrix is ​​used to improve the average action calculation formula of the mean field mechanism to:

[0104]

[0105] Among them, the neighbor division of each IoT terminal node It is obtained by performing K-means clustering on the similarity matrix Sim.

[0106] Step 4: Use the weighted average field deep reinforcement learning method to enable each terminal to interact with the environment and communicate its actions between neighboring nodes. Each agent makes decisions based on the environment state, the weighted average action of the same group of agents, and local features. The specific process is:

[0107] The core idea of ​​the mean field mechanism is to simplify the interaction between an agent and all other agents into the average interaction between the agent and its neighboring agents. 2 The specific mechanism is to decompose the Q-value function based on joint actions into the form of bilateral interactions between the agent and its neighboring average field agents, namely:

[0108]

[0109] in is the set of neighbors of agent n. After decomposition, the dimension of the Q-value function is greatly reduced, and the interaction between agents is maintained.

[0110] Since in the joint decision problem of relay selection and power control, the terminal nodes are isomorphic and the action a is one-hot coded, we define is the average action of other terminal nodes in the same group,

[0111]

[0112] The Q-value function Q of each agent interacting with each other n (s,a) can be further simplified as follows:

[0113]

[0114] That is, for each agent n, its interaction with each other agent can be approximately simplified to the interaction between agent n and a virtual agent, which represents the average effect of all neighboring agents.

[0115] During the training process, the Q value network can be updated by the following formula

[0116]

[0117] Among them, γ is the discount factor, α is the set learning rate, and r n is the revenue of the nth agent, and s′ is the state at the next moment after all agents choose their actions. Mean field V value function It can be expressed as

[0118]

[0119] in are the strategy of the nth agent and the strategies of other agents respectively.

[0120] Therefore, using the mean field approximation, multi-agent reinforcement learning can be transformed into solving the optimal strategy π of the central agent n n The average action of all agents’ n neighbors represents the influence of all neighboring agents on agent n. The average field V value function of each agent is expressed as Therefore, the Q-value function of each agent can be updated, that is,

[0121]

[0122] Where r(s,a) is the set of benefits of each agent when the action set of each agent is a in state s, and P is the average The probability distribution of the process.

[0123] The training of each agent uses the DQN algorithm, and the state value function represented by the neural network The update formula can be expressed as

[0124]

[0125]

[0126] in is the mean field V value function of state s′, y n It is the calculation process of the state value function loss.

[0127] In each decision-making stage, the probability of selecting each action can be obtained through the Q value of each action, that is, the strategy

[0128]

[0129] where β is the temperature parameter. By iterating the average neighbor actions of all agents and its own strategy Alternate optimization.

[0130] The above formula can be used to iteratively train the Q-value network until convergence, so that each agent can learn about the average action of the surrounding The optimal strategy π n . Thus, multiple agents reach Nash equilibrium. in represents the joint decision of all agents except agent n is the optimal strategy of the nth agent, π * is the set of optimal strategies achieved by all agents.

[0131] Compared with independently trained MARL, the core advantage of the mean field mechanism is that it takes into account the impact of other agents’ actions on it, thus maintaining the stability of the environment and reducing the difficulty of convergence of agent training.

[0132]

[0133]

[0134]

[0135] The overall structure of the Q value network of the MARL relay selection algorithm proposed in the present invention is as follows: Figure 4 As shown in the figure, the time series historical information state is used as the input of the Transformer module, and the feature information of the current moment is input and extracted through the fully connected layer, and then spliced ​​with the time series information features, so that the network can simultaneously perceive the knowledge contained in the time series global state information and the feature information of each agent. The Transformer module structure is shown in the figure. Figure 4 As shown in the figure, with the help of the attention mechanism, it can deeply mine the implicit information in the time series data. Compared with time series models such as Long Short-Term Memory (LSTM), it is more suitable for processing time series data with high-dimensional features. Due to the large scale of IoT terminal nodes, the global state information composed of the instant benefit vector has a high dimension, and the Tranformer structure is used to mine its features. For example, when the number of IoT terminal nodes N = 240, the global state vector is a time series sequence of 240-dimensional features, and it is difficult for ordinary LSTM to effectively learn such high-dimensional features.

Claims

1. A method for large-scale relay selection and power control of satellite-to-ground relay Internet of Things, characterized in that The steps include: Step 1: Analyze the satellite-to-ground relay IoT scenario and define the relay selection and power control problem for large-scale terminal node uplinks. The specific process is as follows: LEO constellation provides data backhaul services, and N IoT terminal nodes form a collection M relay nodes form a relay set Let the matrix W = [w m,n ], represents the relay selection matrix of the terminal node, where w m,n ∈{0,1},w m,n =1 indicates terminal node U n Select access relay node R m ; A terminal node can only access one relay node, while a relay node can serve multiple terminal nodes; For the terminal node U n The power control problem is to discretize the continuous transmission power variable into a series of power levels, that is, the terminal node is in the discretized power set Select the appropriate transmission power from U for terminal n The maximum transmission power, N p is the number of gears for quantizing the transmission power; let the matrix P = [p k,n ], represents the power control scheme of the terminal node, where p k,n ∈{0,1}, if p k,n =1 represents the terminal node U n Select the kth transmission power level, the terminal node's transmission power is At the same time, considering the protocol signaling cost required to switch the selected relay node, it will lead to communication overhead; assuming that the communication overhead in a switch is η, the communication switch cost from time t-1 to time t is obtained, and the overall long-term benefit of communication at time t is obtained; Step 2: Model the relay selection and power control decision process of each terminal node as a Markov game; Step 3: Obtain the deep features of the terminal nodes through the autoencoder, thereby calculating the similarity between the terminal nodes, and obtaining the terminal node grouping results through clustering; Step 4: Use the weighted mean field deep reinforcement learning method to enable each terminal node to interact with the environment and communicate its actions between neighboring nodes. Each agent makes decisions based on the environment state, the weighted average action of the same group of agents, and local features.

Images