Wireless decentralized distributed learning system throughput optimization method based on graph reinforcement learning

Abstract

According to the method, a graph neural network technology and a deep reinforcement learning technology are integrated, a wireless distributed network structure formed by a plurality of terminals is focused on, and in order to effectively improve the network throughput in a wireless decentralized distributed learning system and reduce the interference influence, graph reinforcement learning is explored to be introduced into wireless network communication, so that the network throughput of the wireless decentralized distributed learning system is improved. In order to realize high-timeliness and low-delay network communication, a scheme for optimizing the throughput of a D2D network communication system by using a graph reinforcement learning technology is designed. According to the method, the network topology structure, the power distribution and the mutual interference relation influence are considered into the system, and the throughput of the system is optimized and the required communication task is completed with relatively low calculation complexity.

Description

BACKGROUND

[0001] Facing the exponential growth of wireless data traffic and the urgent demand for lower latency and higher spectral efficiency, Device-to-Device (D2D) communication emerges as a key supporting technology in wireless distributed networks, which allows devices in geographical proximity to communicate directly with each other without the need for central proxy nodes such as base stations and access points. The main advantages of D2D communication include making full use of network resources and effectively improving the robustness of single-point failure, thereby significantly improving system capacity and response speed. However, the resource allocation problem in D2D communication, such as the link scheduling problem and the channel resource allocation problem involving integer variables, is usually computationally complex and difficult to obtain a global optimal solution. Traditional methods such as branch-and-bound (B&B) algorithm have high computational complexity and are often time-consuming for large-scale networks. Therefore, these global optimization algorithms are not suitable for solving practical problems in wireless distributed networks.

[0002] Although many effective methods have been proposed to improve the transmission performance of wireless networks, there is still insufficient attention to the interaction between network topology and wireless resource allocation. Based on classical optimization theory, problems involving multiple variables are usually decomposed into several sub-problems and solved by alternating optimization variables. However, this method usually converges slowly and has high computational complexity. At the same time, in the communication scenario where multiple devices share wireless resources, the interference problem cannot be ignored, which greatly increases the difficulty of solving the problem. Graph Neural Network (GNN) is a class of machine learning models specifically designed for processing graph-structured data. Graph-structured data is a data representation form containing nodes and edges. Graph neural networks can capture the complex relationships and connection patterns between nodes. Deep Reinforcement Learning (DRL) combines the perception ability of Deep Learning and the decision-making ability of Reinforcement Learning, solving the "perception-decision" closed-loop problem that previous artificial intelligence cannot handle.

[0003] In view of the above background, the present application combines graph neural network technology and deep reinforcement learning technology, focuses on the wireless distributed network structure composed of multiple terminals, and designs a scheme for optimizing the system throughput of D2D network communication system using graph reinforcement learning technology, with the goal of minimizing system throughput. With low computational complexity, the system throughput is optimized and the required communication tasks are completed. SUMMARY

[0004] The purpose of the present application is that although the existing methods start to use the combination of graph neural network and deep learning to solve the resource allocation problem in wireless network, they almost do not consider the influence of network topology, power allocation and mutual interference relationship. Based on this, the present application explores the introduction of graph reinforcement learning into wireless network communication in order to effectively improve the network throughput in wireless decentralized distributed learning system and reduce the interference influence, and realizes the network communication with high timeliness and low time delay.

[0005] Method introduction: the implementation steps of graph reinforcement learning are as follows:

[0006] Step 1: model the wireless distributed network as a graph structure, that is, represent the graph as a set of nodes and edges, where the nodes represent entities in the network, such as user equipment or relay nodes. The edge represents the connection relationship between devices, such as communication link, interference link or potential connectable relationship.

[0007] Step 2: at time step t, assign a feature vector to each node to represent its dynamic state information. The features include but are not limited to: channel state information (CSI), traffic load, transmit power, geographic location information, etc. The output of the two steps is a graph data structure containing the current network global topology and dynamic state information, which is the state s t .

[0008] Step 3: the decision-making process of graph reinforcement learning agent, which uses GNN to process irregular topological relationship. The graph state constructed in steps 1 and 2 is input into a pre-trained graph reinforcement learning agent. The agent can use graph neural network to process the input graph. GNN efficiently extracts deep feature embedding of each node in the network through the message passing and aggregation mechanism between neighbor nodes, and finally generates a global embedding vector representing the state of the whole network. The global embedding vector obtained above is input into the policy network of the DRL agent. Based on the embedding representation of the current network state, the policy network outputs a joint optimization action a t .

[0009] Step 4: the nodes in the network execute the joint action a t generated by step 3, the network environment enters a new state s t+1 , and generates a reward signal. The reward function is designed to maximize the overall throughput of the network as the primary goal. The agent continuously samples data from the buffer through offline training, updates the parameters of its policy network and value network, and learns the policy that maximizes the cumulative reward. After training, the final agent model can be deployed online to perform real-time and intelligent throughput optimization for the wireless distributed network. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 Single communication service diagram

[0013] Figure 2 A diagram illustrating the single decision-making process;

[0015] Figure 3 Flowchart of each jump; Figure 4 Convergence plot of invention loss Figure 5 Q-value convergence plot of the proposed invention Figure 6 Comparison chart of the proposed invention and various algorithms Detailed Implementation

[0016] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0017] Examples of the present invention are as follows Figure 1 As shown, a wireless distributed network system composed of multiple terminal nodes is used to jointly optimize power control and user association to maximize the total network throughput. The method specifically includes the following steps:

[0018] Step 1:

[0019] In a decentralized distributed learning (D2D) network system, there are N terminal devices. In this example, N=10. We define the source node, target node, and relay node in the network system as nodes in a graph structure. The communication links and interference links between nodes are defined as edges in the graph structure. The feature vectors of a node include transmit power, the number of currently associated users, geographical coordinates, and channel quality. This step ultimately outputs a graph data structure, representing the state of the agent.

[0020] Step 2:

[0021] To address the challenges of complex node relationships and continuous decision states in decentralized wireless networks, this study proposes a dedicated architecture that integrates graph computation and sequence learning. This architecture enhances environmental awareness through a three-level processing mechanism. The proposed architecture is as follows: Figure 2As shown, the system comprises a graph relation encoding module, a cross-temporal memory module, and a graph-based decision-making module. A graph attention network is used as its front-end feature extraction module. The graph attention network layer aggregates the neighbor information of each node through a multi-head attention mechanism to generate updated node embeddings. After processing by the multi-layer graph attention network, global average pooling is performed on the embedding vectors of all base station nodes to obtain a global feature vector representing the entire network state. This global feature vector is then used as input through a multilayer perceptron, outputting a continuous action vector. This action vector represents the joint optimization decision made at the current time step.

[0022] Step 3:

[0023] Considering the inclusion of a memory loop module in the proposed architecture, a Deep Recurrent Q-Learning (DRQN) algorithm is employed to update the parameters θ, which encompass all parameters from the encoding, memory, and decision modules. During training, the agent uses an ε-greedy policy; after executing an action, the environment returns a reward and a binary termination signal, followed by updating the hidden state using a GRU. Due to the memory loop module, each training sample should be a sequence of continuous state transitions. This ensures that the agent fully captures the relationships between state transitions when using the memory loop module. Finally, we express the loss function as the mean squared error between the Q-network estimate and its target.

[0024] Step 4:

[0025] For decentralized wireless networks, we use stochastic gradient descent (SGD) to update the network parameters θ, while the target network parameters are updated synchronously using a soft update method. After training, each terminal node in the decentralized wireless network system can run this strategy. When any two nodes need to establish a communication service, the optimization process starts from the designated source node. At each hop, the current node acts as the agent, selects the transmit power according to the trained strategy, and determines the next hop node; subsequently, the agent's identity is transferred to that node, and the above steps are repeated until the termination condition is met. The final routing path and power allocation scheme are jointly formed by all nodes making decisions locally in sequence.

Claims

1. A method for optimizing throughput in a wireless decentralized distributed learning system based on graph reinforcement learning, characterized in that: Focusing on a wireless distributed network structure composed of multiple terminals, and aiming to minimize system throughput, this paper considers the impact of network topology, power allocation, and mutual interference. It integrates graph neural network technology and deep reinforcement learning techniques to design a scheme that uses graph reinforcement learning to optimize the throughput of a D2D network communication system. This scheme optimizes system throughput and completes the required communication tasks with relatively low computational complexity.

2. The throughput optimization method for a wireless decentralized distributed learning system based on graph reinforcement learning according to claim 1, characterized in that: Wireless distributed networks can be modeled as a graph structure, which is represented as a set of nodes and edges. Nodes represent entities in the network, such as user equipment or relay nodes. Edges represent connections between devices, such as communication links, interference links, or potential connectable relationships.

3. The throughput optimization method for a wireless decentralized distributed learning system based on graph reinforcement learning according to claim 2, characterized in that: At time step t, a feature vector is assigned to each node to represent its dynamic state information. This feature includes, but is not limited to, channel state information (CSI), traffic load, transmit power, and geographic location information. The output of these two steps is a graph data structure containing the current global network topology and dynamic state information, which serves as the state s of the reinforcement learning agent. t .

4. The throughput optimization method for a wireless decentralized distributed learning system based on graph reinforcement learning according to claim 2, characterized in that: The decision-making process of a graph reinforcement learning agent utilizes a graph neural network (GNN) to handle irregular topological relationships. The graph state constructed in steps 1 and 2 is input into a pre-trained graph reinforcement learning agent. The agent processes the input graph using a graph neural network. The GNN efficiently extracts deep feature embeddings of each node in the network through message passing and convergence mechanisms between neighboring nodes, ultimately generating a global embedding vector representing the entire network state. This global embedding vector is then input into the policy network of the DRL agent. Based on the embedding representation of the current network state, the policy network outputs a joint optimization action 'a'. t .

5. The throughput optimization method for a wireless decentralized distributed learning system based on graph reinforcement learning according to claim 2, characterized in that: Nodes in the network execute the joint action a generated in step 3. t The network environment has entered a new state. t+1 The system generates a reward signal. A reward function is designed with the primary objective of maximizing the overall network throughput. The agent continuously samples data from a buffer during offline training, updating the parameters of its policy and value networks to learn a policy that maximizes cumulative rewards. After training, the resulting agent model can be deployed online for real-time, intelligent throughput optimization of wireless distributed networks.

Images