Graph attention network reinforcement learning anti-interference routing optimization method and device

Abstract

The invention discloses a graph attention network reinforcement learning anti-interference routing optimization method and device, and the method comprises the steps: a network node updates a corresponding DQN model according to the collected initialization weight data of a GAT network model, and selects an optimal routing path through the updated DQN model, the latest attention mechanism weight parameters are sent to the GAT network model; processing the node features to generate an updated node feature set, and obtaining updated attention mechanism weight parameters according to the updated node feature set and other initial state observation information to construct a GAT network model; carrying out GAT-DQN network model training based on the environment definition, the state definition, the action definition and the reward definition to obtain a trained GAT-DQN network model; and applying the trained GAT-DQN network model to a mobile ad hoc network node. The invention aims to enhance the anti-interference capability of a mobile ad hoc network (MANET).

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a graph attention network reinforcement learning anti-interference routing optimization method and device. Background Art

[0002] In recent years, the application of Mobile Ad hoc Network (MANET) has become increasingly widespread. With the in-depth research on wireless communication modules and the smaller size of sensors, mobile nodes can be used as a single task execution node, or multiple nodes can interact together to perform more complex tasks, while allowing personnel to avoid potential, life-threatening dangers. It is not only increasingly used in civilian scenarios, but also widely used in military scenarios, such as performing border patrols, disaster relief, and as a relay network in complex and dangerous environments.

[0003] Mobile Ad hoc Network (MANET) is built on these devices to form a decentralized, highly mobile network, which is particularly suitable for communication in complex and dangerous environments. However, it is difficult to maintain a stable and reliable communication network because the topology of the mobile network of nodes is constantly changing. Especially in military applications, the battery power, processing power and storage capacity of mobile devices are limited, which will affect the transmission performance and efficiency of the network. In order to ensure the anti-interference ability of the mobile ad hoc network, ensure the information transmission of each node in military applications, and improve the communication quality, it is crucial to develop an efficient and anti-interference mobile ad hoc network routing optimization technology.

[0004] Mobile ad hoc networks can be quickly established without the need for pre-existing network infrastructure. Communication between nodes does not rely on fixed routers or base stations. They adapt to the mobility of nodes and automatically reconfigure routes to maintain connectivity. When direct communication is not feasible, nodes can act as relays to forward messages to other nodes to achieve multi-hop communication. Moreover, mobile ad hoc networks do not rely on central control, and each node can participate in network maintenance and routing decisions. This type of network has attracted the attention of many experts in the field due to its flexibility and ability to operate in complex environments. However, the free movement of nodes also brings a significant weakness, which makes it difficult to maintain routing tables and ensure correct data forwarding.

[0005] Usually, the routing protocol of an ad hoc network needs to comprehensively consider factors such as network capabilities, node characteristics, and business needs. On the basis of realizing basic routing functions, it needs to improve network performance as much as possible to achieve efficient and fast data transmission. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, this paper proposes a graph attention network reinforcement learning anti-interference routing optimization method, which integrates the deep Q network (DQN) algorithm and the graph attention network principle, aiming to enhance the anti-interference ability of mobile ad hoc networks (MANET).

[0008] Another object of the present invention is to propose a graph attention network reinforcement learning anti-interference routing optimization device.

[0009] To achieve the above-mentioned purpose, the present invention proposes a graph attention network reinforcement learning anti-interference routing optimization method, comprising:

[0010] The network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, uses the updated DQN model to select the optimal routing path, and sends the latest attention mechanism weight parameters to the GAT network model;

[0011] The node features are processed to generate an updated node feature set, and the updated attention mechanism weight parameters are obtained according to the updated node feature set and other initial state observation information to construct the GAT network model;

[0012] Based on the environment definition, state definition, action definition and reward definition, the GAT-DQN network model is trained to obtain a trained GAT-DQN network model;

[0013] The trained GAT-DQN network model is used for mobile ad hoc network nodes. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters. The Q value sequence is calculated according to the current state of the channel, and the action corresponding to the maximum Q value is selected for communication. Finally, the selected action and the updated attention mechanism weight parameters are output for global optimization and synchronization.

[0014] The graph attention network reinforcement learning anti-interference routing optimization method of the embodiment of the present invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, processing node features to generate an updated node feature set includes:

[0016] Node feature set h = {h1,h2,…,h N}; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h′;

[0017] Apply a weight sharing parameter matrix W∈R to each nodeF′×F , perform attention mechanism a:R on each node F′ ×R F = R calculates the corresponding attention coefficient, e ij Indicates the importance of node j to node i ij =a(Whi,Whj).

[0018] In one embodiment of the present invention, for each training cycle, the GAT network model is executed, including:

[0019] For each node N, obtain the initial node state observation information Si = [(X j U(t),Y j U(t)),(REI,EDR),V i ],i belongs to N i ;

[0020] Initialize the adjacency matrix: M adj ;MLP network: MLP;Graph attention network: GAT;Initialize buffer: β;Get an initialized state;

[0021] For each node, calculate the feature value, aggregate the features, calculate the attention and obtain the attention weight;

[0022] Update the global model using the aggregated parameters;

[0023] Update the minimized predicted Q network:

[0024] If β.size>R then, the training is terminated and the target network θ is updated every C steps. Q′ =θ Q ;

[0025] Update the new attention mechanism weight parameter a to a′ and distribute it to all nodes.

[0026] In one embodiment of the present invention, the environment is defined as follows: each node in the region is always dynamically changing, and the actual network is regarded as an undirected graph, where N = {N1, N2...N 20} represents the number of network nodes, M={M ij} represents the link between each node, M represents the wireless communication link between network nodes i and j, and E adj = {I ij}N*N represents the adjacency matrix of nodes in the mobile ad hoc network;

[0027] State definition: For node N, its state s obtains the state observation information S of the initial node i =[(X j U(t),Yj U(t)),(REI,EDR),(V i , T i )],i belongs to N i . refers to the energy information of the current node, expressed as a vector S = [S1, S2, ..., S T ], where x and y represent the location information of the node, RE i Indicates the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated node available energy, V i Represents the speed of the node, T i Indicates the running time of the node;

[0028] Action definition: Action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1,a2,...,a M ], where a M represents the selection of the Mth weight value, where M represents the size of the channel set;

[0029] Reward definition: The reward r depends on the state of the selected node. If the reward is larger, the anti-interference ability of the selected path starting from the i node is the greatest.

[0030] In one embodiment of the present invention, the method further comprises:

[0031] Send the node initialization information to the GAT-DQN network model and update the node initial state;

[0032] Determine whether to send a data packet; if the data packet is sent normally, update the routing table and neighbor table, otherwise, do not send the data packet;

[0033] After updating the routing table and adjacency table, update the adjacency matrix to obtain the initial weight value of the node state;

[0034] Send TC message and calculate routing table;

[0035] Perform DQN deep reinforcement learning training on the current attention mechanism weight parameters, finally select the optimal attention mechanism weight parameters, update the node status, and determine whether to end;

[0036] If the optimal attention mechanism weight parameter is selected, the process ends.

[0037] To achieve the above-mentioned purpose, the present invention proposes, on the other hand, a graph attention network reinforcement learning anti-interference routing optimization device, comprising:

[0038] DQN network model construction module, which is used by network nodes to update the corresponding DQN model according to the collected initialization weight data of the GAT network model, select the optimal routing path using the updated DQN model, and send the latest attention mechanism weight parameters to the GAT network model;

[0039] The GAT network model construction module is used to process the node features to generate an updated node feature set, and obtain the updated attention mechanism weight parameters based on the updated node feature set and other initial state observation information to construct the GAT network model;

[0040] GAT-DQN network model training module, used to train the GAT-DQN network model based on environment definition, state definition, action definition and reward definition to obtain a trained GAT-DQN network model;

[0041] The model application output module is used to apply the trained GAT-DQN network model to the mobile ad hoc network node. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters. The Q value sequence is calculated according to the current state of the channel, and the action corresponding to the maximum Q value is selected for communication. Finally, the selected action and the updated attention mechanism weight parameters are output for global optimization and synchronization.

[0042] The graph attention network reinforcement learning anti-interference routing optimization method and device of the embodiment of the present invention, in order to improve the anti-interference ability in the network, by combining each node in the network topology, assigning different weights according to the attention mechanism, highlighting the characteristics of each node, and using DQN to generate the optimal node routing optimization strategy. This strategy can more efficiently transmit data by assigning different node weights, and improve the anti-interference performance of the mobile ad hoc network.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 is a flowchart of a method for optimizing anti-interference routing through reinforcement learning of a graph attention network according to an embodiment of the present invention;

[0046] Figure 2 is an overall communication network framework diagram according to an embodiment of the present invention;

[0047] Figure 3is a schematic diagram of GAT network model training according to an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of DQN network model training according to an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of a GAT network model according to an embodiment of the present invention;

[0050] Figure 6 is a schematic diagram of a first result obtained by applying the embodiment of the present invention to a node;

[0051] Figure 7 is a schematic diagram of a second result obtained by applying the embodiment of the present invention to a node;

[0052] Figure 8 It is a structural diagram of a graph attention network reinforcement learning anti-interference routing optimization device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0055] The following describes the graph attention network reinforcement learning anti-interference routing optimization method and device proposed according to an embodiment of the present invention with reference to the accompanying drawings.

[0056] Figure 1 is a flow chart of a method for optimizing anti-interference routing through reinforcement learning of a graph attention network according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0057] S1, the network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, uses the updated DQN model to select the optimal routing path, and sends the latest attention mechanism weight parameters to the GAT network model;

[0058] S2, processing the node features to generate an updated node feature set, and obtaining the updated attention mechanism weight parameters based on the updated node feature set and other initial state observation information to construct the GAT network model;

[0059] S3, training the GAT-DQN network model based on the environment definition, state definition, action definition and reward definition to obtain a trained GAT-DQN network model;

[0060] S4, the trained GAT-DQN network model is used for mobile ad hoc network nodes. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters. The Q value sequence is calculated according to the current state of the channel, and the action corresponding to the maximum Q value is selected for communication. Finally, the selected action and the updated attention mechanism weight parameters are output for global optimization and synchronization.

[0061] Figure 2 It is the overall communication network framework of the present invention. The communication framework is built through OPNET software to optimize the routing layer. The optimization of the routing layer is completed through the interaction between OPNET software and python. It is completed through the same TXT path. The GAT-DQN model is built in PYHTON, the optimal attention weight parameters are trained, and then the routing layer in OPNET software is interacted to complete the anti-interference routing optimization.

[0062] In one embodiment of the present invention, constructing a DQN network model includes:

[0063] Initialization: The network nodes receive the initial weight data from GAT and use this initial data for training loops.

[0064] Training cycle: Enter a loop and perform the following steps for each training cycle n=1,2,...:

[0065] a) Model update: After each training cycle, the node updates its DQN model based on the weight data and experience collected from GAT.

[0066] b) Routing path selection: Using the updated local model, the communication network selects appropriate nodes for transmission to optimize its communication efficiency and improve anti-interference ability.

[0067] c) Parameter sending: At a fixed time interval, such as X×T communication time frames, the node sends its latest attention mechanism weight parameters to the GAT network model, and then to the routing process of the communication framework in the OPNET software for further processing.

[0068] In one embodiment of the present invention, constructing a GAT network model includes:

[0069] Initialization module input: node feature set h = {h1,h2,…,h N}, where N is the number of nodes, F is the feature dimension of each node, and the output of the GAT network model is the updated node feature set h′. In order to obtain the updated feature matrix, a weight-sharing parameter matrix W∈R is first applied to each node. F′×F , and then the present invention performs an attention mechanism a:R on each node F′ ×R F = R calculates the corresponding attention coefficient, e ij Indicates the importance of node j to node i: e ij =a(Whi,Whj).

[0070] Aggregation cycle: The GAT network model performs the following steps for each training cycle m=1,2,3,...

[0071] a) For each node N, obtain the initial node state observation information Si = [(X j U(t),Y j U(t)),(REI,EDR),V i ],i belongs to N i .

[0072] b) Initialize the adjacency matrix: M adj ; MLP network: MLP; Graph Attention Network: GAT; Initialization buffer: β. Get an initialized state.

[0073] c) For each node, calculate the feature value, aggregate the features, calculate the attention, and obtain the attention weight.

[0074] d) Global model update: Use the aggregated parameters to update the global model.

[0075] e) Performance evaluation: Update the minimized prediction Q network: This may involve evaluating the model's performance on a specific task or condition.

[0076] f) Condition check: if β.size>R then, terminate the training and update the target network θ every C steps Q′ =θ Q .

[0077] g) Parameter distribution: Update the new attention mechanism weight parameter a to a ′ , and distributed to all nodes so that they can update their own models.

[0078] In one embodiment of the present invention, local model training includes:

[0079] Environment definition: The environment is a mobile ad hoc network area, that is, a dynamically changing network topology scenario. Each node in the area is always changing dynamically. The actual network is regarded as an undirected graph, where N = {N1, N2...N 20} represents the number of network nodes, M={M ij} represents the link between each node, M represents the wireless communication link between network nodes i and j, and E adj = {I ij}N*N represents the adjacency matrix of nodes in the mobile ad hoc network. That is, when I ij When =1, it means that the nodes are adjacent to each other and there is a communication link between the two nodes. Otherwise, they are not adjacent.

[0080] State definition: For node N, its state s obtains the state observation information S of the initial node i =[(X j U(t),Y j U(t)),(REI,EDR),(V i , T i )],i belongs to N i . refers to the energy information at the current node position, which can be expressed as a vector S = [S1, S2, ..., S T ], where x and y represent the location information of the node, RE i Indicates the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated node available energy, V i Represents the speed of the node, T i Indicates the running time of the node.

[0081] Action definition: Action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1,a2,...,a M ], where a M Represents the selection of the Mth weight value, where M represents the size of the channel set.

[0082] Reward definition: The reward r depends on the state of the selected node. If the reward is larger, the anti-interference ability of the selected path starting from the i node is the greatest.

[0083] Start training: During the transmission process of the mobile ad hoc network communication network, each node is first initialized, and each node interacts with each other. Each node can choose an action at the same time and get a reward in the next stage. This involves implementing action a. The current node obtains the link weight based on the node's own information and neighbor node information. Each node is affected by the weight and obtains the complete path of other nodes by adjusting the link cost in the network.

[0084] In one embodiment of the present invention, the trained GAT-DQN training model is used for a mobile ad hoc network node, including:

[0085] Experience pool input: The DQN model in the mobile ad hoc network node receives the data quadruple [s, a, r, s′] in the experience pool as input. Among them, s represents the current channel state value; a represents the current link weight, that is, the action taken during the current communication; r represents the aforementioned reward value, that is, the feedback obtained after taking the action; s′ represents the new state entered after the action, that is, the aggregate feature state of the selected node after the update.

[0086] Model learning and updating: The DQN model of the mobile ad hoc network node learns historical experience through a neural network to update model parameters and optimize node weights. The DQN algorithm randomly selects samples from the experience pool to eliminate the correlation between samples and accelerate the convergence of the model. In addition, the algorithm uses the target Q network to reduce the loss function and promote the stable convergence of the model. Every N training cycles, the parameters of the action Q network are updated to the target Q network, and the loss function is calculated.

[0087] Attention mechanism strategy execution: According to the current state, the DQN model outputs a sequence of Q(s,a;θ) values ​​corresponding to a channel. The attention mechanism strategy selects the corresponding channel for communication based on the maximum Q value in the sequence. At the same time, the mobile ad hoc network node sends its local model parameter θ to the GAT network model to facilitate the update and synchronization of the attention mechanism model.

[0088] The core structure of the Deep Q Network (DQN) algorithm includes four main parts: the estimated Q network, the target network, the experience replay mechanism, and the loss function. In this algorithm framework, the estimated Q network is responsible for estimating the Q value.

[0089] In this process, the mobile ad hoc node performs action a, that is, determines the state s based on the current node's position state information. This state is then updated to a new state s' according to the transition function, and the corresponding reward value r is obtained from the environment. The estimated Q network, as a system built by a neural network, takes the current state s as input and calculates the action Q value under this condition. In Q(s,a;θ), the weight value corresponding to the maximum Q value is regarded as the channel with the lowest interference probability.

[0090] The DQN neural network structure adopted in the present invention comprises an input layer, a hidden layer and an output layer. The input layer receives the defined state value, and the neural network generates the Q value of each channel in the output layer through a fully connected hidden layer. Selecting the channel with the largest Q value means selecting the channel with the lowest probability of interference. Through multiple training and learning processes, the parameters of the neural network are optimized to achieve effective fitting of the channel state and Q value, thereby determining the best channel selection.

[0091] In the DQN algorithm, the application of experience replay plays a vital role. It not only significantly reduces the correlation between data samples, but also helps ensure that the learning model can converge effectively. The experience replay mechanism achieves this goal by storing data for each state transition [s, a, r, s′]. When the amount of stored data exceeds the capacity limit of the experience pool, it automatically discards the earliest data records. Properly increasing the size of the experience pool helps reduce the risk of overfitting. During the training process of the model, the experience replay mechanism randomly extracts a certain amount of historical data from the pool for training and estimating the Q network.

[0092] The estimated Q network and the target Q network are structurally consistent, but they differ in function. The output value of the target Q network is Q(s′, a′; θ′), which retains the previous parameters of θ′, which is critical for calculating the loss function of the current estimated Q network θ parameters. In each interval of N training cycles, the parameters of the estimated Q network are copied to the target Q network. In this way, the DQN algorithm uses the target Q network and the estimated Q network to minimize the loss function and force the model to converge. The loss function uses:

[0093] It is understandable that the GAT network model of the present invention is as follows Figure 5 The training process is as shown in Figure 3 As shown, the specific steps are as follows:

[0094] 1. Pass the node initialization information to the GAT-DQN model and update the node initial state.

[0095] 2. At this time, determine whether to send a hello packet. If the packet is sent normally, update the routing table and neighbor table, otherwise, do not send it.

[0096] 3. After updating the routing table and adjacency table, update the adjacency matrix and obtain the initial weight value of the node state.

[0097] 4. Send TC message. Calculate routing table.

[0098] 5. Perform DQN deep reinforcement learning on the attention weights at this time. Finally, select the optimal node attention weights, update the node status, and determine whether to end.

[0099] 6. The process ends when the optimal weight is selected, otherwise continue to step 5.

[0100] Understandably, Figure 4 is a schematic diagram of DQN network model training according to an embodiment of the present invention, such as Figure 4 As shown:

[0101] 1) First, initialize two deep Q networks, one is the online network (DQN) and the other is the target network, input the initial state s and action a into the online Q network, and input the state s into the target Q network. At the same time, create an experience replay buffer to store the agent's experience (state, action, reward, new state).

[0102] 2) At each time step N, the agent selects action a based on the current state s and DQN. This process can adopt the ε-greedy strategy, that is, randomly select actions with a certain probability ε to explore, and select the current optimal action with a probability of 1-ε to exploit.

[0103] 3) After executing action a, a new state s′ and reward r are obtained after each action, and then the experience (s, a, s′, r) is stored in the experience replay buffer.

[0104] 4) Sample a batch of experiences from the experience replay buffer, calculate the target Q value for each experience, calculate the loss function L(θ) based on the target Q value and the output of DQN, and use backpropagation to update the DQN parameters θ.

[0105] 5) Copy the online Q network parameters of DQN to the target network every fixed C steps to maintain the stability of the target Q value.

[0106] 6) Repeat steps 2)-5) until the optimal weight is obtained.

[0107] Further, Figure 6 and Figure 7 It is a schematic diagram of the result obtained by applying the embodiment of the present invention to a node.

[0108] The beneficial effects of the present invention are:

[0109] (1) Enhanced communication efficiency and stability: By assigning weight coefficients of the attention mechanism to the nodes of the mobile ad hoc network and highlighting the characteristics of each node according to the different weight coefficients of each node, this method significantly improves the communication efficiency and stability of the mobile ad hoc network. Using machine learning models, the mobile ad hoc network can adapt to complex environmental conditions more efficiently, efficiently select available nodes in the case of high node dynamics, and maintain efficient communication connections.

[0110] (2) Adaptive and continuously optimized network performance: The GAT network model interacts with DQN continuously to update parameters, allowing the entire mobile ad hoc network system to continuously adjust and optimize itself. The network topology of the mobile ad hoc network will continue to change due to the movement of nodes. The GAT network model can quickly adapt to this dynamic change and adjust the node weight value according to the real-time situation to better respond to the transmission of the overall network information, thereby improving the performance and reliability of the entire system.

[0111] (3) Effective resource management and interference reduction: This method effectively reduces link overhead, increases the available energy of each node, reduces the total energy consumption of the entire network, and effectively manages network resources. In a highly dynamic network environment, it can not only efficiently select appropriate nodes for fast and good information transmission, but also effectively respond to potential security risks, such as intentional interference and attacks, thereby protecting the security and reliability of communications.

[0112] According to the anti-interference routing optimization method of graph attention network reinforcement learning in the embodiment of the present invention, the communication efficiency and stability of the mobile ad hoc network are tested using different methods under the same network topology environment. In the same scenario, the present invention can more efficiently adapt to the dynamic topology of the network, perform stable information transmission, enhance the anti-interference ability of the mobile ad hoc network, and show the high adaptability and efficiency of the method of the present invention.

[0113] In order to implement the above embodiment, Figure 8 As shown, this embodiment also provides a graph attention network reinforcement learning anti-interference routing optimization device 10, including:

[0114] DQN network model construction module 100, used for network nodes to update the corresponding DQN model according to the collected initialization weight data of the GAT network model, select the optimal routing path using the updated DQN model, and send the latest attention mechanism weight parameters to the GAT network model;

[0115] A GAT network model construction module 200 is used to process node features to generate an updated node feature set, and obtain updated attention mechanism weight parameters based on the updated node feature set and other initial state observation information to construct a GAT network model;

[0116] A GAT-DQN network model training module 300 is used to perform GAT-DQN network model training based on environment definition, state definition, action definition and reward definition to obtain a trained GAT-DQN network model;

[0117] The model application output module 400 is used to apply the trained GAT-DQN network model to the mobile ad hoc network node. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters, calculates the Q value sequence according to the current state of the channel, and selects the action corresponding to the maximum Q value for communication, and finally outputs the selected action and the updated attention mechanism weight parameters for global optimization and synchronization.

[0118] Furthermore, the node features are processed to generate an updated node feature set, including:

[0119] Node feature set h = {h1,h2,…,h N}; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h′;

[0120] Apply a weight sharing parameter matrix W∈R to each node F′×F , perform attention mechanism a:R on each node F′ ×R F = R calculates the corresponding attention coefficient, e ij Indicates the importance of node j to node i ij =a(Whi,Whj).

[0121] Furthermore, for each training cycle, the GAT network model is executed, including:

[0122] For each node N, obtain the initial node state observation information Si = [(X j U(t),Y j U(t)),(REI,EDR),V i ],i belongs to N i ;

[0123] Initialize the adjacency matrix: M adj ;MLP network: MLP;Graph attention network: GAT;Initialize buffer: β;Get an initialized state;

[0124] For each node, calculate the feature value, aggregate the features, calculate the attention and obtain the attention weight;

[0125] Update the minimized predicted Q network:

[0126] If β.size>R then, the training is terminated and the target network θ is updated every C steps. Q′ =θ Q； ;

[0127] Update the new attention mechanism weight parameter a to a′ and distribute it to all nodes.

[0128] Furthermore, the environment is defined as follows: each node in the region is always dynamically changing, and the actual network is regarded as an undirected graph, where N = {N1, N2...N 20} represents the number of network nodes, M={M ij} represents the link between each node, M represents the wireless communication link between network nodes i and j, and E adj = {I ij}N*N represents the adjacency matrix of nodes in the mobile ad hoc network;

[0129] State definition: For node N, its state s obtains the state observation information S of the initial node i =[(X j U(t),Y j U(t)),(REI,EDR),(V i , T i )],i belongs to N i . refers to the energy information of the current node, expressed as a vector S = [S1, S2, ..., S T ], where x and y represent the location information of the node, RE i Indicates the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated node available energy, V i Represents the speed of the node, T i Indicates the running time of the node;

[0130] Action definition: Action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1,a2,...,a M ], where a M represents the selection of the Mth weight value, where M represents the size of the channel set;

[0131] Reward definition: The reward r depends on the state of the selected node. If the reward is larger, the anti-interference ability of the selected path starting from the i node is the greatest.

[0132] Furthermore, it also includes:

[0133] Send the node initialization information to the GAT-DQN network model and update the node initial state;

[0134] Determine whether to send a data packet; if the data packet is sent normally, update the routing table and neighbor table, otherwise, do not send the data packet;

[0135] After updating the routing table and adjacency table, update the adjacency matrix to obtain the initial weight value of the node state;

[0136] Send TC message and calculate routing table;

[0137] Perform DQN deep reinforcement learning training on the current attention mechanism weight parameters, finally select the optimal attention mechanism weight parameters, update the node status, and determine whether to end;

[0138] If the optimal attention mechanism weight parameter is selected, the process ends.

[0139] According to the graph attention network reinforcement learning anti-interference routing optimization device of the embodiment of the present invention, the communication efficiency and stability of the mobile ad hoc network are tested using different methods under the same network topology environment. In the same scenario, the present invention can more efficiently adapt to the dynamic topology of the network, perform stable information transmission, enhance the anti-interference ability of the mobile ad hoc network, and show the high adaptability and efficiency of the method of the present invention.

[0140] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0141] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A graph attention network reinforcement learning anti-interference routing optimization method, characterized in that: include: The network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, uses the updated DQN model to select the optimal routing path, and sends the latest attention mechanism weight parameters to the GAT network model; The node features are processed to generate an updated node feature set, and the updated attention mechanism weight parameters are obtained according to the updated node feature set and other initial state observation information to construct the GAT network model; Based on the environment definition, state definition, action definition and reward definition, the GAT-DQN network model is trained to obtain a trained GAT-DQN network model; The trained GAT-DQN network model is used for mobile ad hoc network nodes. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters. The Q value sequence is calculated according to the current state of the channel, and the action corresponding to the maximum Q value is selected for communication. Finally, the selected action and the updated attention mechanism weight parameters are output for global optimization and synchronization.

2. The method according to claim 1, characterized in that The node features are processed to generate an updated node feature set, including: Node feature set h = {h1,h2,…,h N }; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h′; Apply a weight sharing parameter matrix W∈R to each node F′×F , perform attention mechanism a:R on each node F′ ×R F = R calculates the corresponding attention coefficient, e ij Indicates the importance of node j to node i ij =a(Whi,Whj).

3. The method according to claim 2, characterized in that For each training cycle, the GAT network model is executed, including: For each node N, obtain the initial node state observation information Si = [(X j U(t),Y j U(t)),(REI,EDR),V i ],i belongs to N i ; Initialize the adjacency matrix: M adj ;MLP network: MLP;Graph attention network: GAT;Initialize buffer: β;Get an initialized state; For each node, calculate the feature value, aggregate the features, calculate the attention and obtain the attention weight; Update the global model using the aggregated parameters; Update the minimized predicted Q network: If β.size>R then, the training is terminated and the target network θ is updated every C steps. Q′ =θ Q ; Update the new attention mechanism weight parameter a to a′ and distribute it to all nodes.

4. The method according to claim 3, characterized in that Environment definition: Each node in the region is always changing dynamically. The actual network is regarded as an undirected graph, where N = {N1, N2...N 20 } represents the number of network nodes, M={M ij } represents the link between each node, M represents the wireless communication link between network nodes i and j, and E adj = {I ij }N*N represents the adjacency matrix of nodes in the mobile ad hoc network; State definition: For node N, its state s obtains the state observation information S of the initial node i =[(X j U(t),Y j U(t)),(REI,EDR),(V i , T i )],i belongs to N i . refers to the energy information of the current node, expressed as a vector S = [S1, S2, ..., S T ], where x and y represent the location information of the node, RE i Indicates the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated node available energy, V i represents the speed of the node, T i Indicates the running time of the node; Action definition: Action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1,a2,...,a M ], where a M represents the selection of the Mth weight value, where M represents the size of the channel set; Reward definition: The reward r depends on the state of the selected node. If the reward is larger, the anti-interference ability of the selected path starting from the i node is the greatest.

5. The method according to claim 3, characterized in that: The method further comprises: Send the node initialization information to the GAT-DQN network model and update the node initial state; Determine whether to send a data packet; if the data packet is sent normally, update the routing table and neighbor table, otherwise, do not send the data packet; After updating the routing table and adjacency table, update the adjacency matrix to obtain the initial weight value of the node state; Send TC message and calculate routing table; Perform DQN deep reinforcement learning training on the current attention mechanism weight parameters, finally select the optimal attention mechanism weight parameters, update the node status, and determine whether to end; If the optimal attention mechanism weight parameter is selected, the process ends.

6. A graph attention network reinforcement learning anti-interference routing optimization device, characterized in that: include: DQN network model construction module, which is used by network nodes to update the corresponding DQN model according to the collected initialization weight data of the GAT network model, select the optimal routing path using the updated DQN model, and send the latest attention mechanism weight parameters to the GAT network model; The GAT network model construction module is used to process the node features to generate an updated node feature set, and obtain the updated attention mechanism weight parameters based on the updated node feature set and other initial state observation information to construct the GAT network model; GAT-DQN network model training module, used to train the GAT-DQN network model based on environment definition, state definition, action definition and reward definition to obtain a trained GAT-DQN network model; The model application output module is used to apply the trained GAT-DQN network model to the mobile ad hoc network node. The DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameters. The Q value sequence is calculated according to the current state of the channel, and the action corresponding to the maximum Q value is selected for communication. Finally, the selected action and the updated attention mechanism weight parameters are output for global optimization and synchronization.

7. The device according to claim 6, characterized in that The node features are processed to generate an updated node feature set, including: Node feature set h = {h1,h2,…,h N }; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h′; Apply a weight sharing parameter matrix W∈R to each node F′×F , perform attention mechanism a:R on each node F′ ×R F = R calculates the corresponding attention coefficient, e ij Indicates the importance of node j to node i ij =a(Whi,Whj).

8. The device according to claim 7, characterized in that For each training cycle, the GAT network model is executed, including: For each node N, obtain the initial node state observation information Si = [(X j U(t),Y j U(t)),(REI,EDR),V i ],i belongs to N i ; Initialize the adjacency matrix: M adj ;MLP network: MLP;Graph attention network: GAT;Initialize buffer: β;Get an initialized state; For each node, calculate the feature value, aggregate the features, calculate the attention and obtain the attention weight; Update the minimized predicted Q network: If β.size>R then, the training is terminated and the target network θ is updated every C steps. Q′ =θ Q； ; Update the new attention mechanism weight parameter a to a′ and distribute it to all nodes.

9. The device according to claim 8, characterized in that Environment definition: Each node in the region is always changing dynamically. The actual network is regarded as an undirected graph, where N = {N1, N2...N 20 } represents the number of network nodes, M={M ij } represents the link between each node, M represents the wireless communication link between network nodes i and j, and E adj = {I ij }N*N represents the adjacency matrix of nodes in the mobile ad hoc network; State definition: For node N, its state s obtains the state observation information S of the initial node i =[(X j U(t),Y j U(t)),(REI,EDR),(V i , T i )],i belongs to N i . refers to the energy information of the current node, expressed as a vector S = [S1, S2, ..., S T ], where x and y represent the location information of the node, RE i Indicates the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated node available energy, V i represents the speed of the node, T i Indicates the running time of the node; Action definition: Action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1,a2,...,a M ], where a M represents the selection of the Mth weight value, where M represents the size of the channel set; Reward definition: The reward r depends on the state of the selected node. If the reward is larger, the anti-interference ability of the selected path starting from the i node is the greatest.

10. The device according to claim 8, characterized in that Also includes: Send the node initialization information to the GAT-DQN network model and update the node initial state; Determine whether to send a data packet; If the data packet is sent normally, the routing table and neighbor table are updated, otherwise, the data packet is not sent; After updating the routing table and adjacency table, update the adjacency matrix to obtain the initial weight value of the node state; Send TC message and calculate routing table; Perform DQN deep reinforcement learning training on the current attention mechanism weight parameters, finally select the optimal attention mechanism weight parameters, update the node status, and determine whether to end; If the optimal attention mechanism weight parameter is selected, the process ends.

Images