A graph attention network reinforcement learning anti-interference routing optimization method and device

Abstract

The application discloses a kind of graph attention network reinforcement learning anti-interference routing optimization method and device, the method of the present application includes that network node is according to the initialization weight data of collected GAT network model is updated corresponding DQN model, the optimal routing path is selected using updated DQN model, and the latest attention mechanism weight parameter is sent to GAT network model;Node feature is handled to generate updated node feature set, and updated attention mechanism weight parameter is obtained according to updated node feature set and other initial state observation information to construct GAT network model;GAT-DQN network model training is carried out based on environment definition, state definition, action definition and reward definition to obtain trained GAT-DQN network model;Trained GAT-DQN network model is used for mobile ad hoc network node.The present application aims to enhance the anti-interference capability of mobile self-organizing network (MANET).

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a graph attention network reinforcement learning anti-interference routing optimization method and device. BACKGROUND

[0002] In recent years, the application of mobile ad hoc network (MANET) is becoming more and more widespread, with the in-depth research of wireless communication module and the smaller and smaller size of sensor, mobile nodes can be used as a single execution task node, or multiple nodes interact together to perform more complex tasks, while avoiding potential and life-threatening dangers, not only in civilian scenarios, but also in military scenarios, such as performing border patrol, disaster relief and as a relay network in complex and dangerous environments.

[0003] Mobile ad hoc network (MANET) is based on these devices to build a decentralized and highly mobile network, which is particularly suitable for communication in complex and dangerous environments. However, due to the changing topology of the mobile network of nodes, it is difficult to maintain a stable and reliable communication network. Especially in military applications, the limited battery capacity, processing power and storage capacity of mobile devices will affect the transmission performance and efficiency of the network. In order to ensure the anti-interference ability of the mobile ad hoc network and ensure the information transmission of each node in military applications and improve the communication quality, it is therefore essential to develop an efficient anti-interference mobile ad hoc network routing optimization technology.

[0004] Mobile ad hoc network can be quickly established without pre-existing network infrastructure, and communication between nodes does not depend on fixed routers or base stations, and is suitable for node mobility, automatically reconfiguring routing to maintain connectivity, and in cases where direct communication is not possible, nodes can act as relays to forward messages to other nodes, achieving multi-hop communication. Moreover, mobile ad hoc networks do not rely on central control, and each node can participate in network maintenance and routing decision-making. This network has attracted the attention of many experts in various fields due to its flexibility and ability to operate in complex environments. However, the freedom of node movement also brings a significant weakness, making it difficult to maintain routing tables and ensure correct data forwarding.

[0005] Generally, the routing protocol of self-organizing network needs to consider network capacity, node characteristics and business requirements, etc., on the basis of realizing the basic function of routing, to improve network performance as much as possible and realize efficient and fast data transmission. SUMMARY

[0006] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0007] To this end, the present application proposes a graph attention network reinforcement learning anti-interference routing optimization method, which combines the deep Q network (DQN) algorithm and the principle of graph attention network, aiming to enhance the anti-interference ability of mobile ad hoc network (MANET).

[0008] Another object of the present application is to propose a graph attention network reinforcement learning anti-interference routing optimization device.

[0009] To achieve the above object, the present application proposes a graph attention network reinforcement learning anti-interference routing optimization method, comprising:

[0010] The network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, selects the optimal routing path using the updated DQN model, and sends the latest attention mechanism weight parameter to the GAT network model;

[0011] The node features are processed to generate an updated node feature set, and the updated attention mechanism weight parameter is obtained based on the updated node feature set and other initial state observation information to construct the GAT network model;

[0012] The GAT-DQN network model is trained based on the environment definition, state definition, action definition and reward definition to obtain the trained GAT-DQN network model;

[0013] The trained GAT-DQN network model is used for mobile ad hoc network nodes, and the DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience through the neural network to output the updated attention mechanism weight parameter, calculates the Q value sequence according to the current state of the channel, selects the action corresponding to the maximum Q value for communication, and finally outputs the selected action and the updated attention mechanism weight parameter for global optimization and synchronization.

[0014] The graph attention network reinforcement learning anti-interference routing optimization method of the present application embodiment can also have the following additional technical features:

[0015] In an embodiment of the present application, the node features are processed to generate an updated node feature set, comprising:

[0016] The node feature set h = {h1, h2, …, h N}; wherein N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h´;

[0017] A weight-shared parameter matrix W ∈ R is applied to each nodeF′×F , for each node perform attention mechanism a: R F′ x R F = R calculate the corresponding attention coefficient, e ij represents the importance of node j to node i e ij = a ( Whi , Whj ).

[0018] In an embodiment of the application, the GAT network model is executed for each training cycle, including:

[0019] For each node N, obtain the initial node state observation information Si=[(X j U(t), Y j U(t)), (RE,EDR),V i ], i belongs to N i ;

[0020] Initialize the adjacency matrix: M adj ; MLP network: MLP; graph attention network: GAT; initialize the buffer: beta; get an initialized state;

[0021] Calculate the feature value for each node, aggregate the features, calculate the attention and obtain the attention weight;

[0022] Update the global model using the aggregated parameters;

[0023] Update the minimum prediction Q network: +∑ i (y i -Q(s i ,a i | Q )) 2 ;

[0024] if beta.size>R then, terminate the training, update the target network every C steps Q′ = Q ;

[0025] Update the new attention mechanism weight parameter a to a´, and distribute it to all nodes.

[0026] In an embodiment of the application, the environment is defined: each node in the region is always dynamically changing, and the actual network is regarded as an undirected graph, where N={N1, N2……N 20} represents the number of network nodes, M= { M ij} represents the link between each node, M represents the wireless communication link between network nodes i and j, adopts E adj ={I ij}N*N represents the adjacency matrix of the nodes in the mobile ad hoc network;

[0027] State definition: for node N, its state s obtains the state observation information S i =[(X j U(t), Y j U(t)), (RE,EDR),(V i ,T i )],i belongs to N i , refers to the position energy information of the current node, represented as a vector S = [S1, S2,...,S T ], wherein x, y represent the position information of the node, RE i represents the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated available energy of the node, V i represents the speed of the node, T i represents the running time of the node;

[0028] Action definition: the action a is the weight value W of the current link, the attention coefficient e ij is calculated, and the action set is [a1, a2,..., a M ], wherein a M represents selecting the Mth weight value, and M represents the size of the channel set;

[0029] Reward definition: the reward r depends on the state of the selected node, and if the reward is greater, the anti-interference ability of the selected path from the i node is greater.

[0030] In one embodiment of the present application, the method further comprises:

[0031] Sending the initialization information of the node to the GAT-DQN network model to update the initial state of the node;

[0032] Determining whether to send a data packet; if the data packet is normally sent, updating the routing table and the neighbor table, otherwise, not sending the data packet;

[0033] Updating the adjacency matrix after updating the routing table and the neighbor table to obtain the initial weight value of the node state;

[0034] Sending a TC message to calculate the routing table;

[0035] The current attention mechanism weight parameter is subjected to DQN deep reinforcement learning training, and finally the optimal attention mechanism weight parameter is selected, the node state is updated, and it is judged whether to end or not.

[0036] If the optimal attention mechanism weight parameter is selected, the process is ended.

[0037] To achieve the above purpose, another aspect of the present application provides a graph attention network reinforcement learning anti-interference routing optimization device, comprising:

[0038] A DQN network model construction module is used for updating the corresponding DQN model according to the collected initialization weight data of the GAT network model by the network node, selecting the optimal routing path by using the updated DQN model, and sending the latest attention mechanism weight parameter to the GAT network model;

[0039] A GAT network model construction module is used for processing the node features to generate an updated node feature set, and obtaining the updated attention mechanism weight parameter based on the updated node feature set and other initial state observation information to construct the GAT network model;

[0040] A GAT-DQN network model training module is used for GAT-DQN network model training based on environment definition, state definition, action definition and reward definition to obtain the trained GAT-DQN network model;

[0041] A model application output module is used for applying the trained GAT-DQN network model to the mobile ad hoc network node, receiving the data four-tuple in the experience pool by the DQN model in the mobile ad hoc network node, learning the historical experience by the neural network to output the updated attention mechanism weight parameter, calculating the Q value sequence according to the current state of the channel and selecting the action corresponding to the maximum Q value for communication, and finally outputting the selected action and the updated attention mechanism weight parameter for global optimization and synchronization.

[0042] The graph attention network reinforcement learning anti-interference routing optimization method and device of the embodiment of the present application can improve the anti-interference capability in the network by combining each node in the network topology, assigning different weights according to the attention mechanism, highlighting the features of each node, and generating the optimal node routing optimization strategy by using DQN. The strategy can more efficiently perform data transmission by assigning different node weights, and improve the anti-interference performance of the mobile ad hoc network.

[0043] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein:

[0045] Figure 1 is a flow chart of the graph attention network reinforcement learning anti-interference routing optimization method according to an embodiment of the present application;

[0046] Figure 2 is a communication network framework diagram of the whole according to an embodiment of the present application;

[0047] Figure 3 is a GAT network model training schematic diagram according to an embodiment of the present application;

[0048] Figure 4 is a DQN network model training schematic diagram according to an embodiment of the present application;

[0049] Figure 5 is a GAT network model schematic diagram according to an embodiment of the present application;

[0050] Figure 6 is a first result schematic diagram derived for use in nodes according to an embodiment of the present application;

[0051] Figure 7 is a second result schematic diagram derived for use in nodes according to an embodiment of the present application;

[0052] Figure 8 is a structural diagram of the graph attention network reinforcement learning anti-interference routing optimization device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0054] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0055] The graph attention network reinforcement learning anti-interference routing optimization method and device according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0056] Figure 1is a flow chart of a graph attention network reinforcement learning anti-interference routing optimization method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0057] S1, the network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, selects the optimal routing path by using the updated DQN model, and sends the latest attention mechanism weight parameter to the GAT network model;

[0058] S2, the node features are processed to generate an updated node feature set, and the updated attention mechanism weight parameter is obtained according to the updated node feature set and other initial state observation information to construct the GAT network model;

[0059] S3, the GAT-DQN network model is trained based on the environment definition, state definition, action definition and reward definition to obtain the trained GAT-DQN network model;

[0060] S4, the trained GAT-DQN network model is used for mobile ad hoc network nodes, and the DQN model in the mobile ad hoc network node receives the data four-tuple in the experience pool, and learns the historical experience by the neural network to output the updated attention mechanism weight parameter, calculates the Q value sequence according to the current state of the channel, and selects the action corresponding to the maximum Q value for communication, and finally outputs the selected action and the updated attention mechanism weight parameter for global optimization and synchronization.

[0061] Figure 2 is the overall communication network framework of the present application. The communication framework is built by OPNET software, and the routing layer is optimized. The optimization of the routing layer is completed by the interaction of OPNET software and python. The same TXT path is completed. The GAT-DQN model is built in PYHTON, and the optimal attention weight parameter is trained. Then, the routing layer in the OPNET software is interacted to complete the anti-interference routing optimization.

[0062] In an embodiment of the present application, the DQN network model is constructed, comprising:

[0063] Initialization: the network node accepts the initialization weight data from the GAT, and uses the initial data for training cycle.

[0064] Training period: enter a cycle, and execute the following steps for each training period n=1, 2,...

[0065] a) model update: after each training period, the node updates its DQN model according to the collected weight data of the GAT and experience.

[0066] b) Routing path selection: using the updated local model, the communication network selects appropriate nodes for transmission during the transmission process to optimize its communication efficiency and improve anti-interference ability.

[0067] c) Parameter sending: at a fixed time interval, for example, after XXT communication time frames, the node sends its latest current latest attention mechanism weight parameter to the GAT network model, and then to the routing process of the communication framework in the OPNET software for further processing.

[0068] In an embodiment of the present application, the GAT network model is constructed, including:

[0069] Input of the initialization module: a set of node features h={h1, h2,..., hn}, where N is the number of nodes, F is the feature dimension of each node, and the output of the GAT network model is the updated node feature set h´. N In order to obtain the updated feature matrix, first apply a weight-shared parameter matrix W∈R F′×F to each node, then the present application performs an attention mechanism a: R F′ ×R F =R on each node to calculate the corresponding attention coefficient e ij , which represents the importance of node j to node i: e ij = a ( Whi , Whj )。

[0070] Aggregation period: the GAT network model performs the following steps for each training period m=1,2,3,...

[0071] a) For each node N, obtain the initial node state observation information Si=[(X j U(t), Y j U(t)), (RE, EDR), V i ], i belongs to N i .

[0072] b) Initialize adjacency matrix: M adj ; MLP network: MLP; graph attention network: GAT; initialize buffer: β. Get an initialized state.

[0073] c) For each node, calculate the feature value, aggregate the features, calculate the attention, and obtain the attention weight.

[0074] d) Global model update: update the global model using the aggregated parameters.

[0075] e) Performance evaluation: update the minimum prediction Q network: +∑ i (y i -Q(s i ,a i | Q )) 2 , which can involve evaluating the model's performance on a specific task or under certain conditions.

[0076] f) Condition check: ifβ.size>R then, terminate training, update target network every C steps Q′ = Q 。

[0077] g) Parameter distribution: update the new attention mechanism weight parameters a to a ´ and distribute them to all nodes so that they can update their own models.

[0078] In one embodiment of the invention, local model training includes:

[0079] Environment definition: The set environment is a mobile ad hoc network area, that is, a dynamic network topology scenario. Each node in this area is always changing, and the actual network is regarded as an undirected graph, where N = {N1, N2,..., N 20} represents the number of network nodes, M = {M ij} represents the links between nodes, M represents the wireless communication link between network nodes i and j, and E adj = {I ij}N*N represents the adjacency matrix of nodes in a mobile ad hoc network. That is, when I ij = 1, it means that the nodes are adjacent to each other, and there is a communication link between the two nodes that can communicate, otherwise they are not adjacent.

[0080] State definition: For node N, its state s obtains the initial node state observation information S i = [(X j U(t), Y j U(t)), (RE, EDR), (V i , T i )], i belongs to N i , which refers to the current node's position energy information, which can be represented as a vector S = [S1, S2,..., S T ], where x and y represent the node's position information, RE i represents the node's current available energy, EDR i new represents the updated energy consumption rate, REi new V represents the updated node available energy i T represents the speed of the node i T represents the running time of the node

[0081] Action definition: action a is the weight value W of the current link, and the attention coefficient e is calculated ij , the action set is [a1, a2,..., a M ], where a M represents selecting the Mth weight value, and M represents the size of the channel set.

[0082] Reward definition: the reward r depends on the state of the selected node, and the greater the reward, the greater the anti-interference ability of the selected path from the i node.

[0083] Start training: in the transmission process of the mobile ad hoc network communication network, first, each node is initialized, each node interacts with each other, each node can simultaneously select an action, and obtains a reward in the next stage. This involves implementing action a, and the current node obtains the link weight according to the node information and the neighbor node information. Each node is affected by the weight, and the complete path of other nodes is obtained by adjusting the link cost in the network.

[0084] In an embodiment of the present application, the trained GAT-DQN training model is used for mobile ad hoc network nodes, comprising:

[0085] Experience pool input: the DQN model in the mobile ad hoc network node receives the data quadruple [s, a, r, s'] in the experience pool as input. Wherein, s represents the current channel state value; a represents the current link weight, that is, the action taken at the current communication; r represents the reward value mentioned above, that is, the feedback obtained after taking the action; s' represents the new state after the action, that is, the updated aggregated feature state of the selected node.

[0086] Model learning and updating: the DQN model of the mobile ad hoc network node learns the historical experience through the neural network, thereby updating the model parameters and optimizing the node weight. The DQN algorithm randomly selects samples from the experience pool to eliminate the correlation between samples and accelerate the convergence of the model. In addition, the algorithm uses a target Q network to reduce the loss function and promote the stable convergence of the model. Every N training periods, the parameters of the action Q network are updated to the target Q network, and the loss function is calculated.

[0087] Attention mechanism strategy execution: according to the current state, the DQN model outputs a channel corresponding Q(s, a; ) value sequence. The attention mechanism strategy selects the corresponding channel for communication according to the maximum Q value in the sequence. At the same time, the mobile ad hoc network node transmits its local model parameters to the GAT network model, so as to update and synchronize the attention mechanism model.

[0088] The core of the deep Q network (DQN) algorithm includes four main parts: the estimation Q network, the target Q network, the experience replay mechanism and the loss function. In this algorithm framework, the estimation Q network is responsible for estimating the Q value.

[0089] In this process, the mobile ad hoc network node performs an action a, that is, determines the state s based on the current node's position state information. This state is then updated to a new state s' according to the transition function, and the corresponding reward value r is obtained from the environment. The estimation Q network, as a system constructed by a neural network, takes the current state s as input and calculates the action Q value under this condition. In Q(s, a; ), the weight value corresponding to the maximum Q value is regarded as the channel with the lowest interference probability.

[0090] The DQN neural network structure adopted by the application includes an input layer, a hidden layer and an output layer. The input layer receives the defined state value, and the neural network generates the Q value of each channel through a fully connected hidden layer in the output layer. Selecting the channel with the maximum Q value means selecting the channel with the lowest interference probability. Through multiple training and learning processes, the parameters of the neural network are optimized, effectively fitting the channel state and Q value, and then determining the best channel selection.

[0091] In the DQN algorithm, the application of experience replay plays a crucial role, which not only significantly reduces the correlation between data samples, but also helps to ensure that the learning model can effectively converge. The experience replay mechanism achieves this goal by storing data of each state transition [s, a, r, s´]. When the amount of stored data exceeds the capacity limit of the experience pool, it automatically discards the earliest data record. Appropriately increasing the size of the experience pool helps to reduce the risk of overfitting. During the training process of the model, the experience replay mechanism randomly extracts a certain amount of historical data from the pool for training the estimation Q network.

[0092] The estimation Q network and the target Q network are consistent in structure, but they differ in function. The output value of the target Q network is Q(s´, a´; ), which retains the previous parameters of , which is used to calculate the current estimation Q network The loss function of the parameters is very crucial. In every N training cycle interval, the parameters of the estimated Q network are copied to the target Q network. In this way, the DQN algorithm utilizes the target Q network and the estimated Q network to minimize the loss function and promote the model to reach a convergent state. The loss function adopts: +∑ i (y i -Q(s i ,a i | Q )) 2 .

[0093] It can be understood that the GAT network model of the present application is as shown in Figure 5 , and the training process is as shown in Figure 3 , and the specific steps are as follows:

[0094] 1. The initialization information of the node is transmitted to the GAT-DQN model, and the initial state of the node is updated.

[0095] 2. At this time, it is judged whether to send a hello data packet. If the data packet is normally sent, the routing table and the neighbor table are updated, otherwise, it is not sent.

[0096] 3. After updating the routing table and the neighbor table, the adjacency matrix is updated, and the initial weight value of the node state is obtained at this time.

[0097] 4. Send the TC message. Calculate the routing table.

[0098] 5. DQN deep reinforcement learning is performed on the attention weight at this time. The optimal node attention weight is finally selected, the node state is updated, and it is judged whether to end.

[0099] 6. If the optimal weight is selected, it is ended, otherwise, step 5 is continued.

[0100] It can be understood that Figure 4 is a DQN network model training schematic diagram according to an embodiment of the present application, as shown in Figure 4 .

[0101] 1) First, initialize two deep Q networks, one is an online network (DQN), and the other is a target network. Input the initial state s and action a into the online Q network, and input the state s into the target Q network. At the same time, create an experience replay buffer to store the experience (state, action, reward, new state) of the agent.

[0102] 2) At every time step N, the agent selects an action a according to the current state s and DQN. This process can use the ε-greedy strategy, that is, select a random action with a certain probability ε to explore, and select the current optimal action with a probability of 1-ε to utilize.

[0103] 3) After performing action a, get new state s' and reward r after each action, then store experience (s, a, s', r) into experience replay buffer.

[0104] 4) Sample a batch of experiences from experience replay buffer, calculate target Q value of each experience, calculate loss function L(θ) according to target Q value and output of DQN, update DQN parameters θ using backpropagation.

[0105] 5) Copy online Q network parameters of DQN to target network every fixed C steps to maintain stability of target Q value.

[0106] 6) Repeat steps 2) - 5) until optimal weights are obtained.

[0107] Further, Figure 6 And Figure 7 is a result diagram obtained by applying the method to nodes according to an embodiment of the application.

[0108] The beneficial effects of the application are:

[0109] (1) Enhanced communication efficiency and stability: By assigning weight coefficients to the nodes of the mobile ad hoc network with attention mechanism, the characteristics of each node are highlighted according to different weight coefficients, which significantly improves the communication efficiency and stability of the mobile ad hoc network. Using machine learning model, mobile ad hoc network can more efficiently adapt to complex environmental conditions, efficiently select available nodes in high dynamic conditions, and maintain efficient communication connection.

[0110] (2) Self-adaptive and continuously optimized network performance: GAT network model and DQN continuously interact to update parameters, enabling the entire mobile ad hoc network system to continuously adjust and optimize itself. The network topology of the mobile ad hoc network will change constantly due to the movement of nodes, and the GAT network model can quickly adapt to such dynamic changes and adjust the weight values of the nodes according to real-time conditions to better respond to the transmission of overall network information, thereby improving the performance and reliability of the entire system.

[0111] (3) Effective resource management and reduced interference: This method effectively reduces link overhead, improves the available energy of each node, reduces the total energy consumption of the entire network, and effectively manages network resources. In a highly dynamic network environment, not only can it efficiently select appropriate nodes for fast and good information transmission, but also effectively deal with potential security risks such as intentional interference and attacks, thereby protecting the security and reliability of communication.

[0112] The method for anti-interference routing optimization of the graph attention network reinforcement learning according to the embodiment of the application tests the communication efficiency and stability of the mobile ad hoc network in the same network topology environment by using different methods. In the same scenario, the application can more efficiently adapt to the dynamic topology of the network, perform stable information transmission, and enhance the anti-interference capability of the mobile ad hoc network, which shows that the method has high adaptability and efficiency.

[0113] In order to realize the above-mentioned embodiments, as shown in the embodiment, the anti-interference routing optimization device 10 of the graph attention network reinforcement learning is further provided, which comprises: Figure 8

[0114] The DQN network model construction module 100 is configured to update the corresponding DQN model according to the collected initialization weight data of the GAT network model, select the optimal routing path by using the updated DQN model, and send the latest attention mechanism weight parameter to the GAT network model.

[0115] The GAT network model construction module 200 is configured to process the node features to generate an updated node feature set, and obtain the updated attention mechanism weight parameter based on the updated node feature set and other initial state observation information to construct the GAT network model.

[0116] The GAT-DQN network model training module 300 is configured to train the GAT-DQN network model based on the environment definition, the state definition, the action definition and the reward definition to obtain the trained GAT-DQN network model.

[0117] The model application output module 400 is configured to apply the trained GAT-DQN network model to the mobile ad hoc network node, receive the data quadruple in the experience pool by the DQN model in the mobile ad hoc network node, learn the historical experience by the neural network to output the updated attention mechanism weight parameter, calculate the Q value sequence according to the current state of the channel and select the action corresponding to the maximum Q value for communication, and finally output the selected action and the updated attention mechanism weight parameter for global optimization and synchronization.

[0118] Further, the node features are processed to generate an updated node feature set, which comprises:

[0119] The node feature set h = {h1, h2, …, hN} is obtained, where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h´. N

[0120] A weight-shared parameter matrix W ∈ R F′×F is applied to each node, and the attention mechanism a: R F′ × R​​F = R compute the corresponding attention coefficient, e ij represents the importance of node j to node i e ij = a Whi , Whj

[0121] Further, for each training cycle GAT network model is executed, including:

[0122] For each node N, get the initial node state observation information Si=[(X j U(t), Y j U(t)), (RE,EDR),V i ], i belongs to N i ;

[0123] Initialize adjacency matrix: M adj ; MLP network: MLP; graph attention network: GAT; initialize buffer: β; get an initialized state;

[0124] For each node, calculate the eigenvalue, aggregate features, calculate attention and get attention weight;

[0125] Update the minimum prediction Q network: +∑ i (y i -Q(s i ,a i | Q )) 2 ;

[0126] ifβ.size>R then, terminate training, update target network every C steps Q′ = Q ;

[0127] Update the new attention mechanism weight parameter a to a´, and distribute it to all nodes.

[0128] Further, the environment is defined: each node in the region is always dynamically changing, and the actual network is regarded as an undirected graph, where N={N1, N2……N 20} represents the number of network nodes, M= { M ij} represents the link between each node, M represents the wireless communication link between network nodes i and j, which adopts E adj ={I ij}N*N represents the adjacency matrix of the nodes in the mobile ad hoc network; ​​​​

[0129] State definition: for node N, its state s obtains the initial node state observation information S i = [(X j U(t), Y j U(t)), (REI,EDR),(V i , T i )], i belongs to N i , which refers to the position energy information of the current node, represented as a vector S = [S1, S2,..., S T ], wherein x and y represent the position information of the node, RE i represents the current available energy of the node, EDR i new represents the updated energy consumption rate, RE i new represents the updated available energy of the node, V i represents the speed of the node, and T i represents the running time of the node.

[0130] Action definition: the action a is the weight value W of the current link, and the attention coefficient e ij is calculated, and the action set is [a1, a2,..., a M ], wherein a M represents selecting the Mth weight value, and M represents the size of the channel set.

[0131] Reward definition: the reward r depends on the state of the selected node, and the greater the reward, the greater the anti-interference ability of the selected path from the i node.

[0132] Further, it further comprises:

[0133] The initial information of the node is sent to the GAT-DQN network model, and the initial state of the node is updated.

[0134] It is judged whether to send a data packet; if the data packet is normally sent, the routing table and the neighbor table are updated, otherwise the data packet is not sent.

[0135] After updating the routing table and the neighbor table, the adjacency matrix is updated to obtain the initial weight value of the node state.

[0136] The TC message is sent, and the routing table is calculated.

[0137] The current attention mechanism weight parameter is subjected to DQN deep reinforcement learning training, the optimal attention mechanism weight parameter is finally selected, the node state is updated, and it is judged whether to end.

[0138] If the optimal attention mechanism weight parameter is selected, the process ends.

[0139] The anti-interference routing optimization device of the graph attention network reinforcement learning according to the embodiment of the application tests the communication efficiency and stability of the mobile ad hoc network in the same network topology environment by different methods. In the same scenario, the application can more efficiently adapt to the dynamic topology of the network, perform stable information transmission, enhance the anti-interference capability of the mobile ad hoc network, and show the high adaptability and efficiency of the method of the application.

[0140] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0141] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A method for anti-interference routing optimization based on graph attention network reinforcement learning, characterized in that, The network node updates the corresponding DQN model according to the collected initialization weight data of the GAT network model, selects the optimal routing path by using the updated DQN model, and sends the latest attention mechanism weight parameter to the GAT network model; The node features are processed to generate an updated node feature set, and the updated attention mechanism weight parameter is obtained according to the updated node feature set and other initial state observation information to construct the GAT network model; The GAT-DQN network model is trained based on the environment definition, state definition, action definition and reward definition to obtain the trained GAT-DQN network model; The trained GAT-DQN network model is used for a mobile ad hoc network node, the DQN model in the mobile ad hoc network node receives the data quadruple in the experience pool, and learns the historical experience by using a neural network to output the updated attention mechanism weight parameter, calculates the Q value sequence according to the current state of the channel, selects the action corresponding to the maximum Q value for communication, and finally outputs the selected action and the updated attention mechanism weight parameter for global optimization and synchronization. The node features are processed to generate an updated node feature set, including:

2. The method of claim 1, wherein, Whi node feature set h = {h1, h2,..., h N N}; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h´; apply a weight shared parameter matrix W e R to each node F′×F , perform an attention mechanism a: R on each node F′ F = R compute the corresponding attention coefficients, e ij denotes the importance of node j for node i e ij a Whj The GAT network model is executed for each training period, including: ​​​​​ 3. The method of claim 2, wherein, Calculate the feature value and aggregated feature for each node, and obtain the attention weight by using attention; For each node N, obtain the initial node state observation information Si=[(X j U(t), Y j U(t)), (REI,EDR),V i ],i belongs to N i ; Initialize the adjacency matrix: M adj MLP network: MLP; graph attention network: GAT; initialize the buffer: β; get an initialized state; Update the global model by using the aggregated parameters; Update the new attention mechanism weight parameter a to a', and distribute it to all nodes. Update the minimization prediction Q-network: +∑ i (y i -Q(s i ,a i | Q )) 2 ; if β.size > R then, terminate training, every C steps, update target network Q′ = Q ; 4. The method of claim 3, wherein, The reward definition is that the reward r depends on the state of the selected node, and the greater the reward, the greater the anti-interference ability of the selected path from the i node. Environment definition: each node in the region is always dynamically changing, taking the actual network as an undirected graph, wherein N = {N1, N2, …, N 20} represents the number of network nodes, M = {M ij} represents the links between the nodes, M represents the wireless communication link between network nodes i and j, and E adj = {I ij}N*N represents the adjacency matrix of the nodes in the mobile ad hoc network; State definition: For node N, its state s is obtained from the initial node's state observation information S. i =[(X j U(t), Y j U(t)),(RE,EDR),(V i T i )], i belongs to N i This refers to the energy information at the current node's position, represented as a vector S = [S1, S2,...,S...]. T ], where x and y represent the location information of the node, RE i EDR represents the current available energy of a node. i new Represents the updated energy consumption rate, RE i new V represents the updated available energy of the node. i T represents the velocity of the node. i Indicates the node's runtime; Action definition: Action a is the weight value W of the current link, calculate the attention coefficient e ij , the action set is [a1, a2,...,a M ] , where a M represents selecting the Mth weight value, M represents the size of the channel set; The method further includes:

5. The method of claim 3, wherein, Send the initialization information of the node to the GAT-DQN network model to update the initial state of the node; Determine whether to send a data packet; if the data packet is normally sent, update the routing table and the neighbor table, otherwise, do not send the data packet; Update the adjacency matrix after updating the routing table and the neighbor table to obtain the initial weight value of the node state; Send the TC message and calculate the routing table; Perform DQN deep reinforcement learning training on the current attention mechanism weight parameter, finally select the optimal attention mechanism weight parameter, update the node state, and determine whether to end; If the optimal attention mechanism weight parameter is selected, the process ends. The method includes:

6. A graph attention network reinforcement learning anti-jamming routing optimization apparatus, characterized in that, The DQN network model construction module is configured to update the corresponding DQN model according to the collected initialization weight data of the GAT network model, select the optimal routing path by using the updated DQN model, and send the latest attention mechanism weight parameter to the GAT network model; The GAT network model construction module is configured to process the node features to generate an updated node feature set, and obtain the updated attention mechanism weight parameter according to the updated node feature set and other initial state observation information to construct the GAT network model; ​ The GAT-DQN network model training module is configured to train the GAT-DQN network model based on the environment definition, the state definition, the action definition, and the reward definition to obtain a trained GAT-DQN network model. The model application output module is configured to apply the trained GAT-DQN network model to a mobile ad hoc network node, receive data quadruples in an experience pool by a DQN model in the mobile ad hoc network node, learn and output updated attention mechanism weight parameters by a neural network based on historical experience, calculate a Q value sequence based on a current state of a channel and select an action corresponding to a maximum Q value for communication, and finally output the selected action and the updated attention mechanism weight parameters for global optimization and synchronization.

7. The apparatus of claim 6, wherein, The node features are processed to generate an updated node feature set, including: node feature set h = {h1, h2,..., h N N}; where N is the number of nodes, F is the feature dimension of each node, and the output is the updated node feature set h´; apply a weight shared parameter matrix W e R F′×F , perform an attention mechanism a: R F′ x R F = R compute the corresponding attention coefficients, e ij denotes the importance of node j for node i e ij = a ( Whi , Whj ).

8. The apparatus of claim 7, wherein, The GAT network model is executed for each training period, including: For each node N, obtain the initial node state observation information Si=[(X j U(t), Y j U(t)), (RE,EDR),V i ],i belongs to N i ; Initialize the adjacency matrix: M adj MLP network: MLP; graph attention network: GAT; initialize the buffer: β; get an initialized state; For each node, a characteristic value and aggregated features are calculated, an attention is acquired, and an attention weight is calculated. Update the minimization prediction Q-network: +∑ i (y i -Q(s i ,a i | Q )) 2 ; if β.size > R then, terminate training, every C steps, update target network Q′ = Q ; The new attention mechanism weight parameters a are updated as a' and distributed to all nodes.

9. The apparatus of claim 8, wherein, Environment definition: each node in the region is always dynamically changing, taking the actual network as an undirected graph, wherein N = {N1, N2, …, N 20} represents the number of network nodes, M = {M ij} represents the links between the nodes, M represents the wireless communication link between network nodes i and j, and E adj ={I ij}N*N represents the adjacency matrix of the nodes in the mobile ad hoc network; State definition: For node N, its state s is obtained from the initial node's state observation information S. i =[(X j U(t), Y j U(t)),(RE,EDR),(V i T i )], i belongs to N i This refers to the energy information at the current node's position, represented as a vector S = [S1, S2,...,S...]. T ], where x and y represent the location information of the node, RE i EDR represents the current available energy of a node. i new Represents the updated energy consumption rate, RE i new V represents the updated available energy of the node. i T represents the velocity of the node. i Indicates the node's runtime; Action definition: Action a is the weight value W of the current link, calculate the attention coefficient e ij , the action set is [a1, a2,...,a M ] , where a M represents selecting the Mth weight value, M represents the size of the channel set; The reward definition is that a reward r depends on a state of a selected node, and the greater the reward, the greater the anti-interference capability of a selected path from the i-th node.

10. The apparatus of claim 8, wherein, The apparatus further includes: The initialization information of the node is sent to the GAT-DQN network model to update an initial state of the node. It is determined whether to send a data packet. If the data packet is sent normally, a routing table and a neighbor table are updated, otherwise, the data packet is not sent. After the routing table and the neighbor table are updated, an adjacency matrix is updated to obtain an initial weight value of the node state. A TC message is sent to calculate the routing table. The current attention mechanism weight parameters are trained by DQN deep reinforcement learning, and finally the optimal attention mechanism weight parameters are selected to update the node state and determine whether to end. If the optimal attention mechanism weight parameters are selected, the process ends.

Images