Reinforcement learning-based command and control network structure and key node prediction method

Abstract

The invention relates to a command and control network structure and key node prediction method based on reinforcement learning, and belongs to the field of communication reconnaissance and artificial intelligence. According to the method, a network structure serial number and a key node serial number sequence are obtained through observation, a reinforcement learning algorithm is used for training, the dynamic change rule of a network structure and key nodes is recognized, and the network state at the future moment is predicted. The method represents a state and an action as a character string and stores the character string in a q table. In the training process, the duration of the accumulation time is gradually increased, the reconnaissance state information is combined, and accumulated reward values under different accumulation durations are obtained. And performing multi-step prediction on the reconnaissance data by using the accumulation time with the maximum accumulated reward value and the q table, so that the network structure serial number and the key node serial number at the next moment can be accurately predicted, and the prediction capability of command and control network behaviors is effectively improved.

Description

Technical Field

[0001] The invention discloses a command and control network structure and a key node prediction method based on reinforcement learning, and belongs to the field of communication reconnaissance and artificial intelligence. Background Art

[0002] The command and control network is a vital component in the field of modern information technology. The prediction of its structure and key nodes is of great significance to the stability, reliability and anti-interference ability of the network. However, due to the complex structure of the command and control network and its dynamic changes over time, how to effectively predict the network structure and key nodes based on communication reconnaissance data has become an urgent problem to be solved. In the patent "A method for identifying key nodes in command and control networks based on deep learning" applied by the 54th Institute of China Electronics Technology Group Corporation, with the authorization announcement number CN11478474B, key nodes are intelligently identified based on the link relationship between network nodes. However, this method does not analyze the law of changes in network structure and key nodes. In the article "Research on the Importance of Dynamic Network Nodes Based on Spatiotemporal Graph Convolution" published by Zhang Kai et al. in "College Mathematics", the focus is on the analysis of the importance of extracting network nodes in the process of dynamic changes. In the article "Reliability Analysis of Command and Control Network System Based on Generalized Continuous-Time Bayesian Network" published by Li Yifan et al. in "System Engineering and Electronic Technology", the key nodes of the command and control network can be extracted using the Bayesian network, but the research goal is to analyze from the perspective of reliability, not to analyze the dynamic changes of the network. Current research methods mainly focus on network topology analysis and data-driven key node identification. However, these methods often lack sufficient predictive capabilities when facing dynamically changing command and control networks. With the development of reinforcement learning technology, it is possible to use its adaptive and learning capabilities to solve the problem of predicting command and control networks. Summary of the invention

[0003] The purpose of the present invention is to provide a method for predicting command and control network structure and key nodes based on reinforcement learning. By utilizing the reinforcement learning algorithm, the accumulation time of communication reconnaissance data is gradually increased, the change rules in the command and control network structure and key node sequence obtained by communication reconnaissance are mined, and while identifying the optimal accumulation time, the network structure sequence number and key node sequence number at the next time point are predicted, thereby achieving accurate and stable prediction of network structure and key nodes.

[0004] The technical solution adopted by the present invention is:

[0005] A command and control network structure and key node prediction method based on reinforcement learning, comprising the following steps:

[0006] Step 1: Obtain the number of structural types and key nodes of the command and control network, and number them. The structural type numbers and key node numbers obtained within a set time period are combined into a reconnaissance data set;

[0007] Step 2: Traverse all structure type numbers and key node numbers to build state space and action space;

[0008] Step 3: Set the accumulation time p, and initialize the q table with states as rows and actions as columns; associate the state data of the accumulation time to get the associated state at the current moment, and use the associated state of the next accumulation time as the real data, then select the action corresponding to the associated state at the current moment from the action space. If the real data is the same as the selected action, the reward value is set to 0, otherwise the reward value is set to -1, and the reward value is accumulated; finally, update the q table according to the associated state and the selected action, change the accumulation time, and repeat this step for a set number of times;

[0009] Step 4: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value, and obtain the corresponding q-table. Then associate the state data of the accumulation time to obtain the associated state. In the corresponding q-table, select the action element corresponding to the maximum value in the row where the associated state is located as the predicted value.

[0010] Furthermore, the specific process of step 1 is:

[0011] The number of structure types N and the number of key nodes M of the command and control network are obtained through communication reconnaissance equipment, the structure types and key nodes of the command and control network are numbered in the order of reconnaissance, and T continuous time reconnaissance data sets are obtained in the format of [structure type serial number, key node serial number]; where T is the set value.

[0012] Furthermore, the specific process of step 2 is:

[0013] The formats of state and action are [structure type serial number, key node serial number]; traverse all structure type serial numbers and key node serial numbers, and convert them into string form to construct state space and action space, the dimensions of which are .

[0014] Furthermore, the specific process of step 3 is as follows:

[0015] Step 3-1: Initialization space;

[0016] Step 3-2: Initialize the accumulation time p=1;

[0017] Step 3-3: Initialize the q table, with states as rows, initialized to all elements in the state space, and actions as columns, initialized to all elements in the action space, with all values ​​in the table being 0; and initialize the cumulative reward value is 0;

[0018] Step 3-4: Initialize the screen variable e=0;

[0019] Step 3-5: Initialize the time variable t=0;

[0020] Step 3-6: According to the accumulation time p, associate the state data from the current time t to the time t+p-1, a total of p states, and the associated state at time t is , and associate the state data from time t+1 to time t+p, a total of p states, the associated state at time t+1 is ; Real data at time t is the status data at time t+p;

[0021] Step 3-7: Associate the state at time t , randomly select actions from the action space when executing for the first time ; Generate random numbers after the first time ,if , then randomly select an action from the action space ,if , then select The action element corresponding to the maximum value in the row; is the set value;

[0022] Step 3-8: If the real data With action The same, the reward value r is set to 0; if the real data With action If they are not the same, the reward value r is set to -1, and the reward value r is added to the cumulative reward value ;

[0023] Step 3-9: Update the q table. The update formula is:

[0024] q ( S t , a t ) ← q ( S t , a t ) + α [ r + γ max a q ( S t + 1 , a ) − q ( S t , a t ) ]

[0025] Will The value of is updated to q ( S t , a t ) + α [ r + γ max a q ( S t + 1 , a ) − q ( S t , a t ) ] ,in is the q table OK The value of the element corresponding to the column, For learning efficiency, is the discount factor, Indicates the corresponding Find the column with the maximum value in the row ;

[0026] Step 3-10: If t is less than Tp-1, then , and return to step 3-6; if t is equal to Tp-1, execute step 3-11;

[0027] Step 3-11: If e is less than E, then , and return to step 3-5; if e is equal to E, execute step 3-12; where E represents the maximum screen, which is the set value;

[0028] Step 3-12: Set the reward value Deposit space, and save the q table with the file name corresponding to the accumulation time p;

[0029] Step 3-13: If p is less than P, then , and return to step 3-3, otherwise end this process; where P represents the maximum value of the accumulation time, which is the set value.

[0030] Furthermore, the specific process of step 4 is as follows:

[0031] Step 4-1: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value ;

[0032] Step 4-2: According to Load the q table of the corresponding name, which is ;

[0033] Step 4-3: Based on the accumulation time , associated with the current moment To time Status data of status, The state of the moment association is ;

[0034] Step 4-4: Select The action element corresponding to the maximum value in the row is used as the predicted value.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. Strong adaptive learning ability: It can automatically learn the dynamic changes of the command and control network through reinforcement learning algorithms and can adapt to complex and changing network environments.

[0037] 2. Effective capture and prediction of changing patterns: The method of gradually increasing the state accumulation time can not only effectively identify the patterns in the network structure and key node sequence, but also accurately predict the network structure sequence number and key node sequence number at the next time point on this basis, thereby improving the accuracy of the prediction.

[0038] 3. Strong robustness: The simple and clear reward function (0 and -1) design enables the model to be significantly punished in a timely manner when prediction errors occur, preventing the model from falling into local optimality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The present invention provides a flowchart of a command and control network structure and a key node prediction method based on reinforcement learning according to an embodiment of the present invention.

[0040] Figure 2 This is a diagram showing the relationship between the accumulated time and the accumulated reward value according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0042] A command and control network structure and key node prediction method based on reinforcement learning. The overall process is as follows Figure 1 As shown, the specific steps include:

[0043] Step 1: Construct the structure type and key node data set of the command and control network;

[0044] The number of types N and number of nodes M of the command and control network structure are obtained through communication reconnaissance equipment. Both values ​​are 10. The structure types and nodes of the command and control network are numbered in the order of reconnaissance. The reconnaissance data sets of T consecutive times are obtained in the format of [structure type serial number, key node serial number]. The value of T is 30. The data set is:

[0045] [ (1, 6), (2, 7), (1, 6), (2, 7), (3, 4), (1, 6), (1, 6), (2, 7), (1,6), (2, 7), (3, 4), (1, 6),(1, 6), (2, 7), (1, 6), (2, 7), (3, 4), (1, 6),(1, 6), (2, 7), (1, 6), (2, 7), (3, 4), (1, 6), (1, 6), (2, 7), (1, 6), (2, 7), (3, 4), (1, 6)]

[0046] In order to verify the effectiveness of the algorithm, the data at the next moment is saved for verification, and its value is (1, 6).

[0047] Step 2: Traverse all structure type numbers and key node numbers to build state space and action space;

[0048] The format of state and action is [structure type serial number, key node serial number]. Both state space and action space are composed of traversing all structure type serial numbers and key node serial numbers, and the dimensions are ; Convert the structure type serial number and key node serial number into string form to construct the state space and action space.

[0049] Step 3: Set the accumulation time p, and initialize the q table with states as rows and actions as columns; associate the state data of the accumulation time to get the associated state at the current moment, and use the associated state of the next accumulation time as the real data, and then select the action corresponding to the associated state at the current moment from the action space. If the real data is the same as the selected action, the reward value is set to 0, otherwise the reward value is set to -1, and the reward value is accumulated; finally, update the q table according to the associated state and the selected action, change the accumulation time, and repeat this step for a set number of times; the specific process is:

[0050] Step 3-1: Initialization space;

[0051] Step 3-2: Initialize the accumulation time p=1;

[0052] Step 3-3: Initialize the q table, with states as rows, initialized to all elements in the state space, and actions as columns, initialized to all elements in the action space, with all values ​​in the table being 0; and initialize the cumulative reward value is 0;

[0053] Step 3-4: Initialize the screen variable e=0;

[0054] Step 3-5: Initialize the time variable t=0;

[0055] Step 3-6: Construct the associated state: According to the accumulation time p, associate the state data from the current time t to the time t+p-1, a total of p states, and the associated state at time t is , and associate the state data from time t+1 to time t+p, a total of p states, the associated state at time t+1 is ; Real data at time t It is the status data of t+p;

[0056] Step 3-7: Associate the state at time t , randomly select actions from the action space when executing for the first time ; Generate random numbers after the first time ,if , then randomly select an action from the action space ,if , then select The action element corresponding to the maximum value in the row; is the set value;

[0057] Step 3-8: Update the cumulative reward value: If the real data With action The same, the reward value r is set to 0; if the real data With action If they are not the same, the reward value r is set to -1, and the reward value r is added to the cumulative reward value ;

[0058] Step 3-9: Update q table: Update the q table, the update formula is:

[0059] q ( S t , a t ) ← q ( S t , a t ) + α [ r + γ max a q ( S t + 1 , a ) − q ( S t , a t ) ]

[0060] Will The value of is updated to q ( S t , a t ) + α [ r + γ max a q ( S t + 1 , a ) − q ( S t , a t ) ] ,in is the q table OK The value of the element corresponding to the column, For learning efficiency, is the discount factor, Indicates the corresponding Find the column with the maximum value in the row ;

[0061] Step 3-10: If t is less than Tp-1, then , and return to step 3-6; if t is equal to Tp-1, execute step 3-11;

[0062] Step 3-11: If e is less than E, then , and return to step 3-5; if e is equal to E, execute step 3-12, where E represents the maximum screen, which is 1000 in this embodiment;

[0063] Step 3-12: Set the reward value Deposit space, and save the q table with the file name corresponding to the accumulation time p;

[0064] Step 3-13: If p is less than P, then , and return to step 3-3, otherwise end this process; where P represents the maximum value of the accumulation time, which is 10 in this embodiment.

[0065] Step 4: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value, and obtain the corresponding q table, then associate the state data of the accumulation time to obtain the associated state, and select the action element corresponding to the maximum value in the row where the associated state is located in the corresponding q table as the predicted value; the specific process is:

[0066] Step 4-1: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value , this embodiment is 3, the relationship between the accumulation time and the accumulated reward value is as follows Figure 2 As shown;

[0067] Step 4-2: According to Load the q table of the corresponding name, which is ;

[0068] Step 4-3: Based on the accumulation time , associated with the current moment To time Status data of status, The state of the moment association is , whose values ​​are [(2, 7), (3, 4), (1, 6)];

[0069] Step 4-4: Select The action element corresponding to the maximum value in the row is used as the predicted value, and we get (1, 6).

[0070] The network structure number and key node number predicted by this method at the next moment are (1, 6), which is consistent with the data collected during the communication reconnaissance process. This shows that this method can effectively extract the law of command and control network changes, thereby predicting the network structure number and key node number.

Claims

1. A command and control network structure and key node prediction method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Obtain the number of structural types and key nodes of the command and control network, and number them. The structural type numbers and key node numbers obtained within a set time period are combined into a reconnaissance data set; Step 2: Traverse all structure type numbers and key node numbers to build state space and action space; Step 3: Set the accumulation time p, and initialize the q table with states as rows and actions as columns; associate the state data of the accumulation time to get the associated state at the current moment, and use the associated state of the next accumulation time as the real data, then select the action corresponding to the associated state at the current moment from the action space. If the real data is the same as the selected action, the reward value is set to 0, otherwise the reward value is set to -1, and the reward value is accumulated; finally, update the q table according to the associated state and the selected action, change the accumulation time, and repeat this step for a set number of times; Step 4: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value, and obtain the corresponding q-table. Then associate the state data of the accumulation time to obtain the associated state. In the corresponding q-table, select the action element corresponding to the maximum value in the row where the associated state is located as the predicted value.

2. According to claim 1, a command and control network structure and key node prediction method based on reinforcement learning is characterized in that: The specific process of step 1 is as follows: The number of structure types N and the number of key nodes M of the command and control network are obtained through communication reconnaissance equipment, the structure types and key nodes of the command and control network are numbered in the order of reconnaissance, and T continuous time reconnaissance data sets are obtained in the format of [structure type serial number, key node serial number]; where T is the set value.

3. According to claim 2, a command and control network structure and key node prediction method based on reinforcement learning is characterized in that: The specific process of step 2 is: The formats of state and action are [structure type serial number, key node serial number]; traverse all structure type serial numbers and key node serial numbers, and convert them into string form to construct state space and action space, the dimensions of which are .

4. The command and control network structure and key node prediction method based on reinforcement learning according to claim 3 is characterized in that: The specific process of step 3 is as follows: Step 3-1: Initialization space; Step 3-2: Initialize the accumulation time p=1; Step 3-3: Initialize the q table, with states as rows, initialized to all elements in the state space, and actions as columns, initialized to all elements in the action space, with all values ​​in the table being 0; and initialize the cumulative reward value is 0; Step 3-4: Initialize the screen variable e=0; Step 3-5: Initialize the time variable t=0; Step 3-6: According to the accumulation time p, associate the state data from the current time t to the time t+p-1, a total of p states, and the associated state at time t is , and associate the state data from time t+1 to time t+p, a total of p states, the associated state at time t+1 is ; Real data at time t is the status data at time t+p; Step 3-7: Associate the state at time t , randomly select actions from the action space when executing for the first time ; Generate random numbers after the first time ,if , then randomly select an action from the action space ,if , then select The action element corresponding to the maximum value in the row; is the set value; Step 3-8: If the real data With action The same, the reward value r is set to 0; if the real data With action If they are not the same, the reward value r is set to -1, and the reward value r is added to the cumulative reward value ; Step 3-9: Update the q table. The update formula is: Will The value of is updated to ,in is the q table OK The value of the element corresponding to the column, For learning efficiency, is the discount factor, Indicates the corresponding Find the column with the maximum value in the row ; Step 3-10: If t is less than Tp-1, then , and return to step 3-6; if t is equal to Tp-1, execute step 3-11; Step 3-11: If e is less than E, then , and return to step 3-5; if e is equal to E, execute step 3-12; where E represents the maximum screen, which is the set value; Step 3-12: Set the reward value Deposit space, and save the q table with the file name corresponding to the accumulation time p; Step 3-13: If p is less than P, then , and return to step 3-3, otherwise end this process; where P represents the maximum value of the accumulation time, which is the set value.

5. According to claim 4, a command and control network structure and key node prediction method based on reinforcement learning is characterized in that: The specific process of step 4 is as follows: Step 4-1: Use the elbow method to obtain the accumulation time corresponding to the inflection point of the cumulative reward value ; Step 4-2: According to Load the q table of the corresponding name, which is ; Step 4-3: Based on the accumulation time , associated with the current moment To time Status data of status, The state of the moment association is ; Step 4-4: Select The action element corresponding to the maximum value in the row is used as the predicted value.

Images