A network device identification method and system based on reinforcement learning

Abstract

The application relates to a network equipment identification method and system based on reinforcement learning, and the method steps are as follows: S1, collecting network behavior data of all equipment at key network nodes, cleaning and structuring the network behavior data to form a database; S2, performing feature processing on the data in the database to generate a primary machine learning model; S3, performing multiple learning training on the primary machine learning model and the data in the database to form a final machine learning model; and S4, identifying network behavior data of network equipment to be identified through the final machine learning model. The application discloses a system, computer equipment and computer readable storage medium for realizing network equipment identification by adopting the method. The application identifies equipment through multiple feature indexes of equipment network behavior, the formed machine learning model can continuously learn and feedback to optimize the machine learning model, the identification effect is improved, and the equipment in a management network can be effectively distinguished.

Description

Technical Field

[0001] This invention relates to the field of communication network technology, and in particular to a network device identification method and system based on reinforcement learning. Background Technology

[0002] Currently, network device identification primarily relies on three methods: IMEI (International Mobile Equipment Identity) identification, MAC (Media Access Control) address identification, and UA (User Agent) identification. These methods primarily use single pieces of information such as open ports and MAC addresses for device identification, which lacks versatility. While machine learning has been developed for device classification, it still largely depends on traffic volume as a single metric. This limited number of metrics leads to poor identification results, and the system cannot relearn from the generated model. For example, patent CN109450733 uses traffic volume for identification. This single-metric approach is ineffective and cannot meet the demands of complex problems and constantly changing network environments. It often results in identification errors or failures to identify devices. Furthermore, the inability to relearn from the generated model each time further impacts the identification accuracy. Summary of the Invention

[0003] To address the shortcomings of existing methods, this invention provides a network device identification method and system based on reinforcement learning.

[0004] The technical solution adopted by this invention to solve its technical problem is: a network device identification method based on reinforcement learning, the steps of which are as follows: S1. Collect network behavior data of multiple network behavior characteristics of all devices at key network nodes, clean and structure the data to form a database. The network behavior data includes traffic data, device connection relationship data, device open port data, and data returned from accessing open ports. The collection of the network behavior data includes the following steps: S1a, uses the ARP and / or TCP protocols to detect devices present in the network; S1b uses a Python script to access the open ports of each device in the detected devices to obtain data. S2, the data in the database is characterized to generate a primary machine learning model. The characteristic processing is based on a hierarchical spatial clustering table semantic summarization algorithm or a concept-layered semantic graph summarization algorithm. The concept-layered semantic graph summarization algorithm first layers the attribute values ​​of each node in the network into concepts, and then processes them using the K-SNAP method. The attribute value concept layering is based on the hierarchical K-Summary algorithm. The K-SNAP method includes a hierarchical sequence function and a hierarchical unified evaluation function. The hierarchical sequence function first uses Beta as the first sequence objective function according to NodeDiff, and then uses Delta as the second sequence objective function according to EdgeDiff. The unified evaluation function sorts all group node pairs according to NodeDiff and EdgeDiff respectively, and each group node pair obtains two indices. The group node pair with the smallest sum of indices is selected as the final choice. S3 utilizes the primary machine learning model and data from the database to perform multiple interactive learning and training processes to form the final machine learning model. During the training process, the database generates reward / penalty information based on the recognition results each time and feeds it back to the primary machine learning model for further learning and training. After the final machine learning model is formed, there is also a process of evaluating it. S4 uses the final machine learning model to identify the network behavior data of the network device to be identified.

[0005] Preferably, the semantic graph summarization algorithm based on concept hierarchies first hierarchically stratifies the attribute values ​​of each node in the network, and then processes them using the K-SNAP method. The attribute value concept hierarchies are based on the hierarchical K-Summary algorithm. The K-SNAP method includes a hierarchical sequence function and a hierarchical unified evaluation function. The hierarchical sequence function first uses Beta as the first sequence objective function according to NodeDiff, and then uses Delta as the second sequence objective function according to EdgeDiff. The unified evaluation function sorts all pairs of nodes according to NodeDiff and EdgeDiff respectively, and each pair of nodes obtains two indices. The pair of nodes with the smallest sum of indices is selected as the final choice.

[0006] A system for network device identification based on the reinforcement learning method described above includes a data module and a learning module. The data module is used to collect network behavior data of multiple network behavior features of all devices at key network nodes and clean and structure the data to form a database. The learning module is used to perform feature processing on the data in the database to generate a primary machine learning model, and to perform multiple interactive learning training using the primary machine learning model and the data in the database to form a final machine learning model. The system also includes an evaluation module for evaluating the final machine learning model.

[0007] A computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the device identification method as described in any of the preceding claims.

[0008] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the device identification method as described in any of the preceding claims.

[0009] The beneficial effects of this invention are as follows: This invention collects network behavior data of multiple network behaviors of network devices, and uses multiple feature indicators of the network behavior of the devices to form a machine learning model to identify the devices. At the same time, the formed machine learning model can continuously learn and receive feedback to optimize the machine learning model and improve the recognition effect. It can effectively distinguish and manage devices in the network, and compared with the existing machine learning models that need to relearn each time, it improves efficiency and saves costs. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the process of an embodiment of the present invention; Figure 2 This is a schematic diagram of the learning process according to an embodiment of the present invention; Detailed Implementation To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments, providing a clear and complete description. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention. Furthermore, directional terms mentioned in the present invention, such as "up," "down," "front," "back," "left," "right," "inner," and "outer," are merely for reference to the directions shown in the accompanying drawings. The use of directional terms is for better and clearer explanation and understanding of the present invention, and is not intended to indicate or imply any necessary orientation of the present invention; therefore, it should not be construed as a limitation of the present invention.

[0011] Examples of embodiments of the present invention Figures 1 to 2 As shown, a network device identification method based on reinforcement learning includes the following steps: S1 involves collecting network behavior data from all devices at key network nodes, cleaning and structuring the data to form a database. A key network node is a special node that can significantly influence the network's structure and function. Key network nodes can be identified using existing methods, such as KPP-POS and KPP-NEG. KPP-NEG divides the network into connected components or makes the path length between nodes so large that it's equivalent to disconnection, implicitly representing graph fragmentation. This can be confirmed by measuring network fragmentation, the number of disconnected nodes, and the total distance between nodes after removing key nodes. KPP-POS identifies nodes that can be reached by direct links or short paths, maximizing the number of remaining nodes. This implicitly represents inter-group cohesion, which can be confirmed by calculating the cohesion between the key point set and the rest of the network, or by setting an objective function. For setting an objective function, the objective function is first defined, then the optimal individual is found using the objective function, followed by finding the next optimal individual with the least redundancy, or by using tabu search, simulated annealing, or genetic algorithms for search confirmation. Network behavior characteristics refer to the traffic flowing through various devices, the connectivity between devices, and the open ports of various devices. These characteristics generate corresponding network behavior data, which includes traffic data, device connectivity data, device open port data, and data returned from accessing open ports (including HTML, images, etc.). The collection of this network behavior data can be achieved through the following steps: S1a, uses the ARP and / or TCP protocols to detect devices present in the network; S1b uses a Python script to access the open ports of each device in the detected devices to obtain data. After collecting network data from devices, this data is initially disorganized. Therefore, it is cleaned and then structured to form a database. Different network behavior characteristics correspond to different databases, such as traffic databases and connection relationship databases. The corresponding network device identification system is the data module, which collects network behavior data from all devices at key network nodes, identifies multiple network behavior characteristics, and then cleans and structures this data to form a database.

[0012] S2 involves feature-processing the data in the database to generate a basic machine learning model. Feature-processing reduces the data size while preserving as many original features as possible. This feature-processing can be done using either a hierarchical spatial clustering-based table semantic summarization algorithm or a concept-layered semantic graph summarization algorithm. The hierarchical spatial clustering-based table semantic summarization algorithm is a data feature-processing algorithm for relational data. The semantic summary table retains the rich features of the original table and uses the concept layering of attribute values ​​to generalize the original data table into tuples. It uses a small number of "abstract" tuples to represent a large number of "detailed" tuples, thus reducing the size of the data table. The attribute value concept layering describes a tree-like hierarchical structure of classification relationships within the attribute value domain, reflecting the ontological information of the attributes; for example, the algorithm ResAdjust(T,K). Input: T, set is the encoding set of the original data; K, int compression ratio Output: Rset, the encoding set of the set subspace; Begin 1. Rest := Initial(T); 2. Tmpset := Null; 3.While(|Rest|>|T| / K) 4. Tmpset := Null; 5. foreach (rs in Rset) 6.Trs=Resolution(rs,g,1); 7. Insert(tmpSet, trs); 8. end foreah 9. Rset = tmpSet; 10. end while 11.while(|Rest|!>|T| / K) 12. Set1 = Null, Set2 = Null; 13.Tmpset = Select(Rset); 14.Delete(Rset,Tmpset); 15.Split(Tmpset,&Set1,&Set2); 16.Insert(Rset,Set1,Set2) 17. end while 18. Slim (Rset); 19. Return Rset; End The semantic graph summarization algorithm based on concept hierarchies first hierarchically stratifies the attribute values ​​of each node in the network, and then processes them using the K-SNAP method. The attribute value concept hierarchies are based on the hierarchical K-Summary algorithm. The K-SNAP method includes a hierarchical sequence function and a hierarchical unified evaluation function. The hierarchical sequence function first uses Beta as the first sequence objective function according to NodeDiff, and then uses Delta as the second sequence objective function according to EdgeDiff. The unified evaluation function sorts all pairs of nodes according to NodeDiff and EdgeDiff respectively, and each pair of nodes obtains two indices. The pair of nodes with the smallest sum of indices is selected as the final choice.

[0013] S3 utilizes a primary machine learning model and data from a database for multiple training iterations to form the final machine learning model. The primary model and its learning model continuously interact with the database data, receiving rewards / penalties to adjust their strategies. This addresses the issue of lag in each learning iteration, maximizing cumulative rewards and avoiding getting trapped in local optima. Since each recognition result is not all the data, each learning iteration only returns a portion of the data. After multiple batch learning iterations, the final machine learning model will obtain the optimal solution. This also solves the problem of existing machine learning models requiring relearning each time, improving efficiency and saving costs. S4 uses the final machine learning model to identify the network behavior data of the network device to be identified.

[0014] Further improvements include an evaluation process after the final machine learning model is formed. Evaluation can effectively verify whether the final machine learning model can solve the problem and ensure its practicality. Any existing evaluation method can be used for evaluation.

[0015] A computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the device identification method as described in any of the preceding claims. The computer device includes a central processing unit (CPU), a system memory comprising random access memory (RAM) and system memory (SRAM), and a system bus connecting the system memory and the CPU. It also includes a basic input / output system for facilitating information transfer between various devices within the computer and a mass storage device for storing the operating system, application programs, and other program modules.

[0016] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the device identification method as described in any of the preceding claims, the storage medium including RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technologies, CD-ROM, DVD or other optical storage, magnetic tape, disk storage, or other magnetic storage devices.

[0017] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A network device identification method based on reinforcement learning, characterized in that: The steps are as follows: S1. Collect network behavior data of multiple network behavior characteristics of all devices at key network nodes, clean and structure the data to form a database. The network behavior data includes traffic data, device connection relationship data, device open port data, and data returned from accessing open ports. The collection of the network behavior data includes the following steps: S1a, uses the ARP and / or TCP protocols to detect devices present in the network; S1b uses a Python script to access the open ports of each device in the detected devices to obtain data. S2, the data in the database is characterized to generate a primary machine learning model. The characteristic processing is based on a hierarchical spatial clustering table semantic summarization algorithm or a concept-layered semantic graph summarization algorithm. The concept-layered semantic graph summarization algorithm first layers the attribute values ​​of each node in the network into concepts, and then processes them using the K-SNAP method. The attribute value concept layering is based on the hierarchical K-Summary algorithm. The K-SNAP method includes a hierarchical sequence function and a hierarchical unified evaluation function. The hierarchical sequence function first uses Beta as the first sequence objective function according to NodeDiff, and then uses Delta as the second sequence objective function according to EdgeDiff. The unified evaluation function sorts all group node pairs according to NodeDiff and EdgeDiff respectively, and each group node pair obtains two indices. The group node pair with the smallest sum of indices is selected as the final choice. S3 utilizes the primary machine learning model and data from the database to perform multiple interactive learning and training processes to form the final machine learning model. During the training process, the database generates reward / penalty information based on the recognition results and feeds it back to the primary machine learning model for further learning and training. After the final machine learning model is formed, there is also a process of evaluating it. S4 uses the final machine learning model to identify the network behavior data of the network device to be identified.

2. A system based on the reinforcement learning-based network device identification method as described in claim 1, characterized in that... The system includes a data module and a learning module. The data module is used to collect network behavior data of multiple network behavior features of all devices at key network nodes, and then clean and structure the data to form a database. The learning module is used to perform feature processing on the data in the database to generate a primary machine learning model, and then use the primary machine learning model and the data in the database to perform multiple interactive learning and training to form a final machine learning model. The system also includes an evaluation module, which is used to evaluate the final machine learning model.

3. A computer device, characterized in that... The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the device identification method as described in claim 1.

4. A computer-readable storage medium, characterized in that... The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the device identification method as described in claim 1.

Images