Intranet terminal type identification method based on machine learning and port scanning

Abstract

The invention discloses an intranet terminal type identification method based on machine learning and port scanning, and the method comprises the steps: S1, collecting static features: adding collection equipment manufacturer OUI information except an ip address and an mac address; after the mac address is acquired from the network access layer equipment, equipment manufacturer information is acquired by querying an IEEE official OUI database or a locally maintained OUI mapping table; and S2, service port and response message acquisition optimization: a multi-protocol concurrent detection mechanism is adopted, and detection of protocols such as HTTP, HTTPS, SSH, FTP and the like is simultaneously initiated for common service ports. According to the method, terminal feature data are acquired through network scanning detection, the model is trained in combination with a machine learning algorithm, the network access terminal type is automatically identified, and compared with a traditional mode, the network management efficiency and the data accuracy are remarkably improved, the access strategy is accurately set, and the network security protection capability is effectively enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of network security management, and in particular to an intranet terminal type identification method based on machine learning and port scanning. Background Art

[0002] For large enterprises and institutions, there are hundreds of thousands of internal network nodes and terminals. In terms of network management and terminal security management, we need to identify the device type of each terminal, verify its authenticity and legitimacy, and set different network access policies (ACLs) according to different device types to restrict illegal access, thereby achieving the purpose of strengthening network security management.

[0003] However, using traditional Excel or static management applications to record connected terminal records requires manual on-site identification. This management model becomes unsustainable when the number of devices reaches the thousands or tens of thousands. Furthermore, the quality of the management records is poor, making it impossible to truly achieve network security management objectives. To address this, we designed an intranet terminal type identification method based on machine learning and port scanning to verify the authenticity and compliance of connected terminals. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes an intranet terminal type identification method based on machine learning and port scanning.

[0005] The method for identifying intranet terminal types based on machine learning and port scanning proposed in the present invention includes the following steps: S1: Static feature collection: In addition to IP addresses and MAC addresses, the device manufacturer's OUI information is also collected. After obtaining the MAC address from the network access layer device, the device manufacturer information is obtained by querying the IEEE official OUI database or the locally maintained OUI mapping table; S2: Service port and response message collection optimization: A multi-protocol concurrent detection mechanism is used to simultaneously initiate HTTP, HTTPS, SSH, FTP, and other protocol detections for common service ports. To avoid network congestion, a detection concurrency control is set to dynamically adjust the number of concurrent detections based on network bandwidth and device load. In response message processing, in addition to recording the MD5 value and message length, key field information in the message is extracted; S3: Behavioral feature collection expansion: Introducing traffic feature analysis, using NetFlow or sFlow technology to collect terminal network traffic data. Analyze traffic direction, traffic volume, traffic rate, packet size distribution and other characteristics; S4: Data cleaning and preprocessing: Deeply clean the original ledger data and use data mining algorithms to detect and correct incorrect annotations; S5: Data enhancement: For terminal types with a small number in the training data, data synthesis technology is used to enhance them; S6: Data Partitioning: Divide the processed training dataset T into a training set, a validation set, and a test set. The split is usually in a ratio of 7:1:2. The training set is used for model training, the validation set is used to adjust model hyperparameters and prevent overfitting during training, and the test set is used to evaluate the generalization ability and performance of the final model. S7: Feature engineering optimization: Arrange all possible eigenvalues ​​into a one-dimensional vector. If a terminal contains a certain eigenvalue, the corresponding vector element position is set to 1, otherwise it is 0; S8: For each type of terminal, it will be converted into a sparse matrix SR; S9: Repeat S8 to obtain the feature vectors of all terminal types; S10: Optimize the recognition accuracy, calculate the Euclidean distance between each row vector and the feature vector in the sparse matrix SR, expressed as d, ; S11: Verification of terminals with marked device types: For terminals that have been connected to the network and have marked device types, by calculating the Euclidean distance between them and the feature vector v, if the distance is greater than the threshold th, the type labeling of the device is likely to be incorrect. At this time, a verification process should be initiated to confirm its correctness and compliance.

[0006] Preferably, the manufacturer in S1 can be found to be Cisco through the mac address "00-1B-21". At the same time, the hardware serial number of the device is collected. Some network devices can be obtained through the sysObjectID node of the SNMP protocol or a specific private MIB, further enriching the static identification information of the terminal.

[0007] Preferably, in S2, for the HTTP response message, the Server field and the Content-Type field are extracted; for the SSH response message, the protocol version number is extracted. These information are used as additional feature values ​​for subsequent processing.

[0008] Preferably, surveillance cameras in S3 typically have a continuous outflow of small data packets, which are used for video streaming; office computers, on the other hand, experience large inbound bursts when downloading files. Furthermore, the interaction timing between the terminal and a specific server is recorded. For example, some industrial control devices exchange data with the control server at a fixed period, and this timing information is also included in the behavioral characteristics.

[0009] Preferably, S4 uses cluster analysis to group terminals with similar features into a single category. If most terminals within a category are labeled as the same type, but individual terminals are labeled differently, these abnormal labels are manually reviewed. For missing data, a method based on feature similarity is used to fill in missing data. The feature similarity between the missing terminal and other labeled terminals is calculated. The terminals with the highest similarity are selected as candidates, and the types of these terminals are then filled in using a combination of business logic and expert experience.

[0010] Preferably, the training data is organized by combining the collected data with the original ledger data. The original ledger data contains MAC addresses, IP addresses, and type fields, but the type field is sometimes missing and some are incorrectly labeled. This error may be due to labeling errors or illegal device impersonation. Using the MAC address as the key value, data with unlabeled types is removed to form the training dataset T.

[0011] Preferably, if more than 90% of the device type labels in S10 are correct, the distance threshold th is calculated, and the characteristic vector v and the distance threshold th of the terminal can be automatically identified.

[0012] Preferably, the results also include the determination of terminals that are newly connected to the network or have unlabeled device types: for terminals that are newly connected to the network or have already connected to the network and have unlabeled device types, the trained model is used to provide determination results, which can assist network administrators in making judgments, or directly perform automatic type labeling.

[0013] Beneficial effects of the preparation of the present invention: 1. Improved Management Efficiency: Automatically collects and analyzes terminal characteristics, eliminating the need for extensive on-site manual identification. This allows for rapid processing of large numbers of connected terminals, significantly improving network management efficiency. Leveraging technologies such as multi-protocol concurrent detection and traffic characteristics analysis, the collection and processing of massive amounts of terminal data can be completed in a short period of time, achieving efficiency improvements of dozens of times or even higher compared to traditional manual methods.

[0014] 2. Improved Data Accuracy: Data processing and analysis using machine learning algorithms, combined with data cleaning and feature selection, effectively identifies and corrects labeling errors, improving the accuracy of terminal type identification data and providing reliable data support for network security management. The application of data enhancement technology and model fusion strategies significantly improves the model's accuracy in identifying various types of terminals, especially rare types, and reduces misjudgments caused by data bias.

[0015] 3. Enhanced network security: After accurately identifying the terminal type, precise network access policies (ACLs) can be set based on the type to restrict unauthorized access and effectively prevent unauthorized devices from masquerading as network users. This strengthens network security and ensures a secure and stable network environment. A dynamic policy adjustment mechanism responds to changes in terminal type in real time, promptly blocking potential security threats and reducing the risk of network attacks.

[0016] 4. Reduced Operation and Maintenance Costs: Automated terminal type identification and management processes reduce manual operation and maintenance workload, lowering the company's human resources investment in network management. Accurate identification results also facilitate more rational network resource planning, avoiding resource waste caused by misconfiguration or over-configuration, further reducing operation and maintenance costs. DETAILED DESCRIPTION

[0017] The present invention will be further explained below with reference to specific embodiments.

[0018] This embodiment proposes a method for identifying intranet terminal types based on machine learning and port scanning, which includes the following steps: A. Network detection and data collection S1: Static feature collection: In addition to the IP address and MAC address, the device manufacturer's OUI (Organizationally Unique Identifier) ​​information is also collected. After obtaining the MAC address from the network access layer device, the device manufacturer information is obtained by querying the IEEE official OUI database or the locally maintained OUI mapping table. S2: Service port and response message collection optimization: A multi-protocol concurrent detection mechanism is used to simultaneously initiate HTTP, HTTPS, SSH, FTP, and other protocol detections for common service ports (such as 80, 443, 22, 21, etc.). To avoid network congestion, a detection concurrency control is set to dynamically adjust the number of concurrent detections based on network bandwidth and device load. In response message processing, in addition to recording the MD5 value and message length, key field information in the message is extracted; S3: Behavioral feature collection expansion: Introducing traffic feature analysis, using NetFlow or sFlow technology to collect terminal network traffic data. Analyze traffic direction (inbound and outbound), traffic volume, traffic rate, packet size distribution, and other characteristics; Through network detection and data collection, it can automatically collect and analyze terminal characteristics, without the need for a large number of manual on-site identification, and can quickly handle large-scale network terminals, greatly improving the efficiency of network management. With the help of multi-protocol concurrent detection, traffic feature analysis and other technologies, it can complete the collection and processing of massive terminal data in a short time, which is dozens of times more efficient than traditional manual methods. B. Organization of training data S4: Data cleaning and preprocessing: Deeply clean the original ledger data and use data mining algorithms to detect and correct incorrect annotations; S5: Data enhancement: For terminal types with a small number in the training data, data synthesis technology is used to enhance them; S6: Data Partitioning: Divide the processed training dataset T into a training set, a validation set, and a test set. The split is usually in a ratio of 7:1:2. The training set is used for model training, the validation set is used to adjust model hyperparameters and prevent overfitting during training, and the test set is used to evaluate the generalization ability and performance of the final model. The organization of training data can improve data accuracy. By processing and analyzing data through machine learning algorithms, combined with data cleaning and feature selection, it can effectively identify and correct labeling errors, improve the accuracy of terminal type identification data, and provide reliable data support for network security management. The application of data enhancement technology and model fusion strategy has significantly improved the model's identification accuracy for various types of terminals, especially rare types of terminals, and reduced misjudgments caused by data bias. C. Training model; S7: Feature engineering optimization: Arrange all possible eigenvalues ​​into a one-dimensional vector. If a terminal (mac) contains a certain eigenvalue, the corresponding vector element position is set to 1, otherwise it is 0; S8: For each type of terminal, it will be converted into a sparse matrix SR; S9: Repeat step 2 to obtain the feature vectors of all terminal types; S10: Optimize the recognition accuracy, calculate the Euclidean distance (EuclideanDistance) between each row vector and the feature vector in the sparse matrix SR, expressed as d, ; Enhance network security. After accurately identifying the terminal type, you can set precise network access policies (ACLs) based on different types to restrict illegal access and effectively prevent illegal devices from disguising themselves in the network, thereby strengthening network security protection capabilities and ensuring the security and stability of the network environment. The dynamic policy adjustment mechanism can respond to changes in terminal types in real time, promptly block potential security threats, and reduce the risk of network attacks. D. Utilization of results S11: Verification of terminals with marked device types: For terminals that have been connected to the network and have marked device types, by calculating the Euclidean distance between them and the feature vector v, if the distance is greater than the threshold th, the type labeling of the device is likely to be wrong. At this time, a verification process should be initiated to confirm its correctness and compliance, reducing operation and maintenance costs: Automated terminal type identification and management processes reduce the workload of manual operation and maintenance, and reduce the human cost investment of enterprises in network management. At the same time, accurate identification results help to plan network resources more reasonably, avoid resource waste caused by incorrect configuration or over-configuration, and further save operation and maintenance costs. In this example, the manufacturer in S1 is Cisco, as can be found by searching the MAC address "00-1B-21." The hardware serial number of the device is also collected. For some network devices, this can be obtained through the sysObjectID node of the SNMP (Simple Network Management Protocol) or a specific private MIB (Management Information Base), further enriching the static identification information of the terminal.

[0019] In this embodiment, S2 extracts the Server field (such as "Apache / 2.4.52") and the Content-Type field (such as "text / html; charset=UTF-8") of the HTTP response message; and extracts the protocol version number (such as "SSH-2.0-OpenSSH_8.2p1") of the SSH response message. This information is used as additional feature values ​​for subsequent processing.

[0020] In this example, surveillance cameras in S3 typically have a continuous outflow of small data packets, used for video streaming. Meanwhile, office computers experience large inbound bursts when downloading files. Furthermore, the interaction timing between terminals and specific servers is recorded. For example, some industrial control devices exchange data with control servers at a fixed period. This timing information is also included in the behavioral characteristics.

[0021] In this embodiment, S4 uses cluster analysis to group terminals with similar characteristics into a single category. If most terminals within a category are labeled with the same type, but individual terminals are labeled differently, these abnormal labels are manually reviewed. For missing data, a method based on feature similarity is used to fill in missing data. The feature similarity between the missing terminal and other labeled terminals is calculated. The terminals with the highest similarity are selected as candidate types, and the type filling is performed using a combination of business logic and expert experience.

[0022] More specifically, for example, in the organization of training data, the original ledger data is as follows (the address is fictitious): Mac Address IP address type 0c4e.5431.xxxx aa.bb.cc.59 Call the police 5803.fb43.xxxx aa.bb.cc.56 UPS networking a414.37a3.xxxx aa.bb.cc.42 988b.0ad3.xxxx aa.bb.cc.41xzacl Monitoring host 00e0.4cea.xxxx aa.bb.cc.40 IP Intercom 0026.c50a.xxxx aa.bb.cc.38 Xunmei Intelligent Monitoring The type field in this ledger is somewhat missing, and a small number of types are incorrect. This error may be due to labeling errors or intentional disguise of a certain type of device to achieve illegal network access. The collected data above is combined with the ledger data, and the unlabeled data is removed. The MAC address is used as the key value to form the training data set T.

[0023] T={ "Monitoring host":{ "mac1":[port80,port443,190.160.0.1_80,...], "mac2":[port80,port443,port8420,190.160.0.1_80,...], ...}, "IP Intercom":{ "mac3":[port80,port8080,190.160.0.280,...], "mac3":[port80,http1024,190.160.0.280,...], ...}, ... } In this embodiment, all possible eigenvalues ​​are arranged into a one-dimensional vector. If a terminal (mac) contains a certain eigenvalue, the corresponding vector element position is set to 1, otherwise it is 0. The above training data set T is then transformed into: T={ "Monitoring host":{ "mac1":[0,0,0,1,0,0,0,0,1,...], "mac4":[0,0,0,1,1,0,0,0,1,...], }, "IP Intercom":{ "mac3":[0,1,0,0,0,1,0,0,0,1,...], "mac4":[0,1,0,0,0,1,0,0,1,1,...], ...}, ... } For each type of terminal, it will be converted into a sparse matrix SR, such as: SR_MONITOR_HOST = [ 0,0,0,1,0,0,0,0,1,... 0,0,0,1,1,0,0,0,1,... ... ] Calculate the probability p of each feature appearing in each column to form the feature vector v=[p0, p1, p2, p3, p4,...] of this type of terminal, such as: V_monitoring_host=[0,0,0,0.8,0.7,0,0,0,0.9,...] Repeat SR to obtain the feature vectors of all terminal types; Calculate the Euclidean distance between each row vector and the eigenvector in the sparse matrix SR, expressed as d, ,If more than 90% of the device type labels are correct, the distance threshold th can be calculated, and the terminal’s feature vector v and distance threshold th can be ,automatically identified.

[0024] In this embodiment, the results also include the determination of terminals that are newly connected to the network or have not been labeled with device types: for terminals that are newly connected to the network or have been connected to the network and have not been labeled with device types, the trained model is used to provide determination results, which can assist network administrators in making judgments, or directly perform automatic type labeling.

[0025] This method has the function of automatically collecting and analyzing terminal features, does not require a large amount of manual on-site identification, and can quickly process large-scale terminals entering the network, greatly improving the efficiency of network management. With the help of multi-protocol concurrent detection, traffic feature analysis and other technologies, the collection and processing of massive terminal data can be completed in a short period of time, which is dozens of times more efficient than traditional manual methods, and improves data accuracy: through machine learning algorithms to process and analyze data, combined with data cleaning, feature selection and other means, it can effectively identify and correct labeling errors, improve the accuracy of terminal type identification data, and provide reliable data support for network security management. The application of data enhancement technology and model fusion strategy has significantly improved the model's identification accuracy for various types of terminals, especially rare types of terminals, and reduced misjudgments due to data bias.

[0026] Enhanced network security: After accurately identifying terminal types, precise network access policies (ACLs) can be set based on the type to restrict unauthorized access and effectively prevent unauthorized devices from masquerading as network users. This strengthens network security and ensures a secure and stable network environment. A dynamic policy adjustment mechanism responds to changes in terminal types in real time, promptly blocking potential security threats and reducing the risk of network attacks. Reduced O&M costs: Automated terminal type identification and management processes reduce manual O&M workload and lower the company's human resources investment in network management. Furthermore, accurate identification results facilitate more rational network resource planning, avoiding resource waste caused by misconfiguration or over-configuration, further reducing O&M costs.

[0027] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An intranet terminal type identification method based on machine learning and port scanning, characterized in that: The following steps are involved: S1: Static feature collection: In addition to IP addresses and MAC addresses, the device manufacturer's OUI information is also collected. After obtaining the MAC address from the network access layer device, the device manufacturer information is obtained by querying the IEEE official OUI database or the locally maintained OUI mapping table. S2: Service port and response message collection optimization: A multi-protocol concurrent detection mechanism is used to simultaneously initiate HTTP, HTTPS, SSH, and FTP protocol detection for common service ports. To avoid network congestion, a detection concurrency control is set to dynamically adjust the number of concurrent detections based on network bandwidth and device load. In response message processing, in addition to recording the MD5 value and message length, key field information in the message is extracted. S3: Behavioral feature collection expansion: Introducing traffic feature analysis, using NetFlow or sFlow technology to collect terminal network traffic data and analyze the characteristics of traffic direction, traffic volume, traffic rate, and packet size distribution; S4: Data cleaning and preprocessing: Deeply clean the original ledger data and use data mining algorithms to detect and correct incorrect annotations; S5: Data enhancement: For terminal types with a small number in the training data, data synthesis technology is used to enhance them; S6: Data partitioning: Divide the processed training dataset T into a training set, a validation set, and a test set. The training set is used for model training, the validation set is used to adjust model hyperparameters and prevent overfitting during training, and the test set is used to evaluate the generalization ability and performance of the final model. S7: Feature engineering optimization: Arrange all eigenvalues ​​into a one-dimensional vector. If a terminal contains a certain eigenvalue, the corresponding vector element position is set to 1, otherwise it is 0; S8: For each type of terminal, it will be converted into a sparse matrix SR; S9: Repeat S8 to obtain the feature vectors of all terminal types; S10: Calculate the Euclidean distance between each row vector and the eigenvector in the sparse matrix SR, expressed as d, ; S11: For terminals that have been connected to the network and have been labeled with device types, by calculating the Euclidean distance between them and the feature vector v, if the distance is greater than the threshold th, the type labeling of the device is likely to be incorrect. At this time, a verification process should be initiated to confirm its correctness and compliance.

2. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: S1 also collects the hardware serial number of the device. Some network devices obtain it through the sysObjectID node of the SNMP protocol or a specific private MIB, further enriching the static identification information of the terminal.

3. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: In S2, for HTTP response messages, the Server field and Content-Type field are extracted, and for SSH response messages, the protocol version is extracted. This information is used as additional feature values ​​for subsequent processing.

4. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: Surveillance camera devices in S3 usually have a continuous outflow of small data packets for video streaming, while office computers will have large inbound traffic bursts when downloading files. At the same time, the interaction timing between the terminal and the specific server is recorded.

5. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: S4 uses cluster analysis to group terminals with similar characteristics into one category. If most terminals in a category are labeled as the same type, but some terminals are labeled differently, these abnormal labels will be manually reviewed.

6. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: In the organization of training data, the collected data is combined with the original ledger data. The original ledger data contains mac address, IP address and type fields, but there are cases where the type field is missing and some are incorrectly labeled. The data without labeled types are eliminated to form the training data set T.

7. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: More than 90% of the device type labels in S10 are correct, so the distance threshold th is calculated, and the terminal's feature vector v and distance threshold th can be automatically identified.

8. The intranet terminal type identification method based on machine learning and port scanning according to claim 1 is characterized in that: The results also include the determination of terminals that are newly connected to the network or have unlabeled device types. For terminals that are newly connected to the network or have already been connected to the network but have not been labeled with device types, the trained model is used to provide determination results to assist network administrators in making judgments, or to directly perform automatic type labeling.

Images