A method for identifying intranet topology based on traffic after address translation

Abstract

The present invention discloses a method for identifying intranet network topology based on traffic after address conversion, the method comprising the following steps: 1) dividing network traffic data captured outside a gateway based on quintuple information of the traffic to obtain a series of network flows; 2) extracting features from data packets in each of the network flows, selecting information identifying an internal network host in a network protocol field of the data packet as a benchmark feature for clustering intranet hosts; 3) dividing the benchmark features obtained in step 2) into a plurality of subclasses according to their characterization directions, clustering the network flows respectively according to the benchmark features corresponding to each subclass to obtain a plurality of base clusters; then merging the plurality of base clusters to obtain an intranet host clustering result; 4) determining the number of network hosts according to the intranet host clustering result, and determining the network topology level according to the information in the network flow to obtain an intranet network topology.

Description

Technical Field

[0001] The invention relates to a method for identifying intranet topology based on flow information, belonging to the technical field of computer networks and data communications. Background Art

[0002] Network Address Translation (NAT) is a technology that modifies the source or destination address of network packets to a specific address during routing. Source Address Translation typically modifies the source address of packets sent by internal hosts to the router's address, resolving the issue of internal addresses being unrecognizable on the public network. When processing return packets, the destination address is modified to the internal host's address, allowing the packets to reach the correct host.

[0003] Due to limited IP addresses and security concerns, common network designs often perform source network address translation (NAT) at the gateway, allowing numerous intranet hosts to share one (or a few) public IP addresses. From the outside, all packets within this network appear to originate from the same IP address. This conserves public IP addresses while also obscuring the internal network's structure, preventing external networks from knowing the exact number of hosts within the network. NAT increases the difficulty for attackers, but also creates obstacles for network oversight.

[0004] Machine learning methods have achieved numerous research results in the field of network security, significantly broadening the scope of network security research. Clustering, a commonly used unsupervised learning method in machine learning, is widely used for group classification and detection of target users, as well as for identifying outliers and abnormal values. Clustering involves partitioning a dataset into clusters based on a specific criterion. The goal is to minimize the distance between data objects within the same cluster and maximize their similarity, while maximizing the distance between data objects in different clusters and minimizing their similarity. Clustering is a process that clusters data of the same type together as much as possible, while separating data of different types as much as possible.

[0005] The clustering process generally consists of five steps: data preparation, feature selection, feature extraction, clustering, and clustering result evaluation. The data preparation stage standardizes features or reduces their dimensionality; the feature selection stage selects the most effective features from the original features to form feature vectors; the feature extraction stage transforms the selected features to form new features; the clustering stage uses a similarity metric based on a distance function or similarity function to obtain clusters; and clustering result evaluation uses methods such as distance error sum and profile analysis to analyze the effectiveness of the clustering results. Common clustering methods include Sequential Leader Clustering (SLC), spectral clustering, ROCK clustering, METIS hypergraph clustering, and cluster fusion. Summary of the Invention

[0006] The object of the present invention is to provide a method for identifying intranet network topology based on traffic after address conversion. According to extracting a series of clues that can characterize the communication initiator from network traffic (the information carried by some fields in the traffic can be used as clues to characterize the communication initiator, and can be divided into baseline features and auxiliary candidate features according to its characterization ability to the communication initiator), then based on the quintuple information of the traffic, a machine learning method is used to distinguish the communication initiator from other traffic initiators, thereby identifying how many communication nodes there are in the intranet, and further based on the distance and application relationship exposed by the network communication, the network topology of the intranet is inferred (each time a data packet passes through a routing device, the TTL value is reduced by 1, and the level of the host topology in the network is determined by combining the change of the TTL value in the IP protocol). Since the information carried by these clues in the traffic is very weak and the characterization ability difference is large, the present invention makes full use of the advantages of machine learning, first combs out the communication node information carried in the network traffic, organizes it into feature vectors, and then utilizes machine learning algorithm to realize clustering, distinguishes the communication nodes that generate traffic.

[0007] The technical solutions provided by the present invention are as follows:

[0008] The method of inferring the intranet topology based on traffic after address translation of the present invention includes the following steps:

[0009] 1) Traffic segmentation. The network traffic data in pcap format captured outside the gateway is compared<srcip,dstip,sport,dport,protocol> The five-tuple information is divided into a series of network flows, each of which contains a series of data packets, and each network flow is stored as a pcap package;

[0010] 2) Feature extraction: By analyzing the data packets obtained after segmentation, the information that can identify the internal network host in the network protocol field is selected as the benchmark features for subsequent host clustering, including ip_timestamp, ip_ipid, ip_ttl, tcp_timestamp, tcp_srcip, tcp_srcport, tcp_dstip, tcp_dport, http_path, http_parameters, http_host, http_user-agent, http_content-type, http_cookie, http_proxyauthorization, http_x-csrf-token, tls_servernameindication, tls_ciphersuits, tls_supportedgroups, tls_extensions, tls_sessionid, tls_sessionticket, dns_timestamp, dns_srcip, dns_answer, etc. Other field features in the protocol that are not included in the baseline features but may be helpful for clustering are used as auxiliary candidate features, including ip_version, ip_headerlength, ip_tos, tcp_sequencenumber, http_accept, tls_compressionmethod, dns_type, dns_qclass, etc.

[0011] 3) Host clustering,Host clustering is divided into two stages,

[0012] ① The first stage is base cluster generation. The features obtained above are divided into several small categories according to their representation direction. The network flow is clustered according to the features of each small category to obtain multiple base clusters. Furthermore, the features for generating base clusters can be divided as follows:

[0013] a) IP protocol layer characteristics;

[0014] b) TCP protocol layer characteristics;

[0015] c) TLS protocol layer characteristics;

[0016] d) Application layer protocol layer characteristics;

[0017] e) request-response correlation characteristics;

[0018] f) Periodic access feature.

[0019] ②The second stage is base cluster merging, where the multiple base clusters obtained above are merged through a consensus function to obtain a unified and more accurate host clustering result. The consensus function is a consistency integration function, which includes matrix-based (such as co-cooperation matrix) consensus functions, graph-based consensus functions, and so on. Furthermore, a co-cooperation matrix is ​​calculated based on the obtained base clusters as an intermediate representation of the relationship between data, where each point in the co-cooperation matrix represents the similarity between two sample points (i.e., network sessions, which are usually in the form of pcap files, and a network session is composed of a group of data packets). ROCK clustering, spectral clustering, or hypergraph clustering is then used to cluster the data in the co-cooperation matrix again to obtain the final clustering result.

[0020] 4) Topology inference: Using the number of network hosts determined by clustering and other information in the network flow, the specific network topology hierarchy is inferred. Clustering can group network traffic from the same host into a single category, so the number of clusters corresponds to the number of hosts. After determining the number of hosts based on clustering in step 3, the connectivity between hosts, i.e., the network topology hierarchy, is determined based on other information.

[0021] Furthermore, the topology of routing and host hierarchies can be inferred based on the fact that the TTL value of an IP packet decrements by 1 with each router it passes through. Based on the fact that some processes on the same operating system only have one copy, application information about the relevant processes can be extracted from network flows for topological correction. This allows for the separation of clustered network nodes through services such as operating system update services and time synchronization. For example, WeChat, email clients, and system updates can only have one process on a single host, and the previous clustering may have allocated traffic from two processes to a single host. Based on the fact that some processes only have one copy on the same host, this mis-clustered traffic can be corrected.

[0022] The present invention proposes a method for inferring intranet topology based on traffic information. First, the traffic captured outside the network is segmented into smaller network flows and data packets that are conducive to host clustering according to the TCP / IP four-layer network model; then, key features that are conducive to host clustering are extracted from the segmented network units; then, a cluster fusion algorithm is used to infer the number of hosts inside the network through two-layer clustering. Finally, combined with the changes in the TTL value in the IP protocol, the level of the host topology in the network is determined.

[0023] The present invention can accurately infer the number of hosts within the network and the nested hierarchical structure of the hosts from the traffic after address conversion in a non-cooperative situation, helping defenders to examine their own networks from the attacker's perspective, thereby blocking feasible attack paths and formulating more targeted defense strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1is a flow chart of the method of the present invention;

[0025] Figure 2 This is the flowchart of base clustering fusion. DETAILED DESCRIPTION

[0026] The method for inferring intranet topology based on traffic information of the present invention is described in detail below with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0027] The process of the method of the present invention is as follows Figure 1 As shown, its working process is as follows:

[0028] 1. Traffic segmentation.

[0029] Traffic segmentation breaks network traffic data captured externally into smaller, more easily clustered units. Traffic captured externally is often in pcap format. The traffic segmentation module analyzes the network from a traffic perspective, first segmenting traffic data into network flows to generate a collection of network flows. It then performs packet-level analysis on these flows to generate a collection of data packets.

[0030] In network flow division, the present invention compares and analyzes<srcip,dstip,sport,dport,protocol> Traffic packets are classified into network flows using a 5-tuple consisting of the following: srcip and dstip represent the source and destination IP addresses of the communicating parties; sport and dport represent the source and destination port numbers of the communicating parties; protocol represents the protocol type; and timestamp represents the timestamp. First, traffic packets are traversed to obtain valid 5-tuple information from the packets. Then, by comparing the 5-tuple information, packets with the same 5-tuple are grouped together to obtain the network flow.

[0031] Network packet segmentation categorizes network flows according to network protocols: IP, TCP, HTTP, TLS, and DNS. The TCP / IP four-layer network model divides the network into the network interface layer, network layer, transport layer, and application layer. Because the protocols at different layers have different formats and contain different information, categorizing network flows by protocol facilitates formatted processing and analysis. For example, the network interface layer is closely related to hardware. After NAT conversion, analyzing network interface layer information makes it difficult to obtain valid host information. However, analyzing the network layer, transport layer, and application layer protocols can reveal information related to the intranet topology. Therefore, this paper categorizes network flows according to network protocols into IP network flows, TCP network flows, HTTP network flows, TLS network flows, and DNS network flows. Each type of network flow is processed at the packet level to obtain IP packets, TCP packets, HTTP packets, TLS packets, and DNS packets.

[0032] Traffic segmentation divides traffic packets captured from outside the network into five-tuples, resulting in a collection of all network flows. These flows are then categorized at the protocol level according to the TCP / IP four-layer network model. The resulting network flows and packets include IP packets, TCP data flows, HTTP packets, TLS packets, and DNS packets, which serve as the foundation for subsequent processes.

[0033] 2. Feature extraction.

[0034] Datum Features:

[0035] a) IP packet baseline characteristics:

[0036] ip_timestamp, IP packet Timestamp field, IP packet capture timestamp;

[0037] ip_ipid, IP protocol identification field value;

[0038] ip_ttl, IP protocol Tive To Live field value;

[0039] b) TCP data flow benchmark characteristics:

[0040] tcp_timestamp: TCP timestamp field value, timestamp of flow capture;

[0041] tcp_srcip: TCP flow Source IP Address field value, source IP address;

[0042] tcp_srcport: TCP stream Source Port field value, source port address;

[0043] tcp_dstip: TCP flow Destination IP Address field value, destination IP address;

[0044] tcp_dport: TCP stream Destination Port field value, destination port address;

[0045] c) HTTP packet baseline characteristics:

[0046] http_path: separated from the original URI of the HTTP protocol, indicating the interface of a service exposed by the server;

[0047] http_parameters: separated from the original URI of the HTTP protocol, indicating the parameters provided when requesting the interface;

[0048] http_host: host domain name or IP address of HTTP request;

[0049] http_user-agent: HTTP User-Agent field value, indicating the user agent. The same user agent means the same application software and version;

[0050] http_content-type: The value of the HTTP Content-Type field, which indicates the content type. It defines the type and encoding of the file uploaded by the client and determines the format and encoding in which the browser reads the file.

[0051] http_cookie: HTTP request header, usually records user-related information and carries unique identification information such as user ID;

[0052] http_proxyauthorization: This is the HTTP request header that contains the credentials provided by the user agent to the proxy server for authentication;

[0053] http_x-csrf-token: A header introduced by the HTTP protocol to prevent cross-site request forgery (CSRF / XSRF) attacks.

[0054] d) TLS packet baseline characteristics:

[0055] tls_servernameindication: An extension in the TLS ClientHello packet that indicates the domain name of the site to be accessed and can distinguish different applications;

[0056] tls_ciphersuits: a sequence of code names for the encryption suites used during the TLS handshake process;

[0057] tls_supportedgroups: A TLS extension that indicates the named groups that the client supports for key exchange, arranged in descending order of priority. This field is processed to a sequence of integers.

[0058] tls_extensions: the list of extensions supported by the client;

[0059] tls_sessionid: Session ID, a randomly generated string. The Session IDs of consecutive sessions on the same host and the same server are consistent.

[0060] tls_sessionticket: ticket, which saves the encrypted information of session data.

[0061] e) DNS packet baseline characteristics:

[0062] dns_timestamp: DNS packet capture time;

[0063] dns_srcip: DNS packet source IP address;

[0064] dns_answer: DNS answer.

[0065] Auxiliary candidate features:

[0066] f) The auxiliary candidate features of the IP data packet include all field features of the IP data packet, such as:

[0067] ip_version: IP protocol Version field, IP protocol version;

[0068] ip_headerlength: IP protocol Header Length field, header length;

[0069] ip_tos: IP protocol Type of service field, service type;

[0070] ip_totallength: IP protocol Total Length field, the length of the IP datagram;

[0071] ip_flag: IP protocol flag field, indicating the datagram fragmentation setting;

[0072] ip_fragmentoffset: IP protocol fragment offset field, the offset of the fragment relative to the data part of the original IP datagram;

[0073] ip_protocal: IP protocol field, used to distinguish upper-layer protocols on the IP protocol.

[0074] ip_srdadd and ip_dstadd: IP protocol source IP address and IP protocol destination IP address;

[0075] g) TCP packet auxiliary candidate features include all field features of TCP network flow, such as:

[0076] tcp_sequencenumber: TCP protocol Sequence Number field, sequence number;

[0077] tcp_acknowledgement number: TCP protocol Acknowledgement Number field, confirmation number;

[0078] tcp_windowsize: TCP protocol Window Size field, window size;

[0079] tcp_checksum: TCP protocol Checksum field, checksum;

[0080] tcp_urgentpointer: TCP protocol Urgent Pointer field, urgent pointer;

[0081] h) HTTP auxiliary candidate features include all field features of TCP packets, such as:

[0082] http_accept: Accept field of HTTP protocol, which indicates the acceptable response content type;

[0083] http_acceptcharset: Accept-Charset field of HTTP protocol, acceptable character set;

[0084] http_acceptencoding: HTTP protocol Accept-Encoding field, a list of encoding methods that the protocol can accept;

[0085] http_acceptdatetime: Accept-Datetime field of the HTTP protocol, which indicates the acceptable versions expressed in time.

[0086] http_authorization: HTTP protocol Authorization field, used for authentication information of Hypertext Transfer Protocol;

[0087] http_connection: HTTP protocol Connection field, the connection type you want to use first;

[0088] http_contentlength: HTTP protocol Content-Length field, the length of the request body expressed as an octet array (8-bit bytes);

[0089] http_contentmd5: The binary MD5 hash value of the content of the HTTP protocol request body;

[0090] http_date: HTTP protocol Date field, the date and time the message was sent;

[0091] http_expect: The HTTP protocol Expect field indicates that the client requires the server to perform specific actions;

[0092] http_from: HTTP protocol From field, the email address of the user who initiated this request;

[0093] http_origin: HTTP protocol Origin field, initiates a request for cross-origin resource sharing;

[0094] http_pragma: HTTP protocol pragma fields, which are implementation-dependent and may have various effects at any time in the request / response chain;

[0095] http_range: HTTP protocol range field, requesting a portion of an entity. Byte offset starts at 0;

[0096] i) TLS auxiliary candidate features include all other field features except TLS baseline features, such as:

[0097] tls_compressionmethod: TLS CompressionMethod field, the compression algorithm selected by the server;

[0098] tls_certificatechainlength: the number of bytes occupied by the TLS certificate chain;

[0099] tls_certificatelength: the number of bytes occupied by the TLS certificate;

[0100] tls_certificatecontent: TLS certificate content;

[0101] j) DNS auxiliary candidate features include all other field features other than the DNS packet baseline features, such as:

[0102] dns_type: the type of requested resource;

[0103] dns_qclass: the type of resource record requested;

[0104] dns_qname: requested domain name;

[0105] dns_qtype: the type of resource record requested;

[0106] dns_class: the type of resource record being responded to;

[0107] dns_name: the requested domain name;

[0108] dns_type: defines the content in the RDATA section;

[0109] dns_ttl: The number of seconds that the response resource record is cached

[0110] The baseline features extracted in this step are necessary for host clustering. Depending on the actual network conditions and the quality of the captured packets, some or all of the auxiliary candidate features can be selected to improve the accuracy.

[0111] 3. Host clustering process

[0112] Based on the extracted features, the present invention employs a cluster fusion method to cluster each segmented network flow, obtained in step 1, by host, and predict the number of hosts within the network. The cluster fusion module comprises two submodules: a base cluster generation module and a base cluster fusion module. The base cluster generation module analyzes and clusters the segmented network flows and data packets along multiple dimensions, generating host clustering results corresponding to each dimension. The base cluster fusion module fuses these multidimensional base cluster labels to produce a unified, more accurate host clustering result.

[0113] The base cluster generation module generates base clusters of 6 different dimensions through IP flow clustering, TCP flow clustering, HTTP flow clustering, TLS flow clustering, DNS-TCP association and DNS periodic request association. The base clustering example is shown below

[0114]

[0115] A. IP flow clustering

[0116] IP flow clustering is achieved by clustering IP packets using the SLC algorithm. The input of the SLC algorithm is the IP protocol layer features:<ip_flowid,ip_timestamp,ip_ipid,ip_ttl> , sorted by ip_timestamp. The SLC algorithm's distance function is a modified Manhattan distance. The Manhattan distance between two different IP packets is the sum of the distances of the four fields listed above. The ip_timestamp distance, ip_ipid distance, and ip_ttl distance are the absolute values ​​of the differences between the corresponding fields. The ip_flowid distance is a Boolean value indicating whether the ip_flowids of the two packets are equal. The return value of the distance function is set between 0 and 100. The distance value is a relative value used by the SLC algorithm for comparison when selecting a more appropriate heap. When the arrival time of two packets exceeds the set time limit, the two packets are considered unlikely to be related and the maximum value of 100 is returned directly; otherwise, the Manhattan distance between the two packets is returned. Packets with a distance greater than a threshold are assigned a new category. By analyzing the distance function, the threshold can be set between 15 and 30. Finally, a Map operation is used to map the packet labels to flows. The MAP function receives the network packet and the corresponding host label as input, traverses the host labels of all data packets, selects the set packet_set of all data packets with the host label L, takes the set ip_flowids of the corresponding network flow ids in packet_set, sets the label label for all flows in the set ip_flowids, and finally obtains the clustering result of the IP flow: ip_tag.

[0117] B. TCP Flow Clustering

[0118] TCP flow clustering uses the SLC algorithm to cluster TCP network flows. The distance function is still the modified Manhattan distance, and the distance threshold is set between 5 and 15. The algorithm input is the TCP protocol layer features:<tcp_timestamp,tcp_srcport,tcp_dstip,tcp_dstport,http_ttl> The clustering result of TCP flow is tcp_tag.

[0119] C. HTTP Flow Clustering

[0120] HTTP flow clustering uses spectral clustering to cluster HTTP packets, and the input is HTTP protocol layer features: <http_path,http_parameters,http_host,http_user-agent,http_content-type,http_cookie,http_proxy-authentication,http_x-csrf-token> The spectral clustering algorithm first calculates the global similarity matrix S, which is calculated from the local similarity matrices of each field. The similarity metrics used include direct-similarity, levenshtein-similarity, cookie-similarity, and parameter-similarity.

[0121] Direct-similarity compares objects directly by string comparison. If two objects are equal, the similarity is 1, otherwise it is 0, as shown in Formula 1:

[0122]

[0123] Levenshtein-similarity calculates the edit distance, that is, the minimum number of edit operations required to convert one string into another, as shown in Formula 2, where lev is the Levenshtein ratio:

[0124] S(a,b)=(len(a)+len(b)-lev(a,b)) / (len(a)+len(b)) Formula 2

[0125] Cookie-similarity is a custom similarity function for cookies. When both the key and value are equal, the count cnt is incremented by one. When the key is equal but the value is different, the count cnt is decremented by one. Cookie similarity is the ratio of twice the cnt to the total number of keys. Cookie data is generally client-specific. When the key is the same but the value is different, the delta function imposes a penalty of -1. The calculation of the cookie-similarity function is shown in Formula 3:

[0126]

[0127] Parameter-similarity is a custom similarity function for parameter types. It differs from cookie-similarity in that it uses a different delta function. If the values ​​are not equal, the cookie-similarity delta function returns 1, while the parameter-similarity delta function returns 0. The calculation is shown in Formula 4:

[0128]

[0129] The fields http_path, http_proxy-authentication, and http_x-csrf-tocken use the direct-similarity function to calculate similarity; http_parameter uses the parameter-similarity function to calculate similarity; http_host, http_user-agent, and http_content-type use the leveshtein-similarity function to calculate similarity; and the field http_cookie uses the cookie-similarity function to calculate similarity. The global similarity matrix S is calculated by the weighted average of all local similarity matrices, as shown in Formula 5. Here, α is the total number of fields. Since there are 8 fields here, α = 8; S k is the k-th dimension local similarity matrix, with a size of n×n, where n represents the number of HTTP packets; W k Denotes the weight of the kth dimension, which is given empirically. Spectral clustering is then applied to the global similarity matrix S to obtain the clustering results. Finally, the packet labels are mapped to flows through a MAP operation.

[0130]

[0131] D. TLS Flow Clustering

[0132] The TLS flow clustering process is similar to HTTP flow clustering. Spectral clustering is used to cluster TLS packets by host to obtain TLS packet-level cluster labels. Finally, a label conversion operation is used to map the packet-level label to a flow-level label, obtaining the host label tls_tag of the TLS flow. The input for TLS flow clustering is the TLS protocol layer features: <tls_servernameindication,tls_ciphersuits,tls_supportedgroups,tls_extensions,tls_sessionid,tls_sessionticket> When calculating the similarity matrix, the similarity function used for tls_servernameindication, tls_ciphersuits, tls_supportedgroups, tls_extensions, and tls_sessionid is the direct-similarity function, and the similarity function for tls_sessionticket is the levenshtein-similarity function.

[0133] E. DNS-TCP Correlation

[0134] This method clusters hosts based on the correlation between DNS requests and TCP responses, leveraging the fact that the IP address of the DNS response is equal to the IP address of the TCP connection. Most applications, when providing services, first initiate a DNS request to resolve the domain name, obtain the actual IP address, and then establish a TCP connection with that IP address before communication can begin.

[0135] For example, if an intranet host wants to access www.baidu.com for web browsing, it first sends a DNS domain name resolution request to the domain name server, asking for the IP address of www.baidu.com. The domain name server queries the database and obtains the corresponding IP address: 103.235.46.39. It then returns the DNS response to the intranet host. After obtaining the IP address, the intranet host immediately initiates a TCP connection to IP 103.235.46.39, establishing TCP communication.

[0136] In the above process, the IP address of the DNS response is consistent with the IP address of the TCP connection. This feature can be used to associate the DNS data packet with the TCP data packet. iInclude<tls_timestamp,tls_flowid,tls_type,tls_qname,tls_qtype,tls_qclass,tls_answer> After segmentation, the DNS requests in the same network flow are traversed, the IP address returned in the answer segment of the DNS response is retrieved, and a TCP SYN connection initiated within the time limit for that IP address is searched. If the IP address in the SYN connection is equal to the IP address in the DNS response, the DNS packet and the TCP packet are labeled with the same label. Finally, a Map operation is used to map the packet label to a flow label, resulting in the flow label dns_tag.

[0137] F. DNS flow cycle request association

[0138] This method clusters hosts using the characteristics of periodic DNS request access. To complete specific tasks, both operating systems and application software issue periodic network requests. Operating system update requests include: Ubuntu checks network connectivity (requesting connectivitycheck.ubuntu.com); Windows periodically updates the time (requesting times.windows.com); and the NTP service schedules time updates (requesting a series of NTP server domain names, such as ntp.ubuntu.com, rhel.pool.ntp.org, and asia.pool.ntp.org). Application-related periodic tasks include: email agents periodically checking for new emails according to their configuration; antivirus software regularly updates local virus databases; and software such as Google / Chrome, QQ Browser, and WPS periodically checks for updates. By using periodic requests from operating systems and application software and searching for arithmetic progressions, network packets can be correlated over a longer timeframe, assigning the same label to associated DNS packets. DNS packets with time intervals in an arithmetic progression belong to the same host. A Map operation is then used to convert the DNS packet labels into flow labels, yielding the periodic request label, period_tag.

[0139] So far, through the above 6 clustering methods and association methods, we have obtained 6 base clusters, namely ip_tag, tcp_tag, http_tag, tls_tag, dns_tag, and period_tag. ip_tag and tcp_tag are the divisions of network flows according to hosts, http_tag and tls_tag are the divisions of network flows according to hosts and applications, and dns_tag and period_tag ​​are the divisions of network flows according to hosts in short and long time periods respectively. Since the fields and methods used for clustering or association are different, the cluster labels obtained are also different. Moreover, the above algorithms all have certain pre-assumptions about the network flow itself. If the distribution of the network flow does not conform to this assumption, a completely wrong clustering result will be given. Therefore, we use cluster fusion to merge the clustering results obtained using different parameters and methods, so as to obtain better results than a single algorithm. The base cluster fusion process is as follows: Figure 2 As shown, first generate a co-association matrix. Each point in the co-association matrix represents the similarity between two sample points. Its calculation method is shown in Formula 6, where A t (x t ) represents the object x t In the base cluster A t The value below, here A t Indicates the cluster labels obtained at different protocol levels. δ(a,b) returns 1 if a and b are equal, and returns 0 if they are not equal. i In the base cluster A t If the value of is -1, the delta function returns 0. ij Indicates the number of times network flows i and j share the same cluster label. The co-cooperation matrix includes ip_tag, tcp_tag, http_tag, and tls_tag. However, the inclusion of dns_tag and period_tag ​​depends on the quality of packet capture (e.g., whether the captured packets have the expected characteristics). The key idea behind creating the co-cooperation matrix is ​​to reorganize the partitioning information of all base clusters to fully utilize it, accurately and digitally representing the similarity between data. Therefore, each point in the co-cooperation matrix represents the similarity between two sample points. Once the co-cooperation matrix is ​​obtained, the objects within it are clustered to obtain the final clustering result.

[0140] In this process, the more commonly used clustering methods include ROCK clustering, spectral clustering and hypergraph clustering METIS.

[0141]

[0142] A t (x i) represents the i-th object x i In the tth base cluster A t The labels in , m is the total number of base clusters.

[0143] 4. Topology inference process.

[0144] The topology inference module infers the network hierarchy of a host. This is done based on the TTL value of an IP packet. TTL is a field in the IP protocol. Each time an IP packet passes through a routing device, the TTL value decreases by 1. Therefore, the network hierarchy of a host can be inferred from the TTL field in an IP packet. Different operating systems have different initial TTL values. The default initial TTL value for GNU / Linux is 64, the default initial TTL value for Windows is 128, and the default initial TTL value for most routers is 255. This poses some challenges in inferring the network hierarchy of a host. However, a good characteristic of network hop count is that it never exceeds 30 hops. The TTL value range for GNU / Linux is [34, 64]; the TTL value range for GNU / Linux is [98, 128]; and the TTL value range for routers is [225, 255]. Therefore, based on this characteristic, the initial TTL value can be normalized by mapping all TTL values ​​to the same range, enabling direct comparison. The mapping method is shown in Figure 7. Where ttl represents the TTL value in the IP protocol. map Indicates the TTL value after mapping.

[0145]

[0146] Although specific embodiments and drawings of the present invention are disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. The present invention should not be limited to the preferred embodiments disclosed in this specification and the drawings. The scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for identifying an intranet topology based on traffic after address translation, comprising the following steps: 1) Based on the five-tuple information of the traffic, the network traffic data captured outside the gateway is divided into a series of network flows; 2) extracting features from the data packets in each of the network flows, and selecting information identifying the internal network host in the network protocol field of the data packet as a benchmark feature for clustering the internal network hosts; 3) The baseline features obtained in step 2) are divided into the following six categories: a) IP protocol layer features, b) TCP protocol layer features, c) TLS protocol layer features, d) application layer protocol layer features, e) request-response correlation features, and f) periodic access features. These six categories are then clustered using the baseline features corresponding to each subcategory to obtain six base clusters. These multiple base clusters are then merged to obtain the intranet host clustering results. 4) Determine the number of network hosts based on the intranet host clustering results, and infer the topology of routes and hosts based on the characteristic that the TTL value of an IP packet decreases by 1 each time it passes through a router layer, determine the network topology layer, and obtain the intranet network topology.

2. The method according to claim 1, wherein The baseline features include IP packet capture timestamps ip_ timestamp 、IP protocol identification field value ip_ipid 、IP protocol Tive To Live field value ip_ ttl , timestamp of stream capture tcp_timestamp , source IP address tcp_srcip , source port address tcp_srcport , target IP address tcp_dstip , destination port address tcp_dport , service interface http_path , Parameters provided when requesting the service interface http_parameters , the requested host domain name http_host , User Agent http_user-agent , content type http_content-type , user identification information http_cookie , authentication credentials http_ proxyauthorization , HTTP protocol introduced header to prevent cross-site request forgery attacks http_x-csrf- token , the domain name of the site being visited tls_servernameindication , the code sequence of the cipher suite used during the TLS handshake tls_ciphersuits , the client supports named groups for key exchange tls_supportedgroups , client-supported extension list tls_extensions , Session ID tls_sessionid , ticket tls_ sessionticket , DNS packet capture time dns_timestamp , DNS packet source IP address dns_srcip and DNS responses dns_ answer .

3. The method according to claim 2, wherein Auxiliary candidate features are extracted from the data packets in each of the network flows, including all field features of IP data packets, all field features of TCP network flows, and all field features of TCP data packets.

4. The method according to claim 1, 2 or 3, wherein: In step 4), a co-cooperation matrix is ​​calculated based on the obtained base clustering as an intermediate representation of the relationship between the data, where each point in the co-cooperation matrix represents the similarity between two network conversation data; then, ROCK clustering, spectral clustering, or hypergraph clustering is used to cluster the data in the co-cooperation matrix again to obtain the final base clustering result.

5. The method according to claim 4, wherein By formula Generate the co-correlation matrix; Representative object In the base clustering The value below, Representative object In the base clustering The value under, function Returns 1 if a and b are equal, and 0 if they are not equal; if the object or In the base clustering If the value is -1, then The function returns 0; Represents a network flow and network streams The number of times with the same cluster label, m is the total number of base clusters.

6. The method according to claim 1, wherein Based on the fact that some processes on the same operating system have only one copy, the application information of the corresponding process is extracted from the network flow to perform topology correction.

7. The method according to claim 1, wherein The obtained multiple base clusters are merged through the consensus function to obtain the host clustering result.

8. The method according to claim 7, wherein The consensus function is a consistency integration function, including a matrix-based consensus function and a graph-based consensus function.

Images