Network asset identification method based on structural similarity

Abstract

The invention discloses a network asset identification method based on structural similarity, and belongs to the field of network space security. The method comprises the steps of obtaining webpage data of a plurality of domain names in a webpage crawling mode; constructing a page label tree and a url tree of the domain name; determining the asset ownership of each domain name; constructing a domain name asset database; constructing a page tag tree and a url tree of the domain name for the domain name of which the asset ownership is to be determined; and searching and comparing the page tag tree of the domain name of which the asset ownership is to be determined and the url tree in the database to find the most matched record, thereby determining the asset ownership of the domain name. The web asset identification method can effectively utilize the structural characteristics of the web assets, and can effectively identify the assets according to the similarity of the similar web assets on the network structure or the page layout architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of network security, and in particular to a network asset identification method based on structural similarity. Background Art

[0002] Network asset identification is the detection, analysis and understanding of resources in cyberspace, so as to identify various categories of assets and get a clear picture of the situation before conducting network security analysis. It is the foundation of network security.

[0003] With the continuous development of information technology, the types of network resources in cyberspace are becoming more and more abundant, the number of resources is increasing, and the asset relationships are becoming more and more complex. A large number of network assets are spread throughout cyberspace. These network assets that have not been analyzed and identified may become targets of attackers, or even become a springboard for invading core network assets. Therefore, network asset identification is an important task in maintaining cyberspace security.

[0004] Existing network asset identification technologies are mainly divided into active detection and passive detection. Active detection technology minimizes interaction with various application layer protocols by actively sending packets to identify services, devices and versions in the network space. Passive detection mainly detects specific network devices by capturing and analyzing traffic at the network entrance and exit. For web assets, the same or similar web assets often have high similarity in the web architecture or web page layout framework, which can be used to identify assets. Existing methods cannot take advantage of the structural similarity of web assets themselves, and thus cannot accurately identify them. Summary of the invention

[0005] In order to utilize the structural similarity of web assets, the present invention provides a network asset identification method based on structural similarity, which can effectively utilize the structural characteristics of web assets and perform effective asset identification based on the similarity of similar web assets in network structure or page layout architecture.

[0006] The object of the present invention is achieved in that:

[0007] A network asset identification method based on structural similarity includes the following steps:

[0008] Obtain web page data of multiple domain names by crawling the web pages;

[0009] For each domain name, build the page tag tree and URL tree of the domain name;

[0010] Determine the asset ownership of each domain name;

[0011] Build a domain name asset database, which contains the corresponding relationship between asset ownership and page tag tree and URL tree;

[0012] For a domain name whose asset ownership needs to be determined, construct the page tag tree and URL tree of this domain name;

[0013] Search and compare the page tag tree and URL tree of the domain name whose asset ownership needs to be determined in the database to find the most matching record, so as to determine the asset ownership of this domain name.

[0014] Further, the method for constructing the page tag tree is as follows:

[0015] Obtain the page HTML source code of the specified domain name, and extract the page tags in the page HTML source code; if the page tag starts with "<", then add this tag as a node to the next node of the current node of the multi-way tree; if the page tag starts with "< / ", then return to the previous node until the traversal ends, and the page tag tree is constructed.

[0016] Further, the method for constructing the URL tree is as follows:

[0017] Construct a dictionary URL set, which contains preset typical page names;

[0018] Obtain the page HTML source code of the specified domain name, create an empty URL set, and add the home page URL of the specified domain name to the URL set;

[0019] Obtain an unvisited URL from the URL set, obtain the page information of this URL, and then match all the URLs belonging to the specified domain name in the page information and add them to the URL set; repeat this step until all the URLs in the URL set have been visited;

[0020] Combine the typical page names in the dictionary URL set with the specified domain name to obtain new URLs, access these URLs, and add the actually existing URLs among them to the URL set;

[0021] Extract a URL from the URL set, and split the URL into a string array according to the symbols " / " and "?"; if the last string in the string array contains "&", then remove the last string from the string array, and split the last string into multiple substrings with "&" as the delimiter, then sort the substrings in ascending order. If the substring contains "=", only consider the part before "=" during sorting; add the sorted substrings to the end of the string array;

[0022] The string array corresponding to each URL corresponds to a path from the root node to the leaf node in the URL tree. According to the string arrays of all URLs in the URL set, the URL tree is obtained.

[0023] Furthermore, the page tag tree and URL tree of the domain name whose asset ownership is to be determined are searched and compared in the database. The specific method is as follows:

[0024] Traverse the database and perform the following steps for each record in the database:

[0025] Establish a correspondence between the URL tree of the domain name whose asset ownership is to be determined and the nodes of the URL tree in the current record. If the positions and string contents of the two nodes are the same, the two nodes correspond.

[0026] Calculate the similarity x between the URL tree of the domain name whose asset ownership is to be determined and the URL tree of the current record. The similarity x is expressed as the tree edit distance between the two URL trees, as follows:

[0027] x=e -(|R|+|D|+|I|)

[0028] Wherein, |R| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have the same position as that in the current record url tree but different string content, |D| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have different positions from that in the current record url tree, |I| is the number of nodes in the current record url tree that have different positions from that in the url tree of the domain name of the asset ownership to be determined;

[0029] Using dictionary mapping, starting from the root node, the page tag tree of the domain name of the asset ownership to be determined and the tags of the page tag tree in the current record are converted into one-dimensional strings in a breadth-first manner, and then the similarity y between the page tag tree of the domain name of the asset ownership to be determined and the page tag tree in the current record is calculated:

[0030] y=e -dis

[0031] Where dis is the edit distance between two one-dimensional strings;

[0032] Calculate the weighted sum s of x and y to obtain the similarity between the domain name of the asset to be determined and the currently recorded asset;

[0033] After the traversal is completed, the record with the greatest asset similarity to the domain name of the asset ownership to be determined is taken as the matching record, and the asset ownership of this record is the final determined asset ownership.

[0034] The beneficial effects of the present invention are:

[0035] 1. The present invention can effectively utilize the structural similarity of web assets to identify web assets;

[0036] 2. The present invention does not require the construction of special detection data packets, and can extract asset features by parsing web pages to perform network asset identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 The present invention is a flowchart of a method for identifying network assets based on structural similarity in an embodiment of the present invention.

[0039] Figure 2 It is a schematic diagram of a network asset identification system based on structural similarity in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0041] A network asset identification method based on structural similarity includes the following steps:

[0042] Obtain web page data of multiple domain names by crawling the web pages;

[0043] For each domain name, build the page tag tree and URL tree of the domain name;

[0044] Determine the asset ownership of each domain name;

[0045] Build a domain name asset database, which contains the corresponding relationship between asset ownership and page tag tree and URL tree;

[0046] For a domain name whose asset ownership is to be determined, build the page tag tree and URL tree of the domain name;

[0047] The page tag tree and URL tree of the domain name whose asset ownership is to be determined are searched and compared in the database to find the best matching record, thereby determining the asset ownership of the domain name.

[0048] Among them, the way to build the page tag tree is:

[0049] Obtain the page HTML source code of a specified domain name, and extract the page tags in the page HTML source code; if the page tag starts with "<", then add this tag as a node to the next node of the current node in the multi-way tree; if the page tag starts with "< / ", then return to the previous node until the traversal ends, and the page tag tree is constructed.

[0050] The method for constructing the URL tree is as follows:

[0051] Construct a dictionary URL set, which contains preset typical page names;

[0052] Obtain the page HTML source code of a specified domain name, create an empty URL set, and add the home page URL of the specified domain name to the URL set;

[0053] Obtain an unvisited URL from the URL set, obtain the page information of this URL, and then match all the URLs belonging to the specified domain name in the page information, and add them to the URL set; repeat this step until all the URLs in the URL set have been visited;

[0054] Combine the typical page names in the dictionary URL set with the specified domain name to obtain new URLs, access these URLs, and add the actually existing URLs among them to the URL set;

[0055] Extract a URL from the URL set, and split the URL into a string array according to the symbols " / " and "?"; if the last string in the string array contains "&", then remove the last string from the string array, and split the last string into multiple substrings with "&" as the delimiter, and then sort the substrings in ascending order. If a substring contains "=", only consider the part before "=" during sorting; add the sorted substrings to the end of the string array;

[0056] The string array corresponding to each URL corresponds to a path from the root node to the leaf node in the URL tree. According to the string arrays of all the URLs in the URL set, the URL tree is obtained.

[0057] Search and compare the page tag tree and the URL tree of the domain name whose asset ownership is to be determined in the database. The specific method is as follows:

[0058] Traverse the database, and for each record in the database, perform the following steps:

[0059] Establish the corresponding relationship between the URL tree of the domain name whose asset ownership is to be determined and the nodes in the URL tree of the current record. If the positions and string contents of the two nodes are the same, then the two nodes correspond;

[0060] Calculate the similarity x between the URL tree of the domain name whose asset ownership is to be determined and the URL tree of the current record. The similarity x is expressed as the tree edit distance between the two URL trees, as follows:

[0061] x=e -(|R|+|D|+|I|)

[0062] Wherein, |R| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have the same position as that in the current record url tree but different string content, |D| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have different positions from that in the current record url tree, |I| is the number of nodes in the current record url tree that have different positions from that in the url tree of the domain name of the asset ownership to be determined;

[0063] Using dictionary mapping, starting from the root node, the page tag tree of the domain name of the asset ownership to be determined and the tags of the page tag tree in the current record are converted into one-dimensional strings in a breadth-first manner, and then the similarity y between the page tag tree of the domain name of the asset ownership to be determined and the page tag tree in the current record is calculated:

[0064] y=e -dis

[0065] Where dis is the edit distance between two one-dimensional strings;

[0066] Calculate the weighted sum s of x and y to obtain the similarity between the domain name of the asset to be determined and the currently recorded asset;

[0067] After the traversal is completed, the record with the greatest asset similarity to the domain name of the asset ownership to be determined is taken as the matching record, and the asset ownership of this record is the final determined asset ownership.

[0068] Here is a more specific example:

[0069] like Figure 1 As shown in FIG. 1 , a network asset identification method based on structural similarity has the following process:

[0070] S101, obtaining the asset domain name of target A;

[0071] S102, the detection module detects and obtains the HTML source code of the target A page, and sends it to the feature extraction module;

[0072] S103, the feature extraction module constructs a page tag tree and sends it to the feature storage module for storage;

[0073] Specifically, the feature extraction module extracts page tags from the page html source code; if the page tag starts with "<", add the node to the next node of the current node of the multi-way tree; if the page tag starts with "< / ", return to the previous node until the traversal ends. Thus, the page tag tree is constructed and sent to the feature storage module.

[0074] S104, the detection module detects and obtains all the urls of target A, constructs a set of urls, and sends it to the feature extraction module;

[0075] Specifically, the detection module constructs a set of urls using two methods: traversal search and dictionary matching. Set the target url set and add the home page url of target A to the set; obtain an unvisited url from the set, get the page information of this url, and then match all the urls belonging to target A in the page information and add them to the target url set; repeat the previous step until all the urls in the set have been visited; traverse and visit the urls in the dictionary url set and add the existing urls to the target url set. Thus, the url set of target A is obtained and sent to the feature extraction module.

[0076] S105, the feature extraction module constructs a url tree and sends it to the feature storage module for storage;

[0077] Specifically, extract a url from the url set, and split the url into several ordered arrays according to " / " and "?"; if the last item in the string array contains "&", then split the string according to "&", and sort the substrings according to the result of string comparison of the fields before "=" in each item of the substring, and add them to the ordered array one by one; construct a path from the root to the leaf according to the ordered array, with the root being the site part of the url; extract urls from the url set in turn and add them to the url. Thus, the url tree is constructed and sent to the feature storage module.

[0078] S106, the feature comparison module extracts the page tag tree and url tree of a target asset from the feature storage module, and mark this target as target B;

[0079] S107, the feature comparison module calculates the similarity x of the url trees of target A and B;

[0080] Specifically, represent the similarity by the tree edit distance between the url trees of target A and B; let T be the url tree, and T[i] be the i-th node in the preorder traversal of T, then the mapping set M(i,j) between the tree T1 with the number of nodes n1 and the tree with the number of nodes n2 satisfies, for any (i1,j1), (i2,j2):

[0081] 1) i1 = i2 if and only if j1 = j2;

[0082] 2) If T1[i1] is to the left of T1[i2], then T2[j1] is to the left of T2[j2];

[0083] 3) If T1[i1] is the ancestor node of T1[i2], then T2[j1] is the ancestor node of T2[j2].

[0084] Then the similarity between the URL tree of target A and the URL tree of target B is x = e -(|R|+|D|+|I|) , |R| is the cardinality of the set of unmatched node pairs (i, j), |D| is the cardinality of the set of nodes in T1 that do not appear in M, and |I| is the cardinality of the set of nodes in T2 that do not appear in M. So far, the similarity x of the url trees of targets A and B is obtained.

[0085] S108, the feature comparison module calculates the similarity y of the page tag trees of targets A and B;

[0086] Specifically, the two page tag trees are converted into one-dimensional strings from the root node breadth-first in a dictionary mapping manner; then the edit distance dis of the two strings is calculated, and the similarity y of the two page tag trees is y = e -dis . So far, the similarity y of the page tag trees of targets A and B is obtained.

[0087] S109, the feature comparison module calculates the asset similarity s between targets A and B;

[0088] Specifically, the asset similarity s between targets A and B is s=a×x+(1-a)×y, where a∈(0,1) represents the weight. Thus, the asset similarity s between targets A and B is obtained.

[0089] S110, determining whether there are any uncompared targets in the feature storage module, if yes, proceeding to step S106; if no, proceeding to step S111;

[0090] S111, target B with the maximum similarity is calculated as a similar target asset of target A.

[0091] Based on the above method, a network asset identification system based on structural similarity can be implemented. Figure 2 As shown, the system includes a detection module, a feature extraction module, a feature storage module, and a feature comparison module; wherein:

[0092] The detection module is responsible for detecting the target assets, including detecting and acquiring all URLs of the target assets, detecting and acquiring the page HTML of the target assets, and sending the detected URL set and page HTML to the feature extraction module;

[0093] The feature extraction module is responsible for constructing the URL set of the target asset into a URL tree, constructing the page HTML of the target asset into a page tag tree, and sending the URL tree and page tag tree to the feature storage module and the feature comparison module;

[0094] The feature storage module is responsible for managing the URL tree and page tag tree of all target assets;

[0095] The feature comparison module is responsible for calculating and comparing the similarity of two target assets, including calculating and comparing the URL tree and the page tag tree.

[0096] In summary, the present invention can effectively utilize the structural features of web assets and perform effective asset identification based on the similarity of similar web assets in network structure or page layout architecture.

Claims

1. A network asset identification method based on structural similarity, characterized in that: It includes the following steps: Obtain the web page data of multiple domain names through web crawling; For each domain name, construct the page tag tree and url tree of this domain name; Determine the asset ownership of each domain name; Construct a domain name asset database, which contains the corresponding relationship between asset ownership and page tag tree and url tree; For a domain name whose asset ownership needs to be determined, construct the page tag tree and url tree of this domain name; Search and compare the page tag tree and url tree of the domain name whose asset ownership needs to be determined in the database, find the most matching record, so as to determine the asset ownership of this domain name.

2. A network asset identification method based on structural similarity according to claim 1, characterized in that: The method of constructing the page tag tree is: Obtain the page html source code of the specified domain name, extract the page tags in the page html source code; if the page tag starts with "<", then add this tag as a node to the next node of the current node of the multi-way tree; If the page tag starts with "< / ", then return to the previous node until the traversal ends and the page tag tree is constructed.

3. The network asset identification method based on structural similarity according to claim 1 is characterized in that: The method of constructing the url tree is: Construct a dictionary url set, which contains preset typical page names; Obtain the page html source code of the specified domain name, establish an empty url set, and add the home page url of the specified domain name to the url set; Obtain an unvisited url from the url set, obtain the page information of this url, and then match all the urls belonging to the specified domain name in this page information, and add them to the url set; repeat this step until all the urls in the url set are visited; Combine the typical page names in the dictionary url set with the specified domain name to obtain new urls, visit these urls, and add the actually existing urls to the url set; Extract a url from the url set, and split the url into a string array according to the symbols " / " and "?"; if the last string in the string array contains "&", then remove the last string from the string array, and split the last string into multiple substrings with "&" as the delimiter, and then sort the substrings in ascending order. If the substring contains "=", only consider the part before "=" during sorting; add the sorted substrings to the end of the string array; Each string array of a url corresponds to a path from the root node to the leaf node in the url tree. According to the string arrays of all urls in the url set, the url tree is obtained.

4. The network asset identification method based on structural similarity according to claim 1, characterized in that: Search and compare the page tag tree and url tree of the domain name whose asset ownership needs to be determined in the database. The specific method is: Traverse the database, and for each record in the database, perform the following steps: Establish the corresponding relationship between the nodes of the url tree of the domain name whose asset ownership needs to be determined and the nodes of the url tree in the current record. If the positions and string contents of the two nodes are the same, the two nodes correspond; Calculate the similarity x between the url tree of the domain name whose asset ownership needs to be determined and the url tree of the current record. The similarity x is represented by the tree edit distance between the two url trees, as follows: x=e -(|R|+|D|+|I|) Wherein, |R| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have the same position as that in the current record url tree but different string content, |D| is the number of nodes in the url tree of the domain name of the asset ownership to be determined that have different positions from that in the current record url tree, |I| is the number of nodes in the current record url tree that have different positions from that in the url tree of the domain name of the asset ownership to be determined; Using dictionary mapping, starting from the root node, the page tag tree of the domain name of the asset ownership to be determined and the tags of the page tag tree in the current record are converted into one-dimensional strings in a breadth-first manner, and then the similarity y between the page tag tree of the domain name of the asset ownership to be determined and the page tag tree in the current record is calculated: and=e -dis Where dis is the edit distance between two one-dimensional strings; Calculate the weighted sum s of x and y to obtain the similarity between the domain name of the asset to be determined and the currently recorded asset; After the traversal is completed, the record with the greatest asset similarity to the domain name of the asset ownership to be determined is taken as the matching record, and the asset ownership of this record is the final determined asset ownership.

Images