Website clustering method, system, computer device and storage medium

Abstract

The present application relates to a website clustering method, system, computer device, and storage medium. The method comprises: parsing the websites to be clustered to obtain a corresponding DOM tree structure; extracting various nodes of the DOM tree structure and obtaining corresponding matching vectors based on the various nodes of the DOM tree structure; and clustering the websites to be clustered based on the matching vectors to obtain clustering results. This application solves the problem of complex similarity calculation when clustering a large number of website pages. It realizes that the solution of calculating matching vectors replaces the existing method of calculating DOM tree similarity through complex algorithms. It is applicable to the problem of clustering a large number of websites and improves clustering efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of website classification, and in particular to a website clustering method, system, computer device and storage medium. Background Art

[0002] Cyberspace search engines can map the Internet's IP space by acquiring the network attributes of both physical and virtual resources, collectively discovering comprehensive assets, organizing and evaluating them, and performing the mapping of the Internet's IP space. When mapping virtual resources, large volumes of website page data are identified and generated, necessitating the clustering of similar website page data for further organization.

[0003] While the content presented on website pages may differ, websites built using the same source code are essentially template websites, with a largely identical code structure. Therefore, existing techniques typically use algorithms such as the tree edit distance algorithm and the Jaccard coefficient to calculate the similarity of the website's DOM tree structure for clustering. However, these methods are relatively complex and time-consuming, making them difficult to apply to clustering large numbers of website pages.

[0004] There is currently no effective solution to the problem of complex similarity calculation when a large number of website pages are clustered in related technologies. Summary of the Invention

[0005] In this embodiment, a website clustering method, system, computer device and storage medium are provided to solve the problem of complex DOM tree similarity calculation when clustering a large number of website pages in the related art.

[0006] First, in this embodiment, a website clustering method is provided, comprising:

[0007] Collect the websites to be clustered, and parse the websites to be clustered to obtain the corresponding DOM tree structure;

[0008] Extracting various nodes of the DOM tree structure, and obtaining corresponding matching vectors according to the various nodes of the DOM tree structure;

[0009] The websites to be clustered are clustered based on the matching vectors to obtain a clustering result.

[0010] In some embodiments, collecting the websites to be clustered and parsing the websites to be clustered to obtain the corresponding DOM tree structure includes:

[0011] Acquire page data of the website to be clustered and perform cleaning processing to obtain cleaned source code;

[0012] The source code is parsed and converted to obtain a corresponding DOM tree structure.

[0013] In some embodiments, extracting various types of nodes of the DOM tree structure and obtaining corresponding matching vectors according to the various types of nodes of the DOM tree structure includes:

[0014] Extracting the root node and the secondary node of the DOM tree structure and concatenating them to obtain a first node string; performing a hash operation on the first node string to obtain a basic framework matching vector;

[0015] The final node of the DOM tree structure is extracted and concatenated to obtain a second node string; a hash operation is performed on the second node string to obtain a global frame matching vector.

[0016] In some embodiments, extracting the root node and the secondary node of the DOM tree structure and concatenating them to obtain a first node string; extracting the final node of the DOM tree structure and concatenating them to obtain a second node string includes:

[0017] Extract the label names of the root node and the secondary node, and concatenate them in the extraction order to obtain the first node string;

[0018] Extract the label name of the final node and concatenate them in the extraction order to obtain the second node string.

[0019] In some embodiments, clustering the to-be-clustered websites based on the matching vectors to obtain clustering results includes:

[0020] Based on the basic framework matching vector, clustering is performed to obtain a preliminary clustering result;

[0021] In the preliminary clustering results, clustering is performed based on the global frame matching vector to obtain a final clustering result.

[0022] In some embodiments, further comprising:

[0023] In a cyberspace search engine, the basic framework matching vectors are calculated based on preset websites to obtain the preliminary clustering results;

[0024] In the preliminary clustering result, the global frame matching vector of the matching is calculated based on the preset website to obtain the final clustering result.

[0025] In some embodiments, further comprising:

[0026] The final clustering result is output and returned to the network space search engine.

[0027] In a second aspect, a website clustering system is provided in this embodiment, comprising: a collection module, a splicing module, and a clustering module;

[0028] The collection module is used to collect the websites to be clustered and parse the websites to be clustered to obtain the corresponding DOM tree structure;

[0029] The splicing module is used to extract various nodes of the DOM tree structure and obtain corresponding matching vectors according to the various nodes of the DOM tree structure;

[0030] The clustering module is used to cluster the websites to be clustered based on the matching vectors to obtain a clustering result.

[0031] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the website clustering method described in the first aspect is implemented.

[0032] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the website clustering method described in the first aspect is implemented.

[0033] Compared with related technologies, the website clustering method, system, computer device, and storage medium provided in this embodiment collect websites to be clustered and parse the websites to be clustered to obtain corresponding DOM tree structures; extract various nodes of the DOM tree structure and obtain corresponding matching vectors based on the various nodes of the DOM tree structure; cluster the websites to be clustered based on the matching vectors to obtain clustering results. This solves the problem of complex DOM tree similarity calculation when clustering a large number of website pages, and simplifies the process of clustering a large number of websites by calculating the matching vectors of the websites to be clustered.

[0034] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 FIG1 is a hardware structure diagram of a website clustering method according to an embodiment;

[0037] Figure 2 is a flowchart of a website clustering method in one embodiment;

[0038] Figure 3 A flowchart of a website clustering method in a preferred embodiment;

[0039] Figure 4 FIG. 4 is a structural block diagram of a website clustering system in one embodiment.

[0040] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 410, acquisition module; 420, splicing module; 430, clustering module. DETAILED DESCRIPTION

[0041] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0042] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0043] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a block diagram of the hardware structure of the terminal of the website clustering method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0044] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the website clustering method in this embodiment. The processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The network may include a wireless network provided by the terminal's telecommunications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] In this embodiment, a website clustering method is provided. Figure 2 is a flow chart of the method of this embodiment, such as Figure 2 As shown, the method includes the following steps:

[0047] Step S210 , collecting websites to be clustered, and parsing the websites to be clustered to obtain corresponding DOM tree structures.

[0048] Specifically, web pages are typically hypertext documents written in HTML, or HTML files. The HTML files of the websites to be clustered are parsed to obtain the DOM tree structure corresponding to the websites to be clustered. The DOM tree provides an access model for the entire HTML document, treating the document as a tree structure. Each node in the tree represents an HTML tag or a text item within a tag, and can describe the relationships between tags in the HTML document.

[0049] Step S220 , extracting various nodes of the DOM tree structure, and obtaining corresponding matching vectors according to the various nodes of the DOM tree structure.

[0050] Specifically, the DOM tree structure includes a root node, secondary nodes, and a final node. Each node corresponds to a document tag in the HTML document of the website to be clustered. For example, an HTML document includes multiple levels of tags, and each level of tags corresponds to a different node in the DOM tree structure.

[0051] Furthermore, each type of node is combined separately to calculate the corresponding matching vector. Among them, the matching vector generated by the combination of the root node and the secondary node of the DOM tree structure can, to a certain extent, represent the basic framework characteristics of the website to be clustered. The matching vector generated by the final node of the DOM tree structure can, to a certain extent, represent the uniqueness of the website to be clustered, that is, the global framework characteristics of the website.

[0052] Step S230: clustering the websites to be clustered based on the matching vectors to obtain clustering results.

[0053] Specifically, after extracting the DOM tree structure of each website to be clustered in the above step S220, each website to be clustered has two corresponding matching vectors, which can respectively represent the basic framework features and global framework features of the website to be clustered. Then, the websites to be clustered are clustered based on the two matching vectors to obtain clustering results.

[0054] Although the contents presented on website pages are different, websites built by reusing the same set of source code are essentially template websites, and their code structure frameworks are basically the same. Therefore, the existing method of clustering based on website structure mainly calculates the similarity of the website's DOM tree structure through algorithm models such as tree edit distance algorithm and Jaccard coefficient for clustering. However, the calculation process of the above method is relatively complex and time-consuming, and it is difficult to apply to the clustering problem of large-scale website pages. Therefore, this embodiment effectively supplements the existing technology. After obtaining the DOM tree structure corresponding to the website to be clustered through the above steps, various nodes of the DOM tree structure are extracted to obtain matching vectors that can represent the structural characteristics of the website to be clustered. Then, the websites to be clustered are clustered based on the matching vectors. This replaces the method of calculating DOM tree similarity with complex algorithms in the existing technology, and can be applied to large-scale website clustering problems, realizing a low-cost, lightweight, short and fast clustering method, thereby solving the problem of complex similarity calculation when clustering a large number of website pages.

[0055] In some embodiments, the above-mentioned collecting the websites to be clustered and parsing the websites to be clustered to obtain the corresponding DOM tree structure includes the following steps:

[0056] The page data of the website to be clustered is obtained and cleaned to obtain the cleaned source code; the source code is parsed and converted to obtain the corresponding DOM tree structure.

[0057] Specifically, the page data of the website to be clustered, that is, the HTML source code of the website homepage, is obtained from the HTTP and HTTPS services provided by the port corresponding to the website to be clustered. The obtained source code is further cleaned for dirty data, including abnormal line breaks, spaces, comments, irregular tag syntax, etc. in the source code. The standardized HTML source code after cleaning is retained, and the source code is then parsed to obtain the DOM tree structure corresponding to the website to be clustered. The DOM tree structure not only describes the structure of the document, but also defines the behavior of the node object. Using the methods and properties of the object, the nodes and content of the DOM tree can be easily accessed, modified, added and deleted.

[0058] In this embodiment, the source code of the website pages to be clustered is collected and cleaned, and the corresponding DOM tree structure is obtained by parsing. The DOM tree structure can represent the structure of the website to be clustered, so that the structural characteristics of the website to be clustered can be further represented by extracting the nodes of the DOM tree structure.

[0059] In some embodiments, the above-mentioned step of extracting various nodes of the DOM tree structure and obtaining corresponding matching vectors according to the various nodes of the DOM tree structure includes the following steps:

[0060] Extract the root node and secondary nodes of the DOM tree structure and concatenate them to obtain a first node string; perform a hash operation on the first node string to obtain a basic frame matching vector; extract the final node of the DOM tree structure and concatenate them to obtain a second node string; perform a hash operation on the second node string to obtain a global frame matching vector.

[0061] Specifically, the root node, secondary nodes, and final node of the DOM tree structure are extracted, and the root node and the secondary nodes are sequentially concatenated in a specific order to obtain a first node string, and the final node is sequentially concatenated in a specific order to obtain a second node string. The first node string containing the root node and the secondary node can represent the basic framework characteristics of the website structure to be clustered, and the second node string containing the final node can represent the global framework characteristics of the website structure to be clustered.

[0062] Hash operations are performed on the first node string and the second node string respectively to obtain a basic framework matching vector corresponding to the first node string and a global framework matching vector corresponding to the second node string.

[0063] In this embodiment, two types of matching vectors that can represent the structural features of the websites to be clustered are obtained according to various nodes of the DOM tree structure, so that cluster matching based on the structures of the websites to be clustered can be achieved based on the matching vectors.

[0064] In some embodiments, the steps of extracting the root node and the secondary node of the DOM tree structure and concatenating them to obtain a first node string; and extracting the final node of the DOM tree structure and concatenating them to obtain a second node string include the following steps:

[0065] Extract the label names of the root node and the secondary node, and concatenate them in the order of extraction to obtain the first node string; extract the label name of the final node, and concatenate them in the order of extraction to obtain the second node string.

[0066] Specifically, the tag names of the root node and the second-level nodes in the DOM tree structure are sequentially extracted, and then concatenated in the order of extraction to obtain the first node string. The tag names of the final node in the DOM tree structure are then sequentially extracted and concatenated in the order of extraction to obtain the second node string. This is illustrated using part of the source code of the following website as an example:

[0067]

[0068]

[0069] In the above source code, the first node string obtained by concatenating the tag names of the root node and the second-level nodes in the extraction order is head+meta+meta+meta+meta+meta+title+link+link+link+link+body+header+script, and the second node string obtained by concatenating the tag name of the final node in the extraction order is h1.

[0070] In this embodiment, the first node character string and the second node character string obtained by splicing the node label names in the extraction order can represent the basic framework features and the global framework features of the websites to be clustered to a certain extent through the node label names therein, so that in the subsequent steps, the basic framework matching vector and the global framework matching vector corresponding to the websites to be clustered are respectively generated according to the first node character string and the second node character string.

[0071] In some embodiments, clustering the websites to be clustered based on the matching vectors to obtain clustering results includes the following steps:

[0072] Based on the basic frame matching vector, clustering is performed to obtain a preliminary clustering result; in the preliminary clustering result, based on the global frame matching vector, clustering is performed to obtain a final clustering result.

[0073] Specifically, based on the basic framework matching vector, clustering is performed by comparing the basic framework matching vectors for complete consistency. The websites to be clustered can be abstractly classified and clustered according to the website framework, and a preliminary clustering result with a coarser clustering granularity can be obtained. In the preliminary clustering result, the website structures of the clustered websites to be clustered are similar. Further, based on the global framework matching vector, by comparing the global framework matching vectors for complete consistency, classification and clustering are performed according to the source code of the website framework, and a final clustering result with a finer clustering granularity can be obtained. In the final clustering result, the clustered website structures are completely consistent.

[0074] In this embodiment, classification clustering is performed based on the basic framework matching vector and the global framework matching vector in turn. A preliminary clustering result with similar website structures is first clustered, and then a final clustering result with completely consistent website structures is further obtained, thereby achieving a simpler calculation of the clustering results of the websites to be clustered based on the website structure.

[0075] In some embodiments, the method further comprises the following steps:

[0076] In the cyberspace search engine, a basic framework matching vector is calculated based on the preset website to obtain a preliminary clustering result; in the preliminary clustering result, a global framework matching vector is calculated based on the preset website to obtain a final clustering result.

[0077] Specifically, a cyberspace search engine is a comprehensive cyberspace asset security management system that integrates comprehensive asset discovery and analysis, accurate and detailed information provision, critical vulnerability impact assessment, daily security management, and data output. It is capable of mapping the internet IP space. When mapping the internet space, using a cyberspace search engine to discover, identify, crawl, and store assets within the internet space generates a large volume of website page data.

[0078] In practical application scenarios, such as when searching based on a cyberspace search engine, a website corresponding to the query is retrieved from a database as a preset website, and the basic framework matching vector and global framework matching vector corresponding to the website are associated. Based on the basic framework matching vector of the preset website, matching basic framework matching vectors are calculated for the websites to be clustered, resulting in preliminary clustering results with a structure similar to the preset website. Furthermore, within the preliminary clustering results, matching global framework matching vectors are calculated for the websites to be clustered based on the global framework matching vector of the preset website, resulting in a final clustering result consistent with the preset website structure.

[0079] This embodiment provides a method for pre-acquiring a preset website in an actual cyberspace search engine and then performing matching calculations on the websites to be clustered based on the matching vectors corresponding to the preset website, thereby obtaining website information having the same structure as the preset website.

[0080] In some embodiments, the method further comprises the following steps:

[0081] Output the final clustering results and return them to the cyberspace search engine.

[0082] Specifically, in the cyberspace search engine, after the final clustering result matching the preset website vector is calculated, the final clustering result is output, and website information of the same category is returned when the search engine query result is returned.

[0083] In this embodiment, by outputting and returning the final clustering result in the cyberspace search engine, website information of the same category can be returned when returning the search engine query result, providing a convenient method for users to quickly find batches of homologous websites based on known websites through the cyberspace search engine.

[0084] The present embodiment is described and illustrated below through preferred embodiments.

[0085] Figure 3 Flowchart of the website clustering method of this preferred embodiment is as follows: Figure 3 As shown, the method includes the following steps:

[0086] Step S310 , obtaining page data of the website to be clustered and performing cleaning processing to obtain cleaned source code.

[0087] Step S320: Parse the source code and convert it into a DOM tree structure corresponding to the website to be clustered.

[0088] Step S330 , extracting the tag names of the root node and the secondary nodes of the DOM tree structure, and concatenating them in the extraction order to obtain a first node character string.

[0089] Step S340 , extracting the tag names of the final nodes of the DOM tree structure, and concatenating them in the extraction order to obtain a second node character string.

[0090] Step S350 : performing hash operations on the first node string and the second node string respectively, and obtaining a basic framework matching vector and a global framework matching vector respectively.

[0091] Step S360: query and obtain a preset website in a cyberspace search engine; calculate a matching basic framework matching vector based on the preset website, and cluster to obtain a preliminary clustering result.

[0092] Step S370: In the preliminary clustering results, a global frame matching vector is calculated based on the preset website to obtain the final clustering result, which is output and returned to the cyberspace search engine.

[0093] It should be noted that the steps shown in the above process or the flowcharts in the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions. Moreover, although the flowcharts show a logical order, in some cases, the steps shown or described can be executed in a different order than that shown. For example, the order of step S330 and step S340 can be interchanged, and the order in which the various nodes in the DOM tree structure are extracted is not limited.

[0094] This embodiment also provides a website clustering system for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. The terms "module," "unit," "subunit," etc. used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0095] Figure 4 This is a structural diagram of the website clustering system of this embodiment. Figure 4 As shown, the system includes: an acquisition module 410, a splicing module 420 and a clustering module 430;

[0096] The collection module 410 is used to collect the websites to be clustered and parse the websites to be clustered to obtain the corresponding DOM tree structure;

[0097] The splicing module 420 is used to extract various nodes of the DOM tree structure and obtain corresponding matching vectors according to the various nodes of the DOM tree structure;

[0098] The clustering module 430 is configured to cluster the websites to be clustered based on the matching vectors to obtain clustering results.

[0099] Through the system provided in this embodiment, after obtaining the DOM tree structure corresponding to the websites to be clustered, various nodes of the DOM tree structure are extracted to obtain matching vectors that can represent the structural characteristics of the websites to be clustered. Then, the websites to be clustered are clustered based on the matching vectors. This replaces the method of calculating DOM tree similarity using complex algorithms in the existing technology. It can be applied to large-scale website clustering problems, and realizes a low-cost, lightweight, short and fast clustering method, thereby solving the problem of complex similarity calculation when clustering a large number of website pages.

[0100] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0101] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0102] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0103] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0104] In addition, in conjunction with the website clustering method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the website clustering methods in the above embodiments is implemented.

[0105] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0106] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0107] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0108] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A website clustering method, characterized in that: include: Collect the websites to be clustered, and parse the websites to be clustered to obtain the corresponding DOM tree structure; Extracting the tag names of the root node and the secondary nodes of the DOM tree structure, and sequentially splicing them in the extraction order to obtain a first node string; performing a hash operation on the first node string to obtain a basic framework matching vector; Extracting the tag names of the final nodes of the DOM tree structure and sequentially concatenating them in the extraction order to obtain a second node string; performing a hash operation on the second node string to obtain a global frame matching vector; Based on the basic framework matching vector, clustering is performed when the basic framework matching vector is completely consistent, so as to classify and cluster the websites to be clustered according to the website framework, and obtain a preliminary clustering result; In the preliminary clustering results, clustering is performed based on the global frame matching vector to obtain a final clustering result.

2. The website clustering method according to claim 1, characterized in that: The collecting of the websites to be clustered and parsing of the websites to be clustered to obtain the corresponding DOM tree structure includes: Acquire page data of the website to be clustered and perform cleaning processing to obtain cleaned source code; The source code is parsed and converted to obtain a corresponding DOM tree structure.

3. The website clustering method according to claim 1, characterized in that: Also includes: In a cyberspace search engine, the basic framework matching vectors are calculated based on preset websites to obtain the preliminary clustering results; In the preliminary clustering result, the global frame matching vector of the matching is calculated based on the preset website to obtain the final clustering result.

4. The website clustering method according to claim 3, characterized in that: Also includes: The final clustering result is output and returned to the network space search engine.

5. A website clustering system, characterized in that: include: Acquisition module, splicing module and clustering module; The collection module is used to collect the websites to be clustered and parse the websites to be clustered to obtain the corresponding DOM tree structure; The splicing module is used for the tag names of the root node and the secondary node of the DOM tree structure, and splices them in the extraction order to obtain a first node string; performs a hash operation on the first node string to obtain a basic framework matching vector; The splicing module is further used to extract the tag name of the final node of the DOM tree structure, and sequentially splice the tag names in the extraction order to obtain a second node string; perform a hash operation on the second node string to obtain a global frame matching vector; The clustering module is used to perform clustering based on the basic framework matching vectors when the basic framework matching vectors are completely consistent, so as to classify and cluster the websites to be clustered according to the website frameworks to obtain preliminary clustering results; The clustering module is further configured to cluster the preliminary clustering results based on the global frame matching vector to obtain a final clustering result.

6. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the website clustering method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the website clustering method according to any one of claims 1 to 4 is implemented.

Images