Methods, apparatus, devices, and storage media for generating webpage structural fingerprints

Abstract

This invention belongs to the field of computer technology and discloses a method, apparatus, device, and storage medium for generating webpage structural fingerprints. The method includes: parsing the current webpage into a DOM tree and obtaining the attribute features of each node in the DOM tree and the node relationship chain formed by each node; calculating the hash weight of each node based on the node relationship chain; generating the hash value of each node according to its attribute features; and generating the webpage structural fingerprint of the current webpage based on its hash weight and hash value. By parsing the webpage structure and generating fingerprints through the above method, differentiated webpage structures can be achieved, facilitating the aggregation of pages with similar structures and a high risk of cheating, improving the clustering efficiency of webpage structures, shortening the perception time of webpage generation groups, and providing data support for subsequent rapid detection of cheating website groups and prevention of mass piracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating webpage structure fingerprints. Background Technology

[0002] With the development of the internet, a large number of web pages contain duplicate content. To deduplicate and filter web pages for search engines, combat piracy and track content on news and novel websites, and deduplicate and cluster text on social media platforms, it is necessary to retain web page fingerprints.

[0003] Current fingerprinting methods calculate the MD5 or SHA hash value of the text. Even tiny differences in the text can cause an "avalanche effect," resulting in different fingerprints. However, this method is not suitable for differentiating web page structures.

[0004] While the templates displayed to the human eye may appear vastly different, the actual structural differences are minimal. Pages with similar structures and a high risk of fraud require aggregation through structural fingerprinting. Currently, there are numerous algorithms for text fingerprinting and deduplication; however, a systematic algorithmic framework to demonstrate the rationality and usability of webpage structural fingerprinting is still lacking.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for generating webpage structure fingerprints, which aims to differentiate webpage structures and facilitate the aggregation of pages with similar structures that pose a high risk of cheating.

[0007] To achieve the above objectives, the present invention provides a method for generating a webpage structural fingerprint, the method comprising the following steps:

[0008] The current webpage is parsed into a DOM tree, and the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node are obtained.

[0009] Calculate the hash weight of each node based on the node relationship chain;

[0010] Generate the hash value of each node based on its attribute characteristics;

[0011] The webpage structure fingerprint of the current webpage is generated based on the hash weight and hash value of each node.

[0012] Optionally, calculating the hash weight of each node based on the node relationship chain includes:

[0013] Traverse the node relationship chain and assign a corresponding hash weight to each node in the node relationship chain;

[0014] The hash weight of each node is calculated based on the hash weights assigned to its sibling nodes, the hash weights assigned to its parent nodes, and its own hash weight.

[0015] Optionally, traversing the node relationship chain and assigning each node in the node relationship chain a corresponding hash weight includes:

[0016] Traverse the node relationship chain and assign corresponding hash weights to each node in the node relationship chain according to the basic weight through sequential decay, depth decay, and cumulative decay.

[0017] Optionally, generating the hash value of each node based on its attribute characteristics includes:

[0018] The hash value of each node is generated using a text fingerprinting strategy based on the attribute characteristics of each node.

[0019] Optionally, the attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

[0020] Optionally, generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node includes:

[0021] Construct a multi-dimensional bucket structure;

[0022] The hash weights and hash values ​​of each node are incorporated into the multidimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

[0023] Optionally, the step of incorporating the hash weights and hash values ​​of each node into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage includes:

[0024] The hash weights and hash values ​​of each node are incorporated into the multidimensional bucket structure to generate a long fingerprint.

[0025] The long fingerprint is subjected to feature dimensionality reduction to generate the webpage structure fingerprint of the current webpage.

[0026] Optionally, parsing the current webpage into a DOM tree includes:

[0027] The webpage content is retrieved by a spider.

[0028] The webpage content is parsed into a DOM tree.

[0029] Optionally, after generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes:

[0030] Determine the similarity between the webpage structure fingerprint of the current webpage and the structure fingerprint of the pseudo-original webpage;

[0031] If the similarity between the webpage structure fingerprint of the current webpage and the structure fingerprint of the pseudo-original webpage is less than a preset threshold, then the current webpage is marked as a pseudo-original webpage.

[0032] Optionally, after generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes:

[0033] Based on the webpage structure fingerprint of the current webpage, the current webpage is assigned to the nearest cluster to obtain the webpage clustering result.

[0034] Optionally, after generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes:

[0035] Determine the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage;

[0036] If the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is less than a preset similarity, then the current webpage and the target webpage are deduplicated.

[0037] Furthermore, to achieve the above objectives, the present invention also proposes a webpage structure fingerprint generation device, the webpage structure fingerprint generation device comprising:

[0038] The acquisition module is used to parse the current webpage into a DOM tree and obtain the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node;

[0039] The weight calculation module is used to calculate the hash weight of each node based on the node relationship chain;

[0040] The value calculation module is used to generate the hash value of each node based on the attribute characteristics of each node;

[0041] The fingerprint generation module is used to generate the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node.

[0042] Optionally, the weight calculation module is further configured to traverse the node relationship chain, assign a corresponding hash weight to each node in the node relationship chain, and calculate the hash weight of each node based on the hash weights assigned to the sibling nodes of each node, the hash weights assigned to the parent nodes of each node, and the hash weight of each node itself.

[0043] Optionally, the weight calculation module is further configured to traverse the node relationship chain and assign corresponding hash weights to each node in the node relationship chain according to the basic weight through sequential decay, depth decay, and cumulative decay.

[0044] Optionally, the value calculation module is further configured to generate the hash value of each node based on the attribute characteristics of each node using a text fingerprint strategy.

[0045] Optionally, the attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

[0046] Optionally, the fingerprint generation module is further configured to construct a multi-dimensional bucket structure; and to incorporate the hash weights and hash values ​​of each node into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

[0047] Optionally, the fingerprint generation module is further configured to incorporate the hash weights and hash values ​​of each node into the multidimensional bucket structure to generate a long fingerprint; and to perform feature dimensionality reduction on the long fingerprint to generate the webpage structure fingerprint of the current webpage.

[0048] Furthermore, to achieve the above objectives, the present invention also proposes a webpage structure fingerprint generation device, the webpage structure fingerprint generation device comprising: a memory, a processor, and a webpage structure fingerprint generation program stored in the memory and executable on the processor, the webpage structure fingerprint generation program being configured to implement the webpage structure fingerprint generation method as described above.

[0049] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a webpage structure fingerprint generation program, wherein the webpage structure fingerprint generation program, when executed by a processor, implements the webpage structure fingerprint generation method as described above.

[0050] This invention parses the current webpage into a DOM tree, obtains the attribute features of each node in the DOM tree and the node relationship chains formed by each node; calculates the hash weight of each node based on the node relationship chains; generates the hash value of each node based on its attribute features; and generates the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node. By parsing the webpage structure and generating fingerprints in this way, it achieves differentiated differentiation of webpage structures, facilitating the aggregation of pages with similar structures and a high risk of cheating, improving the clustering efficiency of webpage structures, shortening the perception time of webpage generation groups, providing data support for the rapid detection of cheating website groups and preventing the mass generation of pirated content, further enhancing the identification capability of webpage structure features, and reducing the difficulty of detecting malicious behaviors such as cheating, piracy, and spam websites. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the structure of the webpage structure fingerprint generation device in the hardware operating environment involved in the embodiments of the present invention;

[0052] Figure 2 This is a flowchart illustrating the first embodiment of the webpage structure fingerprint generation method of the present invention;

[0053] Figure 3 This is a flowchart illustrating the second embodiment of the webpage structure fingerprint generation method of the present invention;

[0054] Figure 4 This is a flowchart illustrating the third embodiment of the webpage structure fingerprint generation method of the present invention;

[0055] Figure 5 This is a schematic diagram illustrating the specific process of generating a webpage structure fingerprint according to the present invention;

[0056] Figure 6 This is a structural block diagram of the first embodiment of the webpage structure fingerprint generation device of the present invention.

[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0059] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure for generating webpage structure fingerprints in the hardware operating environment involved in the embodiments of the present invention.

[0060] like Figure 1As shown, the device for generating the webpage structure fingerprint may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0061] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the device for generating the webpage structure fingerprint, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0062] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a webpage structure fingerprint generation program.

[0063] exist Figure 1 In the webpage structure fingerprint generation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the webpage structure fingerprint generation device of the present invention can be set in the webpage structure fingerprint generation device, and the webpage structure fingerprint generation device calls the webpage structure fingerprint generation program stored in the memory 1005 through the processor 1001 and executes the webpage structure fingerprint generation method provided in the embodiment of the present invention.

[0064] This invention provides a method for generating a webpage structural fingerprint, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the webpage structure fingerprint generation method of the present invention.

[0065] In this embodiment, the method for generating the webpage structure fingerprint includes the following steps:

[0066] Step S10: Parse the current webpage into a DOM tree, and obtain the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node.

[0067] Understandably, a parser transforms the HTML or XML document corresponding to the current webpage into a DOM tree. The DOM (Document Object Model) is a tree-like data structure composed of multiple nodes. Each node in the DOM tree represents a webpage tag or a text item within a tag, describing the relationships between tags in the webpage document. In the DOM, each node is an object containing corresponding attributes, such as node name, node value, node type, etc. Node relationships are used to represent the relationships between different nodes in the DOM tree, such as parent-child relationships, sibling relationships, etc. In practical implementations, the DOM tree can be traversed to read the attribute characteristics of each node and the node relationship chain.

[0068] Specifically, parsing the current webpage into a DOM tree includes: crawling the webpage content of the current webpage using a spider; and parsing the webpage content into a DOM tree.

[0069] It should be noted that search engine spiders determine the web pages to be crawled through a list of URLs, crawl the content of each web page, and then use a parser to convert the crawled web page content into a DOM tree.

[0070] Step S20: Calculate the hash weight of each node based on the node relationship chain.

[0071] It should be understood that, based on the relationships between nodes and the inheritance of weight within a webpage, weights are assigned to all nodes of the entire webpage to determine the hash weight of each node. Optionally, the weight of the current webpage itself is preset, and the node relationship chain is traversed according to preset traversal rules. Weights are assigned according to the relationships and decay rules between nodes in the node relationship chain. After multiple traversals and assignments starting from different nodes, each node is assigned a corresponding hash weight.

[0072] Step S30: Generate the hash value of each node based on the attribute characteristics of each node.

[0073] It should be noted that in one implementation, the attribute features of the nodes are concatenated and hashed to generate hash values ​​that represent the attributes of each node.

[0074] Optionally, step S30 includes: generating the hash value of each node based on the attribute characteristics of each node using a text fingerprint strategy.

[0075] In a specific implementation, the text fingerprint strategy can optionally be the simhash method: The text is segmented into feature vectors of keywords using word segmentation techniques, and a weight is added to each word feature vector. A binary hash value is calculated for each word feature vector, and then all hash values ​​are accumulated according to their weights. The accumulated result is then processed into binary to generate the hash value corresponding to each node. Alternatively, the I-match algorithm can be used to calculate the hash value.

[0076] Optionally, the attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

[0077] Understandably, the node name attribute is the `name` attribute of the DOM node, the node class name attribute is the `class` attribute of the DOM node, the node unique identifier attribute is the `id` attribute of the DOM node, and the node style attribute is the `style` attribute of the DOM node. A hash value for each node is generated using a text fingerprint method based on the node name, node class attribute, node id attribute, node name attribute, and node style attribute.

[0078] Step S40: Generate the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node.

[0079] It should be noted that, optionally, the hash values ​​of each node are weighted and summed according to their hash weights, and then processed into binary to generate the webpage structure fingerprint of the current webpage; optionally, a multi-dimensional bucket structure is constructed, and the hash weights and hash values ​​of each node are assigned to the bucket structure to generate the webpage structure fingerprint of the current webpage. In specific implementation, webpage structure fingerprints of different webpages are generated according to the method of this embodiment and stored in the website information database to provide data support for subsequent content anti-piracy, webpage deduplication, and webpage clustering.

[0080] Furthermore, after step S40, the method further includes: determining the similarity between the webpage structure fingerprint of the current webpage and the pseudo-original webpage structure fingerprint; if the similarity between the webpage structure fingerprint of the current webpage and the pseudo-original webpage structure fingerprint is less than a preset threshold, then the current webpage is marked as a pseudo-original webpage.

[0081] It should be understood that the pseudo-original webpage structural fingerprint is a webpage structural fingerprint determined in advance through the method of this embodiment analyzing non-original webpages. The preset threshold is a critical value set to distinguish the degree of webpage structural similarity. If the similarity between the current webpage's structural fingerprint and the pseudo-original webpage's structural fingerprint is less than the preset threshold, it indicates that the current webpage's structure is similar to the pseudo-original webpage. If the similarity between the current webpage's structural fingerprint and the pseudo-original webpage's structural fingerprint is greater than or equal to the preset threshold, it indicates that the current webpage's structure differs significantly from the pseudo-original webpage. Optionally, the cosine similarity between the current webpage's structural fingerprint and the pseudo-original webpage's structural fingerprint is calculated. This embodiment does not limit the similarity calculation method.

[0082] Optionally, multiple non-original web pages are analyzed in advance to determine multiple different pseudo-original web page structure fingerprints. The web page structure fingerprint of the current web page is compared with the structure fingerprints of each pseudo-original web page. If the web page structure differences between the current web page and multiple non-original web pages are large, the current web page is marked as an original web page.

[0083] Furthermore, after step S40, the method further includes: classifying the current webpage into the nearest cluster based on the webpage structure fingerprint of the current webpage, thereby obtaining the webpage clustering result.

[0084] In the specific implementation, the distance between the webpage structure fingerprint of the current webpage and the cluster centers of each cluster is calculated. The nearest cluster is determined, and the current webpage is assigned to that cluster. This embodiment does not limit the distance calculation method. For each webpage in the URL list, clustering is performed using the method of this embodiment to obtain the webpage clustering results.

[0085] Furthermore, after step S40, the method further includes: determining the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage; if the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is less than a preset similarity, then performing deduplication processing on the current webpage and the target webpage.

[0086] It should be noted that the target webpage is any webpage whose webpage structure fingerprint has been collected in the website information database. The preset similarity is a threshold value set to distinguish the size of the webpage structure similarity. If the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is less than the preset similarity, it indicates that the webpage structure of the current webpage is similar to that of the target webpage, and deduplication is performed. If the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is greater than or equal to the preset similarity, it indicates that the webpage structure of the current webpage is significantly different from that of the target webpage, and deduplication is not performed. This embodiment does not limit the similarity calculation method.

[0087] This embodiment parses the current webpage into a DOM tree, obtains the attribute features of each node in the DOM tree and the node relationship chain formed by each node; calculates the hash weight of each node based on the node relationship chain; generates the hash value of each node according to its attribute features; and generates the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node. By parsing the webpage structure and generating fingerprints in this way, webpage structures can be differentiated, facilitating the aggregation of pages with similar structures and a high risk of cheating. This improves the clustering efficiency of webpage structures, shortens the perception time of webpage generation groups, and provides data support for the rapid detection of cheating website groups and the prevention of mass piracy. It further enhances the ability to identify webpage structure features and reduces the difficulty of detecting malicious behaviors such as cheating, piracy, and spam websites.

[0088] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the webpage structure fingerprint generation method of the present invention.

[0089] Based on the first embodiment described above, step S20 of the webpage structure fingerprint generation method in this embodiment includes:

[0090] Step S201: Traverse the node relationship chain and assign a corresponding hash weight to each node in the node relationship chain.

[0091] Understandably, the process involves pre-setting the current webpage's own weight and reference coefficient, traversing the node relationship chain, and assigning weight values ​​to nodes based on the relationships between them (at this point, each node's weight value is composed of the current webpage's own weight and reference coefficient). Since the sum of all node weight values ​​equals the current webpage's own weight, the formula can be used to determine each node's own hash weight. For example, if the current webpage's own weight is preset to L and the reference coefficient to M, the first node encountered is assigned a weight value of L*M, its parent node is assigned a weight value of L*M*b, its child node's weight value is L*M*c, and so on, traversing to all nodes. Here, b and c are weight decay coefficients set according to the node relationships.

[0092] Specifically, step S201 includes: traversing the node relationship chain and assigning each node in the node relationship chain a corresponding hash weight based on the basic weight through sequential decay, depth decay, and cumulative decay.

[0093] It should be noted that a basic weight is set for each node in advance, such as 1. The weight is assigned by considering the sequential decay, depth decay, and cumulative decay between nodes. In the specific implementation, the node relationship chain is traversed according to the preset traversal rules. The weight assigned to the Nth node is Qhkw, where Q is the basic weight, h is the sequential decay corresponding to the Nth node, k is the depth decay corresponding to the Nth node, and w is the cumulative decay of the first N-1 nodes. Thus, each node in the node relationship chain is assigned its own hash weight.

[0094] Step S202: Calculate the hash weight of each node based on the hash weights assigned to its sibling nodes, the hash weights assigned to its parent nodes, and its own hash weight.

[0095] It should be noted that the hash weight of the current node is inherited from its sibling nodes, its parent node, and itself. The hash weight assigned to a sibling node refers to the hash weight assigned to the sibling node when executing step S201, and the hash weight assigned to the parent node refers to the hash weight assigned to the parent node when executing step S201. Specifically, the hash weight of any node in the node relationship chain is calculated using the following formula:

[0096] hash = P hash *q1+E hash *q2+S hash *q3;

[0097] Where hash is the hash weight of the current node, P hash The hash weight assigned to the parent node; E hash The hash weight assigned to sibling nodes; S hash q1 represents the hash weight assigned to the current node; q2 and q3 are the weights set.

[0098] This embodiment traverses the node relationship chain, assigning each node a corresponding hash weight. The hash weight of each node is calculated based on the hash weights of its sibling nodes, its parent node, and its own hash weight. By considering the hash weights of sibling nodes, parent nodes, and its own hash weight to determine the hash weight of each node, the security and rationality of the webpage structure fingerprint are further improved. This enables differentiated distinction of webpage structures, facilitating the aggregation of pages with similar structures and a high risk of fraud.

[0099] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the webpage structure fingerprint generation method of the present invention.

[0100] Based on the first embodiment described above, step S40 of the webpage structure fingerprint generation method in this embodiment includes:

[0101] Step S401: Construct a multidimensional bucket structure.

[0102] Understandably, a multi-dimensional bucket structure is constructed to sort and store the hash weights and weight values ​​of each node. In the specific implementation, the dimensions of the constructed bucket structure can be set by the user according to the actual situation, such as 10007 dimensions.

[0103] Step S402: Assign the hash weight and hash value of each node to the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

[0104] In the specific implementation, the hash value and hash weight of all nodes under a webpage are calculated. The hash value is used to determine the bucket it belongs to. The hash value and hash weight are then grouped into a finite number of buckets in the form of an array to generate a webpage structure fingerprint. For example, a 10007-dimensional bucket structure is constructed, and each webpage is ultimately represented by a 10007-dimensional vector to represent its webpage structure fingerprint.

[0105] Furthermore, in order to ensure fingerprint sensitivity, save storage space, and reduce the computational load when using fingerprints in the future, step S402 includes: incorporating the hash weights and hash values ​​of each node into the multidimensional bucket structure to generate a long fingerprint; performing feature dimensionality reduction on the long fingerprint to generate the webpage structure fingerprint of the current webpage.

[0106] It should be noted that a specific feature dimensionality reduction method can be to perform modulo summation on a multi-dimensional long fingerprint to generate a lower-dimensional fingerprint vector to represent the final webpage structure fingerprint. For example, performing modulo summation on a 10007-dimensional long fingerprint vector produces a 200-dimensional fingerprint vector. Feature dimensionality reduction can also be achieved by setting a threshold or a number of thresholds to select features; this embodiment does not impose any limitations on this approach.

[0107] Reference Figure 5 , Figure 5This is a schematic diagram illustrating the specific process of generating a webpage structure fingerprint according to the present invention. For the text document of a webpage, an HTML parser is used to convert it into a DOM tree, producing DOM node attribute features and node relationship chains. The hash weights of different nodes are calculated using sequential decay, depth decay, basic weight, cumulative decay, and node features. The hash weight of the current node is obtained by inheriting the weighted sum of the hash weights of its sibling nodes, parent nodes, and itself, based on different weights. The hash value of the current node is generated using the text fingerprint method based on the node name, node class attribute, node id attribute, node name attribute, and node style attribute; specifically, the Simhash algorithm can be used. A 10007-dimensional bucket structure is constructed. By calculating the hash values ​​and hash weights of all nodes under a webpage, the node hash is used to determine its bucket, and the node hash weights (sibling node hash weights, parent node hash weights, and itself hash weights) are summed and assigned to the bucket. Each HTML document is represented by a 10007-dimensional vector, representing its long fingerprint of webpage structure. The dimensionality of the 10007-dimensional vector is reduced by taking the remainder and accumulating the results, producing the final fingerprint vector used to represent the webpage structure fingerprint.

[0108] This embodiment constructs a multi-dimensional bucket structure; the hash weight and hash value of each node are incorporated into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage. By using the multi-dimensional bucket structure to generate the webpage structure fingerprint, the storage efficiency of the webpage structure fingerprint is improved, further enhancing the operational efficiency when using the fingerprint subsequently. This enables differentiated distinction of webpage structures, facilitating the aggregation of pages with similar structures and a high risk of cheating.

[0109] Furthermore, this embodiment of the invention also proposes a storage medium storing a webpage structure fingerprint generation program, which, when executed by a processor, implements the webpage structure fingerprint generation method described above.

[0110] Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0111] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the webpage structure fingerprint generation device of the present invention.

[0112] like Figure 6 As shown, the webpage structure fingerprint generation device proposed in this embodiment of the invention includes:

[0113] The acquisition module 10 is used to parse the current webpage into a DOM tree and obtain the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node.

[0114] The weight calculation module 20 is used to calculate the hash weight of each node based on the node relationship chain.

[0115] The value calculation module 30 is used to generate the hash value of each node based on the attribute characteristics of each node.

[0116] The fingerprint generation module 40 is used to generate the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node.

[0117] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0118] This embodiment parses the current webpage into a DOM tree, obtains the attribute features of each node in the DOM tree and the node relationship chain formed by each node; calculates the hash weight of each node based on the node relationship chain; generates the hash value of each node according to its attribute features; and generates the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node. By parsing the webpage structure and generating fingerprints in this way, webpage structures can be differentiated, facilitating the aggregation of pages with similar structures and a high risk of cheating. This improves the clustering efficiency of webpage structures, shortens the perception time of webpage generation groups, and provides data support for the rapid detection of cheating website groups and the prevention of mass piracy. It further enhances the ability to identify webpage structure features and reduces the difficulty of detecting malicious behaviors such as cheating, piracy, and spam websites.

[0119] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0120] In addition, for technical details not described in detail in this embodiment, please refer to the web page structure fingerprint generation method provided in any embodiment of the present invention, which will not be repeated here.

[0121] In one embodiment, the weight calculation module 20 is further configured to traverse the node relationship chain, assign a corresponding hash weight to each node in the node relationship chain, and calculate the hash weight of each node based on the hash weights assigned to the sibling nodes of each node, the hash weights assigned to the parent nodes of each node, and the hash weight of each node itself.

[0122] In one embodiment, the weight calculation module 20 is further configured to traverse the node relationship chain and assign corresponding hash weights to each node in the node relationship chain according to the basic weight through sequential decay, depth decay, and cumulative decay.

[0123] In one embodiment, the value calculation module 30 is further configured to generate the hash value of each node based on the attribute characteristics of each node using a text fingerprint strategy.

[0124] In one embodiment, the attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

[0125] In one embodiment, the fingerprint generation module 40 is further configured to construct a multi-dimensional bucket structure; and to incorporate the hash weights and hash values ​​of each node into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

[0126] In one embodiment, the fingerprint generation module 40 is further configured to incorporate the hash weights and hash values ​​of each node into the multidimensional bucket structure to generate a long fingerprint; and to perform feature dimensionality reduction on the long fingerprint to generate the webpage structure fingerprint of the current webpage.

[0127] In one embodiment, the acquisition module 10 is further configured to crawl the webpage content of the current webpage using a spider; and parse the webpage content into a DOM tree.

[0128] In one embodiment, the webpage structure fingerprint generation device further includes a tagging module;

[0129] The marking module is used to determine the similarity between the webpage structure fingerprint of the current webpage and the pseudo-original webpage structure fingerprint; if the similarity between the webpage structure fingerprint of the current webpage and the pseudo-original webpage structure fingerprint is less than a preset threshold, then the current webpage is marked as a pseudo-original webpage.

[0130] In one embodiment, the webpage structure fingerprint generation device further includes a clustering module;

[0131] The clustering module is used to classify the current webpage into the nearest cluster based on the webpage structure fingerprint of the current webpage, thereby obtaining the webpage clustering result.

[0132] In one embodiment, the webpage structure fingerprint generation device further includes a deduplication module;

[0133] The deduplication module is used to determine the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage; if the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is less than a preset similarity, then the current webpage and the target webpage are deduplicated.

[0134] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0135] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0137] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for generating a webpage structural fingerprint, characterized in that, The method for generating the webpage structure fingerprint includes: The current webpage is parsed into a DOM tree, and the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node are obtained. Calculate the hash weight of each node based on the node relationship chain; Generate the hash value of each node based on its attribute characteristics; The webpage structure fingerprint of the current webpage is generated based on the hash weight and hash value of each node. The calculation of the hash weight of each node based on the node relationship chain includes: Traverse the node relationship chain and assign a corresponding hash weight to each node in the node relationship chain; The hash weight of each node is calculated based on the hash weights assigned to its sibling nodes, the hash weights assigned to its parent nodes, and its own hash weight.

2. The method for generating webpage structural fingerprints as described in claim 1, characterized in that, The step of traversing the node relationship chain and assigning a corresponding hash weight to each node in the node relationship chain includes: Traverse the node relationship chain and assign corresponding hash weights to each node in the node relationship chain according to the basic weight through sequential decay, depth decay, and cumulative decay.

3. The method for generating webpage structural fingerprints as described in claim 1, characterized in that, The process of generating hash values ​​for each node based on its attribute characteristics includes: The hash value of each node is generated using a text fingerprinting strategy based on the attribute characteristics of each node.

4. The method for generating webpage structural fingerprints as described in claim 3, characterized in that, The attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

5. The method for generating a webpage structural fingerprint as described in claim 1, characterized in that, The step of generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node includes: Construct a multi-dimensional bucket structure; The hash weights and hash values ​​of each node are incorporated into the multidimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

6. The method for generating a webpage structural fingerprint as described in claim 5, characterized in that, The step of incorporating the hash weights and hash values ​​of each node into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage includes: The hash weights and hash values ​​of each node are incorporated into the multidimensional bucket structure to generate a long fingerprint. The long fingerprint is subjected to feature dimensionality reduction to generate the webpage structure fingerprint of the current webpage.

7. The method for generating a webpage structural fingerprint as described in any one of claims 1-6, characterized in that, The process of parsing the current webpage into a DOM tree includes: The webpage content is retrieved by a spider. The webpage content is parsed into a DOM tree.

8. The method for generating a webpage structural fingerprint as described in any one of claims 1-6, characterized in that, After generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes: Determine the similarity between the webpage structure fingerprint of the current webpage and the structure fingerprint of the pseudo-original webpage; If the similarity between the webpage structure fingerprint of the current webpage and the structure fingerprint of the pseudo-original webpage is less than a preset threshold, then the current webpage is marked as a pseudo-original webpage.

9. The method for generating a webpage structural fingerprint as described in any one of claims 1-6, characterized in that, After generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes: Based on the webpage structure fingerprint of the current webpage, the current webpage is assigned to the nearest cluster to obtain the webpage clustering result.

10. The method for generating a webpage structural fingerprint as described in any one of claims 1-6, characterized in that, After generating the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node, the method further includes: Determine the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage; If the similarity between the webpage structure fingerprint of the current webpage and the webpage structure fingerprint of the target webpage is less than a preset similarity, then the current webpage and the target webpage are deduplicated.

11. A device for generating webpage structural fingerprints, characterized in that, The device for generating the webpage structure fingerprint includes: The acquisition module is used to parse the current webpage into a DOM tree and obtain the attribute characteristics of each node in the DOM tree and the node relationship chain formed by each node; The weight calculation module is used to calculate the hash weight of each node based on the node relationship chain; The value calculation module is used to generate the hash value of each node based on the attribute characteristics of each node; The fingerprint generation module is used to generate the webpage structure fingerprint of the current webpage based on the hash weight and hash value of each node. The weight calculation module is also used to traverse the node relationship chain and assign a corresponding hash weight to each node in the node relationship chain; and calculate the hash weight of each node based on the hash weights assigned to the sibling nodes, the hash weights assigned to the parent nodes, and the hash weights of each node itself.

12. The webpage structure fingerprint generation apparatus as described in claim 11, characterized in that, The weight calculation module is also used to traverse the node relationship chain and assign corresponding hash weights to each node in the node relationship chain according to the basic weight through sequential decay, depth decay and cumulative decay.

13. The webpage structure fingerprint generation apparatus as described in claim 11, characterized in that, The value calculation module is also used to generate the hash value of each node based on the attribute characteristics of each node through a text fingerprint strategy.

14. The webpage structure fingerprint generation apparatus as described in claim 13, characterized in that, The attribute features include node name, node name attribute, node class name attribute, node unique identifier attribute, and node style attribute.

15. The webpage structure fingerprint generation apparatus as described in claim 11, characterized in that, The fingerprint generation module is also used to construct a multi-dimensional bucket structure; the hash weights and hash values ​​of each node are incorporated into the multi-dimensional bucket structure to generate the webpage structure fingerprint of the current webpage.

16. The webpage structure fingerprint generation apparatus as described in claim 15, characterized in that, The fingerprint generation module is further configured to incorporate the hash weights and hash values ​​of each node into the multidimensional bucket structure to generate a long fingerprint; and to perform feature dimensionality reduction on the long fingerprint to generate the webpage structure fingerprint of the current webpage.

17. A device for generating webpage structural fingerprints, characterized in that, The device includes: a memory, a processor, and a webpage structure fingerprint generation program stored in the memory and executable on the processor, the webpage structure fingerprint generation program being configured to implement the webpage structure fingerprint generation method as described in any one of claims 1 to 10.

18. A storage medium, characterized in that, The storage medium stores a webpage structure fingerprint generation program, which, when executed by a processor, implements the webpage structure fingerprint generation method as described in any one of claims 1 to 10.

Images