A web page deduplication method based on structure perception

Abstract

The invention discloses a webpage deduplication method based on structure perception, which belongs to the fields of security, infrastructure and IT support technology, and includes: step 1, webpage collection; step 2, parsing the URL of the collected webpage and generating a URL parameter feature tag; step 3, constructing a URL feature information and difference analysis result mapping table, initializing comparison and filtering the collected webpage, and updating the URL feature information and difference analysis result mapping table; step 4, if the collected webpage is not filtered, extracting a weighted area for the collected webpage according to the URL feature information and the difference analysis result mapping table, and calculating the SimHash value of the collected webpage; step 5, judging whether there is a similar webpage to the collected webpage according to the SimHash value of the collected webpage and the webpage URL, and if there is a similar webpage, performing deduplication processing. The method improves the deduplication efficiency and accuracy, and is suitable for efficient and accurate deduplication processing of large-scale webpage data.

Description

Technical Field

[0001] The present invention belongs to the fields of security, infrastructure and IT support technology, and in particular relates to a web page deduplication method based on structure perception. Background Art

[0002] With the rapid development of the Internet, the amount of online information has grown exponentially. Many websites contain a large number of duplicate or highly similar web pages, such as multiple versions of reports on the same event on news websites, or product list pages on e-commerce websites. This situation results in a large amount of redundant information being generated when collecting data, reducing the efficiency and quality of information acquisition.

[0003] In the prior art, URL-based deduplication methods are mainly implemented through URL mapping relationships, including the use of Hash algorithms, MD5 (message digest algorithm) or bit mapping (such as Bitmap or Bloom filter) and other technologies. These methods can only complete deduplication at the URL level and cannot effectively deal with web pages with the same or highly similar content but different URLs. For example, different versions of the same news event report links may have very similar web page contents, although the URLs are different. In addition, such methods are powerless against dynamically generated web pages, because the URLs of dynamic web pages usually contain parameters or timestamps. Although the URLs are different, the actual content may be the same.

[0004] Another type of prior art is a web page deduplication method based on SimHash, the basic principle of which is to segment the entire text content of a web page, calculate the hash value of each word, and then weighted average these hash values ​​to generate the SimHash fingerprint of the web page. By calculating the Hamming distance between the SimHash fingerprints of different web pages, it is possible to determine whether a web page is duplicated. However, this method has the following defects: it fails to fully consider the differences in the importance of the web page structure and the content of different regions, and regards the entire web page text as equally important, resulting in excessive interference of the template content on the similarity calculation. For example, common parts such as navigation bars and footers may cause unrelated pages to be misjudged as similar. In addition, this method is difficult to effectively identify web pages with the same text content but different template content, such as the reprinting of the same news on different websites. Although the text content is the same, template differences may lead to misjudgment. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a web page deduplication method based on structure perception in view of the deficiencies in the prior art.

[0006] In order to solve the above technical problems, the present invention discloses a web page deduplication method based on structure perception, which includes the following steps.

[0007] Step 1: Web page collection.

[0008] Step 2: parse the URL of the collected web page, extract the parameter part in the URL, and generate a URL parameter feature tag based on the pattern of the parameter name and value.

[0009] Step 3: construct a URL feature information and difference analysis result mapping table, perform initial comparison and filtering on the collected web pages, and update the URL feature information and difference analysis result mapping table.

[0010] Step 4: If the collected web page is not filtered, a weighted area is extracted from the collected web page according to the URL feature information and the difference analysis result mapping table, and the SimHash value of the collected web page is calculated.

[0011] Step 5: Determine whether there are similar web pages to the collected web page based on the SimHash value and web page URL of the collected web page. If there are similar web pages, perform deduplication processing.

[0012] Furthermore, step 2 includes replacing the mark of the URL parameter name, parameter value and path according to the rules, rejoining the replaced parts into a string, calculating the unique request ID of the string, and using the unique request ID as a URL parameter feature tag.

[0013] Furthermore, constructing the URL feature information and difference analysis result mapping table in step 3 includes: using a Map data structure to store and manage the URL feature information and its difference analysis results, and the mapping table uniqueMarkedMap is defined as: Map<String, DiffRes> uniqueMarkedMap

[0014] The key Key of the mapping table is the URL parameter feature tag, that is, the unique request ID uniqueMarkedId, of type String; the value Value is a DiffRes object, which includes an identifier of whether to directly filter directFilter, a cumulative number of comparisons compareCount, a cumulative number of similarities in the comparisons similarCount, a SimHash value of the last comparison lastSimhash, the last web page source code rawHtml, and a web page content change area path set diffPathList, wherein the web page content change area path set diffPathList includes a content area change type, an element XPath, and an element change number.

[0015] Furthermore, in step 3, the collected web pages are initialized for comparison and filtering, and the URL feature information and difference analysis result mapping table are updated, including: searching the mapping table for difference analysis results based on the URL parameter feature tags of the collected web pages, if not, it means that this is the first time that the web page with the URL parameter feature tags has been collected, initializing the difference analysis table of the feature in the mapping table, saving the web page source code to the rawHtml attribute of the last web page source code of DiffRes, using the standard SimHash algorithm to calculate the SimHash value of the current source code, and saving it to the lastSimhash attribute of the last compared SimHash value of DiffRes.

[0016] If it exists and the flag directFilter in the mapping table indicating whether it is directly filtered is true, the collected web page is determined to be duplicated and the collected web page is directly filtered; if the flag directFilter in the mapping table indicating whether it is directly filtered is false, and the cumulative number of comparisons is greater than or equal to the comparison number threshold, the initialization comparison is completed and step 4 is executed.

[0017] If the cumulative comparison times are less than the comparison times threshold, the content change area of ​​the collected web page is identified, a content change area path list is generated, the list is updated to the mapping table, and step 1 is executed.

[0018] Furthermore, in step 3, the content change area of ​​the collected web page is identified, and a content change area path list is generated, and updated to the mapping table, including: comparing the DOM structure, taking the web page corresponding to the previous web page source code rawHtml in the difference analysis result with the same URL parameter feature label in the mapping table as a comparison sample, starting from the root node of the DOM tree, and using a breadth-first traversal method to compare the structure and content of the nodes layer by layer.

[0019] Compare the number of child nodes corresponding to the current node. If the number of child nodes is inconsistent, obtain the text of all child nodes under the current node and compare them. If the summary text is inconsistent, record the content area change type under the current node as a structural change and increase the number of changes.

[0020] If the number of child nodes is the same, compare the text content of each node one by one. If the text content is inconsistent, record the content area change type under the current node as a text change and increase the number of changes. If the text content is consistent, continue to compare all child nodes of the current node.

[0021] After the comparison is completed, if no content change area is found, the collected web page is directly filtered; otherwise, a content change area path list corresponding to the collected web page is generated, updated to the web page content change area path set diffPathList corresponding to DiffRes in the mapping table, the cumulative comparison times compareCount is increased, the last web page source code rawHtml is updated to the source code of the collected web page, the SimHash value of the new source code is recalculated, and compared with the last one. If the Hamming distance is less than 2, the cumulative number of similarities similarCount in the comparison is increased by 1, and the SimHash value of the new web page source code is used to update lastSimhash; when the cumulative number of comparisons compareCount reaches the comparison times threshold, if the cumulative number of similarities similarCount in the comparison is greater than or equal to the similarity times threshold, the flag directFilter of whether to filter directly is updated to true.

[0022] Before SimHash calculation, a pre-filtering mechanism based on URL features is introduced. For web pages with specific URL features, if no content change area is found or the Hamming distance of the SimHash value is less than 2, special marking is performed. The feature links marked as true in the direct filtering flag of the DiffRes object in the mapping table can be directly filtered without downloading and then calculating the similarity, thereby improving efficiency and resource utilization.

[0023] Further, step 4 includes: step 4-1, using a headless browser to download the web page content of the collected web page.

[0024] Step 4-2, parsing the downloaded web page content.

[0025] Step 4-3, calculate the weight of the content change area corresponding to the same URL parameter feature tag in the mapping table.

[0026] Step 4-4, traverse all content change areas, use the XPath locator to find the DOM element corresponding to the web page content parsed in step 4-2, and extract the text content of the DOM element.

[0027] Step 4-5, calculate the SimHash value of the collected web page according to the content change area weight and the corresponding text content.

[0028] Furthermore, step 4-3 includes: recording n content-changed regions, and calculating a weight Wi of the i-th content-changed region.

[0029] Wi = (Ci / M) * (1 + log(Ci))

[0030] Wherein, n>0, 1 ≤ i ≤ n, Ci represents the number of changes of the i-th content change area, and M represents the comparison number threshold.

[0031] The above adaptive weight calculation formula comprehensively considers the importance and marginal effect of the frequency of changes, highlighting important change areas without exaggerating the impact of frequent changes. It provides a reasonable weighting basis for SimHash calculation and improves the system's ability to identify substantive content changes.

[0032] Further, step 4-5 includes: step 4-5-1, creating a 64-bit integer array bits

[64] , initialized to 0; initializing the total weight sumWeight = 0, the maximum weight maxWeight = 0.

[0033] Step 4-5-2, for each content change area, calculate the weighted features and obtain the cumulative combined weight of all content change areas:

[0034] Use the word segmentation tool to segment the text content of the i-th content change area to obtain the word set Ti;

[0035] For each word t in Ti, calculate the 64-bit hash value h, word weight w and combined weight cw of word t, and update the total weight sumWeight, maximum weight maxWeight and integer array bits

[64] ;

[0036] Among them, the combined weight cw = round(w * Wi), the total weight sumWeight is updated to the existing total weight sumWeight plus the combined weight cw, and the maximum weight maxWeight is updated to the maximum value of the existing maximum weight maxWeight and the combined weight cw; traverse the 0 to 63 bits of the hash value h, if the j-th bit of the hash value h is 1, then bits[j] is updated to the existing bits[j] plus the combined weight cw, otherwise bits[j] is updated to the existing bits[j] minus the combined weight cw, j represents the index, 0≤ j ≤ 63.

[0037] Step 4-5-3, generate the SimHash value of the collected web page based on the combined weight accumulated from all content change areas:

[0038] Initialize the empty string simHash, traverse bits 0 to 63 of the integer array bits

[64] , and if bits[k]>0, append 1 to the string simHash, otherwise append 0 to the string simHash. k represents the index, 0 ≤ k ≤ 63.

[0039] This application introduces the concept of document region structure and assigns different weights to different regions. When calculating SimHash, the word weight is combined with the region weight to form a combined weight. This method can more accurately reflect the structural characteristics and content importance of the web page, and improve the expression ability and discrimination of SimHash.

[0040] Furthermore, step 5 includes: creating L index tables, wherein L can equally divide the string SimHash, and the L index tables are used to store L parts of the SimHash values ​​of the historically collected web pages respectively.

[0041] Divide the SimHash values ​​of the collected web pages into L equal parts, search for the same values ​​from the corresponding index tables, obtain the web page IDs corresponding to the same values, and form a list of candidate web pages.

[0042] Get the SimHash value corresponding to each web page in the candidate web page list, calculate the Hamming distance between the SimHash value of the collected web page and the SimHash of the candidate web page, if the Hamming distance is greater than or equal to the preset similarity threshold, then the two web pages are determined to be dissimilar, the L parts of the SimHash value of the collected web page are respectively stored in the corresponding index table, and the SimHash value and URL of the collected web page are recorded; otherwise, the two web pages are determined to be similar, and the SimHash value and URL of the collected web page are not saved.

[0043] Furthermore, the step 2 of performing mark replacement on the URL parameter name, parameter value and path according to the rules includes: for parameter names exceeding a fixed length, using a first mark replacement.

[0044] For parameter names that do not exceed the fixed length, if the parameter name and / or parameter value contains numbers, the numbers are replaced with the second marker.

[0045] Replace Chinese characters in parameter values ​​and / or paths with the third token.

[0046] Replace the path with the Unicode characters in the fourth token.

[0047] Replace the time in the parameter value with the fifth marker.

[0048] By replacing features such as numbers, time, and overly long strings with unified tags, the diversity of parameters can be effectively reduced. Based on the assumption that sub-pages with the same URL features under the same website usually have the same page structure, the main text content that needs to be focused on is identified, and the template content is ignored, thereby improving the efficiency and accuracy of deduplication.

[0049] Beneficial effects: The present invention proposes a web page deduplication method based on structure perception, aiming to improve the accuracy and efficiency of the existing technology when processing similar template websites. By analyzing the DOM tree structure of the web page, identifying different areas and assigning different weights according to their importance, the present invention can accurately extract text content and perform weighted similarity calculations, thereby achieving higher-precision web page deduplication. In addition, combined with URL parameter feature tags and SimHash optimization technology, the efficiency and accuracy of deduplication are further improved, so that the method can efficiently and accurately process large-scale web page data, and ultimately effectively reduce storage costs and server loads. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0051] Figure 1 A schematic flow chart of a method for deduplicating web pages based on structure perception provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0053] The relevant terms are explained below.

[0054] Headless Browser: A browser without a graphical user interface whose behavior can be controlled programmatically, such as loading web pages, executing JavaScript, taking screenshots, etc. It is often used in scenarios such as web page testing and web page content extraction.

[0055] DOM tree (Document Object Model Tree): A data structure that parses HTML or XML documents into a tree structure, used to represent the elements, attributes, and text content in the document. The DOM tree is the basis for web page structure analysis.

[0056] XPath (XML Path Language): A language used to locate nodes in XML documents. XPath is also commonly used to locate nodes in HTML documents.

[0057] SimHash (Similarity Hash): A technique that converts text content into a fixed-length fingerprint, which is used to quickly calculate the similarity between texts. The closer the SimHash values ​​are, the more similar the text content is.

[0058] Hamming distance: An indicator used to compare the degree of difference between two strings of the same length. The Hamming distance refers to the number of different characters in the corresponding positions of the two strings.

[0059] The structure-aware web page deduplication method provided in the present application can be applied to various scenarios where web page deduplication is required, such as: search engines, improving the quality of search results and reducing redundant information; news aggregation, filtering duplicate news, and providing more comprehensive information; network content monitoring, identifying duplicate information, and improving the efficiency of network content analysis; e-commerce platforms, knowledge questions and answers, etc., recommending non-duplicate information to users and increasing user stickiness; market research, user portrait analysis and other data analysis and mining fields can obtain more accurate and reliable analysis results based on the deduplicated web page data.

[0060] The present application embodiment discloses a method for deduplicating web pages based on structure perception, such as Figure 1 As shown, the following steps are included.

[0061] Step 1: Web page collection.

[0062] This step uses efficient web content extraction technology, combined with headless browser rendering technology, to batch collect multiple web pages under the same domain name. To improve collection efficiency, web content extraction supports multi-threading and asynchronous requests, and simulates the real browser environment, such as setting header information such as User-Agent and Referer, and generating browser fingerprints. During the collection process, the URL address of the web page, the HTML source code after the web page is rendered, the HTTP status code, and other information, as well as metadata such as the collection time, are recorded to provide a complete data foundation for subsequent web page deduplication analysis.

[0063] Step 2: Generate URL parameter feature tags

[0064] This step parses each collected URL and extracts the parameter part (such as query parameters and paging parameters in the search page). Then, based on the parameter name and value pattern, a URL parameter feature tag is generated. By replacing features such as numbers, dates, and overly long strings with unified tags, the diversity of parameter names, values, and paths can be effectively reduced, facilitating subsequent feature extraction and comparison. This includes replacing the URL parameter names, parameter values, and paths according to the rules, rejoining the replaced parts into a string, and calculating the unique request ID of the string. The unique request ID is used as the URL parameter feature tag.

[0065] The marking and replacing of the URL parameter name, parameter value and path according to the rules includes: for parameter names exceeding a fixed length, using a first mark {{long}} to replace; in this embodiment, the fixed length can be 32 characters.

[0066] For parameter names that do not exceed the fixed length, if the parameter name and / or parameter value contains numbers, the numbers are replaced with the second token {{number}}.

[0067] Replace Chinese characters in parameter values ​​and / or paths with the third token {{chinese}}.

[0068] Replace the path part containing Unicode characters with the fourth token {{unicode}}.

[0069] Replace the time in the parameter value with the fifth token {{time}}.

[0070] The replacing of the mark of the URL parameter name, parameter value and path according to the rules also includes: replacing the pure uppercase letter part in the parameter value and / or path with the fifth mark {{upper}}, and replacing the pure lowercase letter part with the sixth mark {{lower}}.

[0071] Replace URL-encoded strings in parameter values ​​and / or paths with the seventh token {{urlencode}}.

[0072] Replace the part of the parameter value containing the Boolean value with the eighth token {{bool}}.

[0073] 2.1 Feature Labeling Process

[0074] In order to generalize URLs that meet certain characteristics, this embodiment marks each component of the URL.

[0075] 2.1.1 Label parameter names.

[0076] Mark URL parameter names so that parameters with similar structures can be grouped together, thereby reducing the diversity of parameter names and improving the efficiency and accuracy of URL feature extraction. First, traverse all parameter names and values. For parameter names that are purely alphabetic and not longer than 32 characters, no processing is done; for parameter names that are longer than 32 characters, use the {{long}} tag to replace them, indicating that the parameter name is too long; for other parameter names, use regular expressions to replace the numbers in them with the {{number}} tag.

[0077] Example:

[0078] The parameter name "keyword" is not processed and remains as it is.

[0079] The parameter name "productName123" will be replaced with "productName{{number}}".

[0080] The parameter name "aVeryVeryVeryLongParameterName" will be replaced with "{{long}}".

[0081] 2.1.2 Mark the parameter values.

[0082] According to the type of the parameter value, different marks are used for replacement. Examples are as follows.

[0083] Numbers: Mark the parameter value "123" as "{{number}}".

[0084] Chinese: Mark the parameter value "你好" as "{{chinese}}".

[0085] 2.1.3 Mark the paths.

[0086] Split the path into multiple parts according to " / ", and mark each part: For the pure number part: Replace it with the {{number}} mark.

[0087] The part containing Chinese: Use the {{chinese}} mark.

[0088] The part containing Unicode characters: Use the {{unicode}} mark.

[0089] Finally, splice the marked parts back into a string as the marked path.

[0090] 2.1.4 Calculate the marked unique request ID.

[0091] Replace the parameter names, parameter values, and paths matched in the original URL with the corresponding feature templates, and then perform a hash calculation (such as MD5) on them to obtain a 32-bit unique request ID, and the unique request ID is used as the URL parameter feature label.

[0092] For example, for the URL "https: / / www.example.com / path / api / search?page=2&begin=2024-09-21", its unique request ID can be calculated as:

[0093] MD5("https: / / www.example.com / path / api / search?page={{number}}&begin={{time}}")

[0094] 2.2 Example

[0095] Suppose there are two URLs:

[0096] URL1: http: / / www.test.com / search?name=张三&age=21

[0097] URL2: http: / / www.test.com / search?name=李四&age=24

[0098] After feature marking, their corresponding unique request IDs are both:

[0099] MD5("http: / / www.test.com / search?name={{chinese}}&age={{number}}")

[0100] In this way, URLs with the same features can be grouped together, facilitating subsequent web page structure analysis and duplicate removal operations.

[0101] Step 3: Construct a mapping table of URL feature information and difference analysis results, initialize the comparison and filtering of the collected web pages, and update the mapping table of URL feature information and difference analysis results.

[0102] 3.1 Construct a mapping table uniqueMarkedMap of URL feature information and difference analysis results.

[0103] Use the Map data structure to store and manage URL feature information and its difference analysis results. This data structure is named "mapping table of URL feature information and difference analysis results" (hereinafter referred to as the mapping table), and its specific implementation is as follows.

[0104] 3.1.1. Definition of the mapping table.

[0105] Map<String, DiffRes> uniqueMarkedMap

[0106] 3.1.2. Explanation of key-value pairs in the mapping table.

[0107] Key: The unique request ID uniqueMarkedId after calculation and marking, with the type of String.

[0108] Value: DiffRes object, containing the content change information of the feature web page.

[0109] 3.1.3. DiffRes object structure.

[0110] public class DiffRes {

[0111] private boolean directFilter; / / Identifies whether direct filtering is possible

[0112] private int compareCount; / / The cumulative number of comparisons is limited to 20 times

[0113] private int similarCount; / / The number of similarities in 20 comparisons

[0114] private String lastSimhash; / / The last simhash (calculated using the original method, unweighted)

[0115] private String rawHtml; / / The source code of the previous webpage

[0116] private Set <diffpath>diffPathList; / / Content change area path collection

[0117] }

[0118] 3.1.4. DiffPath object structure.

[0119] public class DiffPath {

[0120] private String pathType; / / Change type

[0121] private String path; / / XPath of the element

[0122] private Integer count; / / Number of element changes

[0123] }

[0124] 3.1.5. Description of pathType values.

[0125] - "TEXT_DIFF": indicates text changes

[0126] - "STRUCTURE_DIFF": indicates changes in page structure

[0127] In step 2, after labeling the currently collected sub-links, the corresponding uniqueMarkedId can be calculated, and it is determined whether there is a corresponding comparison result in the URL feature information and difference analysis result mapping table uniqueMarkedMap.

[0128] 3.2 There are corresponding comparison results.

[0129] First check the directFilter field in the comparison result:

[0130] If directFilter is true, the link is filtered directly without downloading and calculating similarity.

[0131] If directFilter is false, it is checked whether the comparison times reaches the comparison times threshold (for example, the comparison times threshold is 20 in this embodiment).

[0132] If the comparison times threshold is reached, the initialization comparison is completed and the process goes to step 4.

[0133] If the comparison times threshold is not reached, continue to perform 3.3 to perform comparison and update the comparison result.

[0134] 3.3 There is no corresponding comparison result or further comparison is required.

[0135] If there is no comparison result corresponding to the current uniqueMarkedId in uniqueMarkedMap, it means that this is the first time to collect this feature on the web page. In this case, you need to:

[0136] 1) Initialize the difference analysis table of the feature in uniqueMarkedMap.

[0137] 2) Save the web page source code to the rawHtml attribute of DiffRes.

[0138] 3) Use the standard SimHash algorithm to calculate the SimHash value of the current source code and save it to the lastSimhash attribute of DiffRes.

[0139] When a web page with the same characteristics is collected again, the following steps will be performed:

[0140] 1) Compare the newly collected source code with the rawHtml saved in DiffRes (that is, the source code collected last time), and record the difference path in the diffPathList of DiffRes.

[0141] 2) Update rawHtml to the newly collected source code.

[0142] 3) Recalculate the SimHash value of the new source code and compare it with the previous one. If the Hamming distance is less than 2, the similarCount in DiffRes is increased by 1.

[0143] 4) Update lastSimhash with the SimHash value of the new source code.

[0144] The specific comparison steps are as follows.

[0145] a) DOM structure comparison

[0146] The most recently collected web pages in the same feature category (with the same URL parameter feature label) are selected as comparison samples.

[0147] Compare the current web page source code with the selected sample to identify the content change area.

[0148] b) Breadth-first traversal

[0149] Starting from the root node of the DOM tree, a breadth-first traversal is used.

[0150] Compare the structure and content of nodes layer by layer.

[0151] c) Node difference identification

[0152] ci) Structural difference: compare the number of child nodes corresponding to the current node.

[0153] If the number is inconsistent: Get the text of all child nodes under the current node and compare them. If the summary text is inconsistent, record the area as STRUCTURE_DIFF (structural change) and increase the number of changes.

[0154] c.ii) Text differences: If the number of child nodes is the same, compare the ownText text content of each node one by one. If ownText is inconsistent, record the area as TEXT_DIFF (text change) and increase the number of changes. If ownText is consistent, continue to compare all child nodes of the node in depth.

[0155] d) Generate a list of paths for content-changed areas

[0156] After the comparison is completed, if no content change area is found, the collected web pages are directly filtered; otherwise, a content change area path list corresponding to the characteristic web page is generated.

[0157] The path list is bound to the current URL feature tag for subsequent processing of similar category web pages.

[0158] e) Update uniqueMarkedMap

[0159] The generated comparison results (including the path list of content-changed areas and special annotations) are stored in uniqueMarkedMap.

[0160] Use the current uniqueMarkedId as the key to ensure quick retrieval later.

[0161] f) Comparison frequency management

[0162] After each comparison, increase the comparison count. If similar, increase the similarCount count.

[0163] f.1) When the comparison times reaches the comparison times threshold of 20 times: If similarCount is greater than or equal to the similarity times threshold, in this embodiment, the similarity times threshold can be set to 18, then directFilter in DiffRes is set to true, indicating that the web pages corresponding to the feature are highly similar.

[0164] The initialization comparison phase is completed and the execution proceeds to step 4.

[0165] f.2) If the number of comparisons is less than 20: save the current progress and wait for the next acquisition to continue the comparison.

[0166] Step 4: If the collected web pages are not filtered, extract the weighted area of ​​the collected web pages according to the URL feature information and the difference analysis result mapping table, and calculate the SimHash value of the collected web pages.

[0167] This step processes the collected web pages and calculates similarity based on the content change area information corresponding to the characteristic web pages obtained in step 3.

[0168] 4.1 Web page collection.

[0169] Use a headless browser to download the web page content corresponding to the current URL.

[0170] If the download fails or times out, log the error and end the current process.

[0171] 4.2 Content extraction.

[0172] Use HTML parser (Jsoup) to parse the downloaded web page content.

[0173] Extract the text content of a web page, removing HTML tags and scripts.

[0174] 4.3 Calculation of weights of changing regions.

[0175] Let n content-changed regions be included, and calculate the weight Wi of the i-th content-changed region.

[0176] Wi = (Ci / M) * (1 + log(Ci))

[0177] Where: n>0, 1 ≤ i ≤ n, Ci represents the number of changes in the i-th content change area, and M is the total number of comparisons (the comparison number threshold, the maximum is 20).

[0178] Weight formula design idea: This weight design comprehensively considers the importance and marginal effect of the change frequency, which can not only highlight the important change areas, but also not over-exaggerate the impact of frequent changes, providing a reasonable weighted basis for SimHash calculation. This design helps improve the system's ability to identify substantial content changes while maintaining sensitivity to subtle changes.

[0179] Assume that the total number of comparisons M = 20 (maximum value) and calculate the weights for some typical scenarios.

[0180] Use the formula: Wi = (Ci / M) * (1 + log(Ci))

[0181] Example 1: Low frequency variation

[0182] Ci = 1 (changed only once)

[0183] Wi = (1 / 20) * (1 + log(1)) = 0.05 * 1 = 0.05

[0184] Example 2: Medium frequency changes

[0185] Ci = 5

[0186] Wi = (5 / 20) * (1 + log(5)) ≈ 0.25 * 2.61 ≈ 0.65

[0187] Example 3: High-frequency changes

[0188] Ci = 10

[0189] Wi = (10 / 20) * (1 + log(10)) ≈ 0.5 * 3.30 ≈ 1.65

[0190] Example 4: Very high frequency changes

[0191] Ci = 15

[0192] Wi = (15 / 20) * (1 + log(15)) ≈ 0.75 * 3.71 ≈ 2.78

[0193] Example 5: Extreme Case (Maximum Number of Changes)

[0194] Ci = 20

[0195] Wi = (20 / 20) * (1 + log(20)) = 1 * 3.99 ≈ 3.99

[0196] This design ensures that frequently changing regions are indeed given higher weights, reflecting that they may contain more important or newer information.

[0197] But at the same time, the rate of weight growth is controlled to prevent excessive impact in extreme cases.

[0198] Low-frequency changes are weighted less but are still taken into account, preserving the ability to capture occasional but potentially important changes.

[0199] 4.4 Weighted feature extraction.

[0200] a) Traverse all content change areas.

[0201] b) For each region: Use the XPath locator to find the corresponding DOM element. Extract the text content of the element. In step 4.5, the text content will be weighted according to the weight Wi of the region.

[0202] 4.5 Calculate the SimHash value of the collected web page based on the content change area weight and the corresponding text content.

[0203] a) Input preparation: Prepare weighted text regions as input.

[0204] Input: Document D, consisting of n text regions, each region Ri has a corresponding weight Wi

[0205] D = {(R1, W1), (R2, W2), ..., (Rn, Wn)}

[0206] b) Initialization: Initialize necessary variables and data structures. Similar to traditional SimHash: create a 64-bit integer array bits

[64] , initialized to 0; sumWeight = 0 (total weight), maxWeight = 0 (maximum weight).

[0207] c) Text region processing: process each text region and calculate weighted features.

[0208] For i = 1 to n:

[0209] Use the word segmentation tool to segment Ri and obtain the word set Ti.

[0210] For each word t in Ti:

[0211] Calculate the word hash: h = hash(t) / / 64-bit hash value

[0212] Calculate word weight: w = wordWeight(t) / / Weight function based on word features

[0213] Calculate the combined weight: cw = round(w * Wi) / / round to an integer

[0214] Update total weight: sumWeight += cw

[0215] Update the maximum weight: maxWeight = max(maxWeight, cw)

[0216] Update the integer array bits

[64] : For index j = 0 to 63:

[0217] If the jth bit of h is 1:

[0218] bits[j] += cw

[0219] otherwise:

[0220] bits[j] -= cw

[0221] The weight function based on word features can determine the weight of a word according to the following rules.

[0222] Empty or blank word: weight is 0.

[0223] Single-character word: weight is 1. Since the number of bits of a single-character word after hashing is limited (about 40 bits), too high a weight can easily cause the high bits of the SimHash value to be all 0, thereby reducing the discrimination.

[0224] Multi-character words: Differentiate between words that begin with a CJK unified ideograph and other types of words. The judgment is based on whether the Unicode code point of the first character is greater than or equal to 0x3040 (the starting code point of the CJK unified ideograph).

[0225] Words starting with CJK characters and with a length of 2: weight 8.

[0226] Words starting with CJK characters and with a length greater than 2: weight is 16.

[0227] Words starting with non-CJK characters and with a length of 2: weight 2.

[0228] Words starting with non-CJK characters and with a length greater than 2: weight 4.

[0229] This weight distribution strategy aims to give CJK characters higher weights because they usually carry more important information in the text. Words of different lengths are also given different weights, and longer words usually have stronger semantic information and therefore have higher weights.

[0230] d) Generate SimHash value: Generate the final SimHash value of the collected web page based on the accumulated combined weight.

[0231] Similar to traditional SimHash, a brief description is as follows.

[0232] Initialize an empty string simHash

[0233] For index k = 0 to 63:

[0234] If bits[k]>0:

[0235] simHash += "1"

[0236] otherwise:

[0237] simHash += "0"

[0238] Returns the result.

[0239] Step 5: Based on the SimHash value and URL of the collected web page, determine whether there are similar web pages to the collected web page. If there are similar web pages, perform deduplication processing.

[0240] a) Create an index table.

[0241] Create L index tables, where L can equally divide the string SimHash. In this embodiment, L can be 4. The L index tables are used to store L parts of the SimHash values ​​of the historically collected web pages, respectively.

[0242] b) Screen candidate web pages.

[0243] Divide the SimHash values ​​of the collected web pages into L equal parts, search for the same values ​​from the corresponding index tables, obtain the web page IDs corresponding to the same values, and form a list of candidate web pages.

[0244] c) Calculate similarity.

[0245] Iterate over each web page ID in the candidate web page list.

[0246] Get the complete SimHash information of the corresponding web page.

[0247] The degree of difference between the SimHash of the collected web page and the SimHash of the candidate web page is calculated, which is called the Hamming distance.

[0248] d) Identify similar web pages.

[0249] If the Hamming distance is greater than or equal to the preset similarity threshold, the two web pages are determined to be dissimilar, and the L parts of the SimHash value of the collected web page are respectively stored in the corresponding index table, and the SimHash value and URL of the collected web page are recorded; if the Hamming distance is less than the preset similarity threshold, the two web pages are considered similar, and the SimHash value and URL of the collected web page are not saved. In this embodiment, the preset similarity threshold can be set to 4.

[0250] The number of visits to similar web pages can be recorded for use in subsequent index cleaning.

[0251] The present application provides a structure-aware web page deduplication method that can effectively distinguish web page structures, reduce template influence, and improve deduplication accuracy and efficiency. The method can:

[0252] Effectively identify web pages with identical or similar content, even if the URLs are different or the page structures are different.

[0253] Reduce the impact of website template content on similarity calculation and improve the accuracy of deduplication.

[0254] Accurately identify web pages with similar body content but different template content.

[0255] Compared with the traditional SimHash-based deduplication technology, this application has the following advantages:

[0256] 1.Structure-aware precise deduplication.

[0257] Traditional method: Traditional SimHash method usually regards the entire web page content as a whole, without considering the structural characteristics of the web page and the importance differences of different regions.

[0258] This application: Innovatively introduces document region structure and region weighting mechanism. Based on the rule that web pages with the same URL features have similar structures, the frequency of content differences in different regions is calculated by batch analysis and comparison of web pages with the same feature URLs (such as fixed areas such as navigation bars, headers, and footers that change less, while core areas such as titles and texts have obvious differences). Differentiated weights are assigned to each region based on this, so that SimHash can more accurately reflect the structural characteristics and content value of the web page. This method effectively distinguishes between template content and core content, and significantly improves the deduplication effect on web pages with similar structures but different substantive content.

[0259] 2. Adaptive content weight.

[0260] Traditional method: Traditional SimHash usually uses a fixed weight calculation method, which is difficult to adapt to the changing characteristics of content of different web pages.

[0261] This application adopts an innovative adaptive weight calculation formula: Wi = (Ci / M) * (1 + log(Ci)). This formula can dynamically adjust the weight according to the frequency of changes in different areas of the web page, highlighting important change areas while avoiding exaggerating the impact of frequent changes. This method enhances the ability to capture substantive content changes while maintaining sensitivity to subtle changes, making the deduplication results more in line with actual needs.

[0262] 3. Efficient pre-filtration mechanism.

[0263] Traditional method: Traditional SimHash technology usually requires downloading and processing all web page content, and then calculating the SimHash value for comparison, which is inefficient when processing large-scale web page collections.

[0264] This application: introduces a pre-filtering mechanism based on URL features. By analyzing the URL structure and parameters, a preliminary judgment can be made on potential duplicate content before downloading the web page content. For web pages with specific URL features, if no content change area is found or the SimHash value Hamming distance is less than the threshold, it can be filtered directly without downloading and further processing. This pre-filtering mechanism effectively reduces the number of web pages that need to be processed and improves the efficiency and resource utilization of the deduplication process.

[0265] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a web page deduplication method based on structure perception provided by the present invention and some or all steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0266] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention are essentially or the part that contributes to the prior art can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0267] The present invention provides a webpage deduplication method based on structure perception. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.< / diffpath>

Claims

1. A web page deduplication method based on structure perception, characterized in that: The following steps are involved: Step 1, web page collection; Step 2: parse the URL of the collected web page, extract the parameter part in the URL, and generate the URL parameter feature tag according to the pattern of the parameter name and value; Step 3: construct a URL feature information and difference analysis result mapping table, perform initial comparison and filtering on the collected web pages, and update the URL feature information and difference analysis result mapping table; Step 4: If the collected web page is not filtered, extract the weighted area of ​​the collected web page according to the URL feature information and the difference analysis result mapping table, and calculate the SimHash value of the collected web page; Step 5: Based on the SimHash value and URL of the collected webpage, determine whether there are similar webpages on the collected webpage. If there are similar webpages, perform deduplication processing; Step 4 includes: step 4-1, using a headless browser to download web page content of the collected web page; Step 4-2, parsing the downloaded web page content; Step 4-3, calculating the weight of the content change area corresponding to the same URL parameter feature tag in the mapping table; Step 4-4, traverse all content change areas, use the XPath locator to find the DOM element corresponding to the web page content parsed in step 4-2, and extract the text content of the DOM element; Step 4-5, calculate the SimHash value of the collected web page according to the content change area weight and the corresponding text content; Step 4-3 includes: recording n content change regions, and calculating the weight Wi of the i-th content change region: Wi = (Ci / M) * (1 + log(Ci)) Where n > 0, 1 ≤ i ≤ n, Ci represents the number of changes in the i-th content change area, and M represents the comparison number threshold; Step 4-5 includes: Step 4-5-1, creating a 64-bit integer array bits[64], initializing it to 0; initializing the total weight sumWeight = 0, the maximum weight maxWeight = 0; Step 4-5-2, for each content change area, calculate the weighted features and obtain the cumulative combined weight of all content change areas: Use the word segmentation tool to segment the text content of the i-th content change area to obtain the word set Ti; For each word t in Ti, calculate the 64-bit hash value h, word weight w and combined weight cw of word t, and update the total weight sumWeight, maximum weight maxWeight and integer array bits[64]; Among them, combined weight cw = round(w * Wi), the total weight sumWeight is updated to the existing total weight sumWeight plus the combined weight cw, and the maximum weight maxWeight is updated to the maximum value of the existing maximum weight maxWeight and the combined weight cw; traverse the 0 to 63 bits of the hash value h, if the j-th bit of the hash value h is 1, then bits[j] is updated to the existing bits[j] plus the combined weight cw, otherwise bits[j] is updated to the existing bits[j] minus the combined weight cw, j represents the index, 0 ≤ j≤63; Step 4-5-3, generate the SimHash value of the collected web page based on the combined weight accumulated from all content change areas: Initialize an empty string simHash, traverse bits 0 to 63 of the integer array bits[64], and if bits[k] > 0, append 1 to the string simHash, otherwise append 0 to the string simHash, where k represents the index, 0 ≤ k≤63.

2. A webpage deduplication method based on structure perception according to claim 1, characterized in that: Step 2 includes replacing the mark of the URL parameter name, parameter value and path according to the rules, rejoining the replaced parts into a string, calculating the unique request ID of the string, and using the unique request ID as the URL parameter feature tag.

3. A webpage deduplication method based on structure perception according to claim 2, characterized in that: Constructing the URL feature information and difference analysis result mapping table in step 3 includes: using the Map data structure to store and manage the URL feature information and its difference analysis results. The mapping table uniqueMarkedMap is defined as: Map<String, DiffRes> uniqueMarkedMap The key Key of the mapping table is the URL parameter feature tag, that is, the unique request ID uniqueMarkedId, of type String; the value Value is a DiffRes object, which includes an identifier of whether to directly filter directFilter, a cumulative number of comparisons compareCount, a cumulative number of similarities in the comparisons similarCount, a SimHash value of the last comparison lastSimhash, the last web page source code rawHtml, and a web page content change area path set diffPathList, wherein the web page content change area path set diffPathList includes a content area change type, an element XPath, and an element change number.

4. A method for deduplicating web pages based on structure perception according to claim 3, characterized in that: In step 3, the collected web pages are initialized for comparison and filtering, and the URL feature information and difference analysis result mapping table are updated, including: Check whether there is a difference analysis result in the mapping table according to the URL parameter feature tag of the collected web page. If not, it means that this is the first time to collect the URL parameter feature tag of the web page. Initialize the difference analysis table of the URL parameter feature tag in the mapping table, save the web page source code to the rawHtml attribute of the last web page source code of DiffRes, use the standard SimHash algorithm to calculate the SimHash value of the current source code, and save it to the lastSimhash attribute of the last compared SimHash value of DiffRes; If it exists and the flag directFilter in the mapping table indicating whether it is directly filtered is true, the collected web page is determined to be duplicated and the collected web page is directly filtered; if the flag directFilter in the mapping table indicating whether it is directly filtered is false and the cumulative number of comparisons is greater than or equal to the comparison number threshold, the initialization comparison is completed and step 4 is executed; If the cumulative comparison times are less than the comparison times threshold, the content change area of ​​the collected web page is identified, a content change area path list is generated, the list is updated to the mapping table, and step 1 is executed.

5. A method for deduplicating web pages based on structure perception according to claim 4, characterized in that: In step 3, the content change area of ​​the collected web page is identified, a content change area path list is generated, and the updated mapping table includes: Compare the DOM structure, take the web page corresponding to the last web page source code rawHtml in the difference analysis result with the same URL parameter feature label in the mapping table as the comparison sample, start from the root node of the DOM tree, and use the breadth-first traversal method to compare the structure and content of the nodes layer by layer; Compare the number of child nodes corresponding to the current node. If the number of child nodes is inconsistent, obtain the text of all child nodes under the current node and compare them. If the summary text is inconsistent, record the content area change type under the current node as a structural change and increase the number of changes. If the number of child nodes is the same, compare the text content of each node one by one. If the text content is inconsistent, record the content area change type under the current node as text change and increase the number of changes. If the text content is consistent, continue to compare all child nodes of the current node. After the comparison is completed, if no content change area is found, the collected web page is directly filtered; otherwise, a content change area path list corresponding to the collected web page is generated, updated to the web page content change area path set diffPathList corresponding to DiffRes in the mapping table, the cumulative comparison times compareCount is increased, the last web page source code rawHtml is updated to the source code of the collected web page, the SimHash value of the new source code is recalculated, and compared with the last one. If the Hamming distance is less than 2, the cumulative number of similarities similarCount in the comparison is increased by 1, and the SimHash value of the new web page source code is used to update lastSimhash; when the cumulative number of comparisons compareCount reaches the comparison times threshold, if the cumulative number of similarities similarCount in the comparison is greater than or equal to the similarity times threshold, the flag directFilter of whether to filter directly is updated to true.

6. A method for deduplicating web pages based on structure perception according to claim 5, characterized in that: Step 5 includes: creating L index tables, wherein L can equally divide the string SimHash, and the L index tables are used to store L parts of the SimHash values ​​of the historically collected web pages respectively; Divide the SimHash values ​​of the collected web pages into L equal parts, search for the same values ​​from the corresponding index tables, obtain the web page IDs corresponding to the same values, and form a list of candidate web pages; Get the SimHash value corresponding to each web page in the candidate web page list, calculate the Hamming distance between the SimHash value of the collected web page and the SimHash of the candidate web page, if the Hamming distance is greater than or equal to the preset similarity threshold, then the two web pages are determined to be dissimilar, the L parts of the SimHash value of the collected web page are respectively stored in the corresponding index table, and the SimHash value and URL of the collected web page are recorded; otherwise, the two web pages are determined to be similar, and the SimHash value and URL of the collected web page are not saved.

7. A method for deduplicating web pages based on structure perception according to claim 6, characterized in that: The marking and replacement of URL parameter names, parameter values, and paths according to the rules described in step 2 include: For parameter names exceeding a fixed length, the first token is used for replacement; For parameter names that do not exceed the fixed length, if the parameter name and / or parameter value contains numbers, the numbers are replaced with the second marker; Replace Chinese characters in parameter values ​​and / or paths with the third token; Replace the part of the path containing Unicode characters with the fourth token; Replace the time in the parameter value with the fifth marker.

Images