A webpage template structure determination method and device, and a storage medium

By constructing the DOM tree of the webpage and comparing the fingerprint hash value of the main content area, the problem of misjudgment in webpage template structure determination in the existing technology is solved, and efficient and accurate webpage template structure determination is achieved.

CN123115496APending Publication Date: 2026-11-17SHANGHAI MIDU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611237290.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for determining whether web pages belong to the same page template structure have limitations in URL rule generalization, are easily affected by the similarity of the main text, have high costs for visual layout comparison and are prone to misjudgment, and the whole page DOM features or structure hashes can easily include common components such as navigation bars in the judgment, leading to misjudgment.

Method used

The DOM tree is constructed based on the original HTML source code of the web pages to be compared, the main content area is determined, and the fingerprint hash value of the main content area is obtained for comparison. The result of whether the DOM structure of the web pages is the same is output.

Benefits of technology

Without relying on browser visual rendering, it reduces the interference of non-main content areas on the judgment results, improves the efficiency and certainty of webpage template structure judgment, and reduces the impact of changes in main content and page address on the judgment results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115496A_ABST
    Figure CN123115496A_ABST
Patent Text Reader

Abstract

This disclosure provides a method, apparatus, and storage medium for determining a webpage template structure. The method includes: constructing a first Document Object Model (DOM) tree and a second DOM tree based on the original HTML source code of a first webpage and a second webpage to be compared; determining a first main content area of ​​the first webpage based on the first DOM tree, and determining a second main content area of ​​the second webpage based on the second DOM tree; obtaining a first fingerprint hash value of the first main content area and a second fingerprint hash value of the second main content area; comparing the first fingerprint hash value and the second fingerprint hash value to obtain a comparison result; when the first fingerprint hash value and the second fingerprint hash value are the same, outputting a first result, indicating that the DOM architecture of the first webpage and the second webpage are the same; when the first fingerprint hash value and the second fingerprint hash value are different, outputting a second result, indicating that the DOM architecture of the first webpage and the second webpage are different.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of web page technology, and in particular to a method, apparatus and storage medium for determining web page template structure. Background Technology

[0002] In scenarios such as web page scraping, web page archiving, public opinion monitoring, and site information disclosure monitoring, it is often necessary to determine whether two web pages belong to the same page template structure. Existing solutions can make this determination based on Uniform Resource Locator (URL) rules, text similarity, screenshot visual layout, Document Object Model (DOM) template feature vectors, or DOM structure hashes.

[0003] However, URL rules have limited generalization capabilities. The similarity of the main text is easily affected by changes in headings and paragraph content. Comparison of screenshots or visual layouts relies on the browser rendering environment, resulting in high processing costs. Furthermore, whole-page DOM features or whole-page structure hashing can easily include common components such as navigation bars, footers, pop-ups, toolbars, and paginations in the judgment, leading to misjudgments between pages with the same common navigation but different main content templates within the same site.

[0004] Therefore, a webpage template structure determination scheme based on the original (HyperText Markup Language, HTML) source code is needed. This scheme should suppress interference from the main text content, list item count, pagination number, and common navigation components on template determination without relying on browser visual rendering, and output interpretable binary classification results. Summary of the Invention

[0005] This disclosure provides a method, apparatus, and storage medium for determining a webpage template structure.

[0006] Firstly, a method for determining the structure of a webpage template is provided, including: Based on the original HTML source code of the first and second web pages to be compared, construct the first document object model DOM tree and the second DOM tree; The first main content area of ​​the first webpage is determined based on the first DOM tree, and the second main content area of ​​the second webpage is determined based on the second DOM tree. Obtain the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area; The comparison result is obtained by comparing the hash value of the first fingerprint with the hash value of the second fingerprint. When the first fingerprint hash value and the second fingerprint hash value are the same, a first result is output, wherein the first result indicates that the DOM structure of the first webpage and the second webpage is the same; when the first fingerprint hash value and the second fingerprint hash value are different, a second result is output, wherein the second result indicates that the DOM structure of the first webpage and the second webpage is different.

[0007] Secondly, a webpage template structure determining device is provided, comprising: The building module is used to construct a first Document Object Model (DOM) tree and a second DOM tree based on the original Hypertext Markup Language (HTML) source code of the first and second web pages to be compared. The determination module is used to determine the first main content area of ​​the first webpage based on the first DOM tree, and to determine the second main content area of ​​the second webpage based on the second DOM tree; The acquisition module is used to acquire the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area; The comparison module is used to compare the first fingerprint hash value with the second fingerprint hash value to obtain a comparison result; The output module is configured to output a first result when the first fingerprint hash value and the second fingerprint hash value are the same, wherein the first result indicates that the DOM structure of the first webpage and the second webpage are the same; and to output a second result when the first fingerprint hash value and the second fingerprint hash value are different, wherein the second result indicates that the DOM structure of the first webpage and the second webpage are different.

[0008] Thirdly, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the webpage template structure determination method of the first aspect.

[0009] The technical solution provided in this disclosure constructs first and second DOM trees directly based on the original HTML source code of the first and second web pages, and determines the main content areas of the two web pages respectively. This concentrates the object of web page template structure determination on the main content part of the page, reducing the interference of non-main content areas on the determination result. By obtaining the fingerprint hash values ​​of the first and second main content areas respectively, and outputting a first result or a second result based on whether the two fingerprint hash values ​​are the same, this application can transform the judgment of whether the web page DOM architecture is the same into a hash value comparison, thereby improving the efficiency and certainty of web page template structure determination. At the same time, since the determination is based on the fingerprint hash value corresponding to the main content area, rather than the web page text content itself or URL rules, the impact of factors such as changes in specific text content and page address changes on the web page template structure determination result can be reduced. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 This is a flowchart illustrating a method for determining a webpage template structure according to an embodiment of this disclosure; Figure 2A This is a schematic diagram illustrating the comparison of fingerprint hash values ​​between two main content areas of a webpage, provided in an embodiment of this disclosure. Figure 2B This is a schematic diagram illustrating the comparison of fingerprint hash values ​​between the main content areas of two web pages, provided by an embodiment of this disclosure. Figure 3 This is a schematic diagram of an HTML source code cleaning and DOM tree construction process provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram showing the difference in structural labels between sibling subtrees before and after merging, provided in an embodiment of this disclosure. Figure 5 This is a schematic diagram illustrating the division of the main content area, navigation area, and exclusion area, as well as fingerprint generation, provided in an embodiment of this disclosure. Figure 6 This is a schematic diagram illustrating a scoring mechanism for determining the main content area and excluded areas, provided in an embodiment of this disclosure. Figure 7 This is a comparative diagram showing the difference between the normalization processing of the main content area and the pagination structure in the main content area before and after the normalization process provided in this embodiment of the disclosure; Figure 8 This is a schematic diagram of an embodiment of the present disclosure that outputs a determination result based on the fingerprint hash value of the main content area and uses the navigation area for auxiliary diagnosis; Figure 9This is a flowchart illustrating another method for determining a webpage template structure provided in this embodiment of the disclosure; Figure 10 This is a structural block diagram of a webpage template structure determining device provided in an embodiment of this disclosure; Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0014] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0015] In this embodiment, HyperText Markup Language (HTML) can be used to represent webpage source code. Document Object Model (DOM) can be used to represent the node tree structure obtained from parsing HTML source code. Uniform Resource Locator (URL) can be used to represent webpage addresses or resource addresses. JavaScript Object Notation (JSON) can be used to represent the serialized template skeleton data. Secure Hash Algorithm 256-bit (SHA-256) can be used to perform hash calculations on the serialized string of the template skeleton.

[0016] It should be noted that the following characters appearing in this document are HTML tag names: script, style, body, div, section, article, main, aside, header, footer, nav, p, ul, ol, li, table, thead, tbody, tr, td, th, a, span, and h1 to h6. The following characters are HTML attribute names: onclick, onload, href, src, srcset, poster, class, id, role, title, and style. The following characters are data field names, role names, or preset tags: main_hash, nav_hash, content_body, pagination, many, URL_TOKEN, PAGE_ID_TOKEN, TIME_TOKEN, RANDOM_TOKEN, and CLASS_TOKEN. These tag names, attribute names, field names, and preset tags are code literals or data literals, and can be directly represented in English or underscore form in the specification; they do not need to be translated into Chinese.

[0017] In this embodiment, the identical DOM architecture of the web pages can be understood as the two web pages being compared having the same template structure at the main content area level, rather than the two web pages having completely identical body text, title text, link addresses, image resources, script events, visual styles, or business themes. The main content area can be understood as the area in the web page that carries the main template structure of the page. The body text container in a news details page, the list container in a news list page, and the announcement details container or announcement list container in a government information disclosure web page can all serve as specific examples of the main content area.

[0018] The embodiments of this application can be applied to scenarios such as webpage collection quality inspection, webpage archiving, public opinion monitoring, government website information disclosure monitoring, news site template identification, webpage structure change detection, and page template clustering. For example, in the webpage collection quality inspection scenario, the collection system can determine whether a newly added webpage still belongs to an existing collection template. In the government website monitoring scenario, the monitoring system can determine whether the template structure of the information disclosure page has changed. In the news site template identification scenario, the system can determine whether two news webpages use the same detail page template or list page template. In the template clustering scenario, the system can group webpages with the same fingerprint hash value of the main content area into the same template group.

[0019] The execution entity in this application embodiment can be a server, terminal device, web page acquisition device, web page archiving device, data processing device, or a web page structure analysis program deployed in the aforementioned devices. The device may include a processor and a memory. The processor can execute HTML source code acquisition, HTML cleaning, DOM tree construction, merging of duplicate sibling subtrees, candidate region scoring, main content region determination, template normalization, serialization, hash calculation, hash comparison, and result output. The memory can store web page source code, cleaned HTML source code, DOM tree data, merged tree data, candidate region scoring data, main content template skeleton, serialized string, fingerprint hash value, navigation hash value, and structure determination results.

[0020] Reference Figure 1 The diagram illustrates a flowchart of a method for determining a webpage template structure according to an embodiment of this application. The method may include the following steps.

[0021] S101, based on the original HTML source code of the first and second web pages to be compared, construct the first DOM tree and the second DOM tree.

[0022] In this embodiment, the original HTML source code of the first and second web pages to be compared can be obtained first. The first web page can be a snapshot of the web page corresponding to the first URL. The second web page can be a snapshot of the web page corresponding to the second URL. The original HTML source code can be HTML text obtained from the response of the target website server, or it can be a historical page snapshot read from local storage media, a web page snapshot library, a web page archive library, or a network cache. The first and second web pages mentioned above can come from the same site or from different sites.

[0023] After obtaining the original HTML source code of the first and second web pages, they can be parsed separately to construct the corresponding first and second DOM trees. The DOM tree construction process may include: parsing the HTML source code into a node tree structure and recording node information for each node. Node information may include one or more of the following: node identifier, parent node identifier, tag name, attribute set, hierarchy relationship, parent-child relationship, sibling order, text density, link density, number of child nodes, and node path. Attribute sets may include information that can characterize the node category or page area, such as id, class, role, title, name, rel, and type.

[0024] For example, for HTML fragments<divclass="news-container"> <h1> title< / h1> text You can use a div node as the parent node and h1 and p nodes as its child nodes. The div node's category attribute can include class=news-container. The h1 and p nodes can serve as content skeleton tags in subsequent main content area identification. For HTML fragments...<ulclass="news-list"> <ahref=" / news / 1"> News 1 <ahref=" / news / 2"> News 2 You can use the ul node as a list container, the two li nodes as sibling nodes, and record the fact that there is an a node under each li node.

[0025] In some embodiments, both the first and second DOM trees can use the body node as the root node. If the body node does not exist in the HTML source code, or if the body node fails to be parsed, the highest-level container node obtained from the parsing can be used as the root node. If the HTML source code contains missing tags, nesting errors, or non-standard tags, a fault-tolerant HTML parser can be used to complete the closing relationships, ignore isolated text nodes that cannot be attached, or treat unknown tags as ordinary structural nodes. Through the above processing, a traversable, scoreable, and serializable first and second DOM trees can be obtained.

[0026] S102, determine the first main content area of ​​the first webpage based on the first DOM tree, and determine the second main content area of ​​the second webpage based on the second DOM tree.

[0027] In this embodiment, the main content area refers to the area on a webpage that carries the main information or main template structure of the page. The main content area is distinct from common or auxiliary areas such as navigation bars, footers, copyright information, pop-up ads, toolbars, pagination controls, sidebars, related recommendations, and comment sections. The first and second main content areas can be determined using the same set of area identification rules to ensure that the two webpages can be compared under the same standard.

[0028] In some embodiments, each node in the first DOM tree can be traversed to extract its id, class, role, title attributes, child node tag set, text density, link density, and subtree size. If a node region contains positive keywords such as main, content, article, detail, news, post, entry, body, or list, it can be identified as a candidate region for the first main content. If a node region contains exclusion keywords such as footer, copyright, toolbar, advertiser, sidebar, popup, float, breadcrumb, recommend, or related, its candidate score can be reduced, or it can be removed from the candidate pool. For the second webpage, corresponding candidate region identification and exclusion processes can be performed based on the second DOM tree to determine the second main content region.

[0029] In some embodiments, the first main content area can be a body container in a news detail page, such as a div node with the class "news-container", "article-content", or "detail-content". The second main content area can be a list container in a news list page, such as a ul, div, or section node with the class "news-pc", "news-list", or "am-list-news". It is understood that the embodiments of this application compare the template skeleton of the main content area, not the web page business tags themselves. When two web pages are both detail pages, if the main content template skeletons are different, the output may show different DOM structures; in some scenarios, if the main content template skeletons of the detail page and the list page are consistent after normalization, the output may show the same DOM structure.

[0030] S103, obtain the first fingerprint hash value of the first main content area and obtain the second fingerprint hash value of the second main content area.

[0031] In this embodiment, the fingerprint hash value can be understood as a digital digest that characterizes the DOM template structure of the main content area. The fingerprint hash value is not generated directly from the webpage text or the webpage link address, but rather from the template skeleton corresponding to the main content area. The template skeleton can retain structural tags, hierarchical relationships, parent-child relationships, sibling order, node category attributes, and merged subtree information, while minimizing the influence of specific text, specific URLs, script events, and dynamic parameters on the determination result.

[0032] Specifically, the process of obtaining the fingerprint hash value may include: performing template normalization processing on the first main content area to obtain the first main content template skeleton; performing template normalization processing on the second main content area to obtain the second main content template skeleton; converting the first main content template skeleton and the second main content template skeleton into normalized text representations or JSON strings respectively; and performing hash calculation on the normalized text representations or JSON strings to obtain the first fingerprint hash value and the second fingerprint hash value.

[0033] In some embodiments, the SHA-256 algorithm can be used to calculate the hash value, and the first 16 hexadecimal characters, the first 32 hexadecimal characters, or other fixed lengths can be extracted as the fingerprint hash value. SHA-1, SHA-512, MurmurHash, CityHash, or other hash algorithms can also be used. This application does not limit this. As long as the same template skeleton generates the same fingerprint under the same serialization rules, and different template skeletons have distinguishable fingerprints under the same serialization rules, they can be used for determining the webpage template structure in the embodiments of this application.

[0034] S104, compare the hash value of the first fingerprint with the hash value of the second fingerprint to obtain the comparison result.

[0035] In this embodiment, the first fingerprint hash value and the second fingerprint hash value can be compared bit by bit. When the two hash values ​​are completely identical, the comparison result is the same. When the two hash values ​​differ in any bit, the comparison result is different. In some embodiments, the serialized strings of the first and second main content template skeletons can also be retained in addition to the hash values. If the hash values ​​are different, the difference positions in the serialized strings can be further output for manual verification. These difference positions can serve as diagnostic information and do not change the rule of outputting the first or second result based on the fingerprint hash value.

[0036] S105, when the first fingerprint hash value and the second fingerprint hash value are the same, output the first result; when the first fingerprint hash value and the second fingerprint hash value are different, output the second result.

[0037] The first result indicates that the DOM structure of the first webpage and the second webpage is the same. The second result indicates that the DOM structure of the first webpage and the second webpage is different. For example, the first result may include result_code=1 and result_label=same DOM structure. The second result may include result_code=0 and result_label=different DOM structure. The result may also include a first webpage identifier, a second webpage identifier, a first main content area identifier, a second main content area identifier, a first fingerprint hash value, a second fingerprint hash value, and a determination reason. In some embodiments, the first result and the second result may be the Boolean values ​​of a Boolean variable.

[0038] Through steps S101 to S105, this embodiment of the application can directly extract the structural fingerprint of the main content area from the original HTML source code without relying on browser visual rendering, and perform binary classification to determine whether two web pages belong to the same template structure. Since the determination object is concentrated in the main content area, interference from common components such as navigation bars, footers, floating windows, and toolbars on the determination result can be reduced. Because the fingerprint hash value is generated based on the template skeleton, the impact of changes in body text, link addresses, and dynamic page parameters on the determination result can be reduced.

[0039] like Figure 2A As shown, A represents the midu details page on the right, and B represents the midu details page on the right. Based on Figure 2A It can be seen that the fingerprint hash values ​​(main-hash) of the main content area (i.e., main area) of A and B are obviously different, and the corresponding output result is the second result (false).

[0040] like Figure 2B As shown, C represents the right-hand midu details page, and D represents the right-hand midu details page. Based on Figure 2B It can be seen that the fingerprint hash values ​​(main-hash) of the main content area (i.e., main area) of C and D are obviously different, and the corresponding output results can be obtained as the first result (True).

[0041] In some embodiments, the original HTML source code of the first webpage and the second webpage may be cleaned before performing step S101.

[0042] Specifically, the original HTML source code of the first webpage can be cleaned to obtain the first cleaned HTML source code, and a first DOM tree can be constructed based on the first cleaned HTML source code. Similarly, the original HTML source code of the second webpage can be cleaned to obtain the second cleaned HTML source code, and a second DOM tree can be constructed based on the second cleaned HTML source code.

[0043] For example, Figure 3 The image shows a schematic diagram of cleaning the original HTML source code to obtain the cleaned HTML, and then constructing a DOM tree based on the cleaned HTML.

[0044] Cleaning can include removing unstructured tags and deleting or replacing one or more tag attributes. Unstructured tags can include script tags, style tags, inline frame tags, and media tags. Script tags can include `script` and `noscript` tags. Style tags can include `style` tags, which typically contain Cascading Style Sheets (CSS) code. Inline frame tags can include `iframe` tags. Media tags can include `img`, `video`, `audio`, `canvas`, `embed`, and `object` tags. These tags may carry script execution logic, display styles, external resources, or media content; their changes often do not consistently reflect the main content template structure and can therefore be removed before tree construction.

[0045] Tag attributes can include event attributes, display style attributes, and resource address attributes. Event attributes can include HTML attributes that begin with "on," such as onclick, ondblclick, onmouseover, onmouseout, onload, onerror, onchange, oninput, onsubmit, onscroll, and / or onresize.

[0046] In some embodiments, `onclick` represents the code triggered when the mouse clicks an element, often used for click interactions such as buttons and images; `ondblclick` is executed only when the mouse double-clicks an element, commonly used to implement operations such as double-clicking to open details or double-clicking to edit text; `onmouseover` is triggered immediately when the mouse cursor moves into the element's area, generally used to create hover color changes and tooltip effects; this event bubbles and is triggered repeatedly when the mouse moves through the element's child elements; `onmouseout` is triggered when the mouse moves out of the current element's area, often used in conjunction with `onmouseover` to implement style changes when the mouse moves in and style restoration when the mouse leaves; `onload` is a loading event, unrelated to mouse action. When bound to the `body`, it executes after all images and resources on the entire page have been loaded; when bound to an image or script tag, it runs the corresponding code after the resource has been successfully loaded. To ensure page elements are fully loaded before performing operations to avoid errors; `onerror` is triggered when resources such as images and scripts fail to load, and can be used to display a backup image when an image fails to load; `onchange` is mostly used for form controls, and is only triggered when the content of an input box needs to be modified and the cursor leaves the input box and loses focus, while drop-down selection boxes and radio buttons will trigger immediately when switching the selected state, suitable for validation and querying after selection and input are completed; `oninput` also adapts to form input controls, and is triggered in real time whenever the input content is added, deleted, modified, or pasted, without losing focus, suitable for real-time search and real-time text count; `onsubmit` can be bound to the `<form>` tag and is triggered when the form submit button is clicked or the Enter key is pressed, which can perform information validation before the data is actually submitted and redirected, and prevent the form from being submitted if the validation fails; `onscroll` Binds to a browser window or a box with a scrollbar. It will be triggered continuously as long as the scrollbar is scrolled by the mouse wheel or dragged. It is often used to make the navigation bar scroll stick to the top, scroll to load more content, and scroll to display a back to top button. onresize is triggered when the browser window size is manually resized or when the mobile device switches between portrait and landscape modes. It is used to adjust the page layout in real time and make the page adapt to the window size.

[0047] Display style attributes can include a type (style) attribute. Resource address attributes can include attributes such as src, srcset, poster, and data-src. For these attributes, deletion or replacement can be performed based on the attribute type. For example, event attributes, which carry the entry point for interactive scripts, can be deleted. The style attribute mainly affects the visual presentation and can be deleted. Specific values ​​for resource address attributes such as src, srcset, and poster can be deleted, or replaced with resource placeholders.

[0048] During the cleansing process, structural tags such as body, div, section, article, main, aside, header, footer, nav, p, ul, ol, li, table, thead, tbody, tr, td, th, a, span, and h1 to h6 can be retained. These structural tags can express container relationships, list relationships, table relationships, heading relationships, and link relationships between webpage nodes, thus helping to determine the template skeleton later. Attributes such as id, class, role, title, name, rel, and type used to characterize page areas or node categories can also be retained. For the href attribute, its specific link address can be deleted, or it can be replaced with URL_TOKEN. This preserves the structural fact that "this node has a link" while avoiding inconsistencies in the template skeleton caused by specific link addresses on different pages.

[0049] In some embodiments, the cleaning process can first perform tag-level processing and then attribute-level processing. Tag-level processing is used to delete non-structural tags such as script, style, noscript, iframe, img, video, audio, canvas, embed, and object. Attribute-level processing is used to handle event attributes, display style attributes, and resource address attributes. After cleaning, the HTML source code can be parsed with fault tolerance. For example, when HTML tags are missing closing tags, the HTML parser can complete the node relationships; when unknown tags appear, their tag names can be retained and processed as ordinary structural nodes; when isolated text nodes that cannot be attached appear, they can be attached to the nearest available parent node or ignored.

[0050] In some embodiments, cleaning does not mean deleting all non-textual information. Node category attributes that can characterize page regions or template categories can be retained. For example, attributes such as class=news-container, class=article-body, id=main-content, and role=main can be retained. Noisy class names that may change with the build environment can be replaced with CLASS_TOKEN in the subsequent template normalization stage. Through the layered processing of cleaning and subsequent normalization, the indistinguishability of the template skeleton caused by the abrupt deletion of all attributes can be avoided, as can the generation of different fingerprints for the same template by retaining too many page identifier parameters, time parameters, random parameters, or frame generation class names.

[0051] Through the cleaning methods described above, the first and second DOM trees can primarily reflect the static structural relationships of a webpage. Since scripts, styles, media resources, and resource addresses may change depending on the page loading environment, publication time, resource version, or advertising placement, the cleaned DOM tree is more suitable for determining the template structure.

[0052] In some embodiments, before determining the first main content area based on the first DOM tree and the second main content area based on the second DOM tree, the duplicate sibling subtrees in the DOM tree may be merged first.

[0053] First, we can determine whether the specified node exists in both the first and second DOM trees. The specified node can be a `<body>` tag node or a pre-configured page root container node. For example, the specified node can be a node whose class contains characters such as `page`, `container`, `wrap`, `wrapper`, `main`, or `content`. The specified node can also be a node whose id contains characters such as `main`, `content`, `article`, `news`, or `list`. If the pre-defined specified node does not exist in the DOM tree, the root node of the DOM tree can be used as the specified node.

[0054] The first DOM tree can include one or more first subtrees. The second DOM tree can include one or more second subtrees. When a specified node exists in the first DOM tree, multiple sibling first subtrees can be traversed under that specified node. For each sibling first subtree, its structural label can be calculated. The structural label can be determined by the label name of the subtree root node, the sequence of child node labels, the hierarchy relationship, the sibling order, and the node category attribute. For ease of engineering implementation, this structural label can also be referred to as a structural signature.

[0055] In some embodiments, the structural signature can be generated in one or more of the following ways: The label names of all nodes in the subtree can be concatenated into a string in preorder traversal order to serve as the structural signature. For example, when an `li` tag contains an `a` tag and a `span` tag, its structural signature can be `li-a-span`. Node category attributes can be added to the label name to form a combination of label name and class name. For example, when `liclass="item"` contains `aclass="link"`, the structural signature can be `li.item-a.link`. The number of direct child nodes can also be recorded based on the label name and node category attributes, for example, `li.item(2)-a.link(0)-span.text(0)`.

[0056] When multiple sibling first subtrees have the same structural tag, these sibling first subtrees can be merged to obtain a first merged tree. The first merged tree includes the first merged subtree formed by merging the multiple sibling first subtrees. During merging, one of the multiple sibling first subtrees can be retained as a template subtree, and the number of sibling subtrees merged can be recorded. For example, if a ul tag contains 15 li tags with the same structure, these 15 li tags can be merged into a li[merged=15] node. This li[merged=15] node indicates that there is a duplicate list item structure at this position, and merged=15 indicates that there were 15 sibling subtrees before merging.

[0057] When a specified node exists in the second DOM tree, it can be processed in the same way. Multiple sibling second subtrees with the same structural tags can be found under a specified node in the second DOM tree, and these sibling second subtrees can be merged to obtain a second merged tree. The second merged tree includes the second merged subtree formed by merging these multiple sibling second subtrees. For example, if the news list container of a second webpage contains 20 li tags with the same structure, these 20 li tags can be merged into a single li[merged=20] node.

[0058] After obtaining the first and second merged trees, the first main content area of ​​the first webpage can be determined based on the first merged tree, and the second main content area of ​​the second webpage can be determined based on the second merged tree. Merged trees reduce structural differences caused by variations in the number of repeating sibling subtrees. Therefore, the main content area can be identified more stably during subsequent scoring.

[0059] For example, the first page of the same news list template contains 10 news items, and the second page contains 20. If we directly compare the unmerged DOM trees, the different number of `li` nodes under the two list containers can easily lead to differences in the template skeleton. After merging repeated sibling subtrees, the first page can be represented as `li[merged=10]`, and the second page as `li[merged=20]`. In the subsequent normalization stage, they can be further unified as `li[merged=many]`. Therefore, changes in the number of list items will not cause the same template page to be misjudged as having different DOM structures. Figure 4 The image shows a comparison of the structural tags of multiple sibling subtrees with the same structure under the same node on a webpage before and after merging.

[0060] For example, a navigation bar may contain multiple navigation items with the same structure. These items can be merged into a merged subtree within the navigation area. However, the hash value of the navigation area is used for diagnostic purposes and does not change the first or second result output based on the fingerprint hash value of the main content area. This processing is helpful in explaining situations where "the navigation items are the same but the main content is different" when outputting the second result.

[0061] In some embodiments, after obtaining the first merged tree and the second merged tree, the first main content area and the second main content area can be determined through a multi-dimensional attribute information extraction and scoring mechanism.

[0062] For the first webpage, the first attribute information of each content area in the first webpage can be extracted based on the first merged tree. The first attribute information may include one or more of the following: content skeleton tags, first type keywords, first tag keywords, number of first subtrees, number of first merged subtrees, and first specified keywords. Content skeleton tags may include tags such as article, section, div, main, p, ul, ol, li, table, and h1 to h6. First type keywords may include keywords that characterize the page type or area type, such as main, content, detail, list, news, article, product, post, entry, and body. First tag keywords may include keywords contained in the id, class, role, or title of the area node. The number of first subtrees may represent the number of direct child nodes or descendant nodes contained in the content area. The number of first merged subtrees may represent the number of merged subtrees within the area and the merge count value of each merged subtree. The first specified keywords must include at least keywords from excluded areas in the first webpage, such as footer, copyright, pagination, sidebar, advertisement, toolbar, breadcrumb, login, register, comment, related, and recommend.

[0063] The initial base score for each content area in the first webpage can be determined based on the first attribute information. For example, when the area node's id or class contains "main", a first score can be added. When the area node's id or class contains "content", "article", "detail", "news", or "list", a second score can be added. When the area tag is "main" or "article", a third score can be added. When the area contains h1 to h6 heading tags, p paragraph tags, ul or ol list tags, or table tags, the score can be increased based on the number of occurrences. To avoid significant score fluctuations caused by the number of body paragraphs themselves, an upper limit can be set for the score added to paragraph tags. For example, if the number of p tags exceeds 5, it can still be scored as 5.

[0064] When a content area on the first webpage matches the exclusion criteria corresponding to the first specified keyword, its initial base score can be lowered, or the content area can be removed from the first candidate areas. For example, if the area node's ID or class contains footer, foot, copyright, copy, or legal, a higher score can be deducted or it can be removed directly. If the area node's ID or class contains pagination, page-nav, pager, or page, and the area mainly contains links, a score can be deducted or it can be removed. If the area node's ID or class contains sidebar, aside, widget, ad, advertisement, banner, promo, or sponsor, its base score can be lowered. If the area node's ID or class contains toolbar, breadcrumb, sub-nav, top-nav, or menu, its link density and text density can be used to determine whether it is a navigation or auxiliary area.

[0065] Based on the adjusted initial base score, the first main content area can be determined from the first candidate areas. One implementation method is to select the candidate area with the highest adjusted initial base score as the first main content area. Another implementation method is to select the candidate area whose adjusted initial base score exceeds a scoring threshold and whose subtree size meets preset conditions as the first main content area. If multiple areas have the same score, the area with the larger subtree size can be selected, or areas labeled "main" or "article" can be selected according to preset priority. If there is a parent-child containment relationship among the candidate areas, the area with the higher score and that can cover the main structure can be selected to avoid mistakenly selecting a single title node or paragraph node as the main content area.

[0066] For the second webpage, second attribute information for each content area can be extracted based on the second merged tree. This second attribute information may include one or more of the following: content skeleton tags, second type keywords, second tag keywords, number of second subtrees, number of second merged subtrees, and second specified keywords. The second specified keywords must include at least the keywords from excluded areas of the second webpage. The same basic scoring rules, exclusion rules, and candidate area selection rules as those used for the first webpage can be used to determine the second main content area.

[0067] In some embodiments, scoring can be implemented using configurable weights. The more content skeleton tags are matched, the higher the area score. Area scores increase when type keywords match "main," "content," "article," "news," "detail," or "list." Area scores decrease when exclusion keywords match "footer," "copyright," "toolbar," "float," "popup," or "advert." If an area's link density is significantly higher than its text density, and it also matches characters like "nav," "menu," or "header," this area can be identified as a navigation area, not a main content area. If an area has high text density, a high proportion of paragraph tags, and low link density, this area is more likely to be the main content area of ​​a details page. If an area has multiple merged subtrees with the same structure, and its class name contains "list," "news-list," or "item," this area is more likely to be the main content area of ​​a list page.

[0068] The scoring mechanism described above allows for the differentiation between main content areas and common component areas in complex web pages. Since areas such as footers, copyright areas, toolbars, advertisements, pop-ups, and sidebars are penalized or removed, the final determined main content area better reflects the web page template structure. Because the first and second web pages use corresponding scoring rules, their main content areas are comparable.

[0069] For example, such as Figure 5 As shown, the main content area (MAIN) is the basis for the final judgment, the navigation area (NAV) is only used to assist in diagnosis (assist in judgment / determination) and does not participate in voting (i.e., it does not affect the output of the final result), while some common template areas are used as a type of exclusion area and also do not participate in the final judgment (i.e., they do not affect the output of the final result).

[0070] For example, such as Figure 6 As shown, based on the scoring mechanism, the main content area has the highest score, the excluded area does not participate in the fingerprint generation of the main content area, and the navigation area is only used to assist in the diagnosis of the webpage template structure.

[0071] In some embodiments, obtaining the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area may include the following processing: Template normalization can be performed on the first main content area to obtain the first main content template skeleton. Template normalization can also be performed on the second main content area to obtain the second main content template skeleton. Template normalization can be understood as processing information in the main content area that is unrelated to or weakly related to the template structure into a form that does not affect the fingerprint. Template normalization does not delete the main content area itself, but rather, while preserving the template skeleton, deletes or replaces information that changes with page content, publication time, link addresses, script events, or build environment. Figure 5 The example shown contains a normalized template skeleton for the main content area, but the actual implementation is not limited to this.

[0072] For the first main content area, template normalization processing may include one or more of the following: Preserving structural tags, hierarchical relationships, parent-child relationships, sibling order, node category attributes, and information about the first merged subtree used to characterize the template structure within the first main content area. Structural tags may include tags such as div, section, article, main, p, ul, ol, li, table, thead, tbody, tr, td, th, a, span, and h1 to h6. Hierarchical relationships can be represented by node depth. Parent-child relationships can be represented by parent-child pointers between nodes. Sibling order can be represented by the index of a node in its parent node's children array. Node category attributes may include attributes such as id, class, role, and title used to characterize page areas or node categories. Information about the first merged subtree may include merge tags and the number of merges.

[0073] For the first main content area, you can delete the text content and event attributes within it. Text content can include the specific text of the news headline, the specific content of the body paragraphs, link display text, button display text, and / or date display text, etc. Deleting text content means not recording the text string itself from the template skeleton. Deleting event attributes means not recording the script event attribute name and its value from the template skeleton.

[0074] For the first main content area, you can replace the link address, page identifier parameter, time parameter, random parameter, and frame-generated class name in the first main content area with the corresponding preset tags. The link address can include the href attribute value of the tag. Page identifier parameters can include id, articleId, newsId, category, channel, pageId, etc. Time parameters can include time, timestamp, date, publishTime, updateTime, etc. Random parameters can include rand, random, nonce, token, _, etc. Frame-generated class names can include class names corresponding to prefixes or fragments such as ant-, ng-, el-, vue-, css-, hash-, _v-. As an example, you can replace the specific href attribute value with URL_TOKEN, the page identifier parameter with PAGE_ID_TOKEN, the time parameter with TIME_TOKEN, the random parameter with RANDOM_TOKEN, and the frame-generated class name with CLASS_TOKEN.

[0075] The "deletion" and "replacement" mentioned above represent different processing methods. Deletion means that the information is not included in the template skeleton. For example, body text and event attributes can be removed from the template skeleton. Replacement means that the structural fact that this type of information exists is retained, but the specific values ​​are not. For example, the fact that an `` tag has an `href` attribute can be retained, but the specific URL of `href` can be replaced with `URL_TOKEN`. Similarly, the fact that a node has a frame generation class name can be retained, but the random suffix in that class name can be replaced with `CLASS_TOKEN`. By distinguishing between deletion and replacement, the objects, actions, and results of normalization processing can be made clearer.

[0076] For the primary content area, structural differences caused by variations in the number of paragraphs, list items, or pagination links can be standardized. For example, the outer structure tags and node category attributes can be retained for the main content node set, while the number of internal paragraphs and text content can be ignored. Similarly, the outer structure tags and node category attributes can be retained for the pagination node set, while the number of internal page number links and page number text content can be ignored. Furthermore, when the number of the first merged subtree meets a threshold, the number of the first merged subtree can be replaced with a first preset quantity marker.

[0077] For the second main content area, template normalization processing corresponding to that of the first main content area can be applied. The structural tags, hierarchical relationships, parent-child relationships, sibling order, node category attributes, and information about the second merged subtree used to represent the template structure in the second main content area can be retained. Text content and event attributes in the second main content area can be deleted. Link addresses, page identifier parameters, time parameters, random parameters, and frame generation class names in the second main content area can be replaced with corresponding preset tags. Structural differences in the second main content area caused by variations in the number of body paragraphs, list items, or pagination links can be standardized.

[0078] Figure 7 The image shows a comparison of the structure of a webpage before and after normalization, specifically: a comparison of the main content container and / or pagination before and after page collapse in the main content area.

[0079] Serializing and hashing the first main content template skeleton yields the first fingerprint hash value. Serializing and hashing the second main content template skeleton yields the second fingerprint hash value. Serialization can output node information in preorder traversal or convert the template skeleton into a JSON object. To ensure stable serialization results, fields can be sorted in a preset order using a fixed delimiter. Fields can include tag, id, class, role, merged, children, etc. For example, a simple template skeleton can be serialized as `div|class=news-container|h1|content_body`, or as a JSON string containing tag, class, role, and children fields.

[0080] In some embodiments, a SHA-256 hash calculation can be performed on the serialized string, and the first 16 hexadecimal characters can be extracted as the fingerprint hash value. For example, the main content template skeleton of a details page can generate the fingerprint hash value 606e1d75e1389ba0. The main content template skeleton of a list page can generate the fingerprint hash value f4dfe38331b63b29. The above values ​​are exemplary outputs in sample scenarios and are not intended to limit the hash algorithm or the truncation length.

[0081] By using template normalization, serialization, and hash calculation, complex tree structures can be compressed into fixed-length fingerprint hash values, facilitating rapid comparison later. Because normalization filters out text differences, link address differences, event binding differences, dynamic parameter differences, and frame noise class name differences between different instances of the same template, two web pages with different content but the same template structure can still obtain the same or consistent fingerprint hash values.

[0082] In some embodiments, the standardization of structural differences in the main content area caused by variations in the number of paragraphs, list items, or pagination links may include one or more of the following methods.

[0083] For the first set of main content nodes in the first main content area, the outer structure tags and node category attributes of the first set of main content nodes can be retained, while the number of paragraph nodes and text content within the first set of main content nodes can be ignored. The first set of main content nodes can be a collection of nodes composed of multiple main paragraph nodes. Main paragraph nodes can be p tags, or div, span, or other tags that carry main text. In actual web pages, the outer container of the set of main content nodes is usually a div, section, or article node with class names such as content, article-body, detail-content, post-content, entry-content, etc.

[0084] As an optional implementation, the first set of main content nodes can be represented as `content_body` role nodes. This `content_body` role node does not simply transform the nodes into incomprehensible "containers," but rather represents the set of main content nodes within the template skeleton as a placeholder structure with a main content role. This placeholder structure can retain information such as the tag, class, and role of the outer nodes, and uses `role=content_body` to indicate that it is a set of main content nodes. This placeholder structure does not record the number of internal `p` nodes, the text content of each `p` node, or the title text, main text text, or button display text.

[0085] For example, the main content area of ​​the first webpage's details page includes 3 p nodes, while the main content area of ​​the second webpage's details page includes 8 p nodes. If both webpages' outer structure tags and node category attributes correspond to the same main content container, then both sets of main content nodes can be treated as content_body role nodes. In this way, changes in the number of main content paragraphs will not cause a substantial change in the main content template skeleton.

[0086] For the second set of main content nodes in the second main content area, the same processing method as for the first set of main content nodes can be used. The outer structure tags and node category attributes of the second set of main content nodes can be retained, while the number of paragraph nodes and text content within the second set can be ignored. The second set of main content nodes can be represented as a `content_body` role node. In this way, even if the number of paragraphs in the main content of the two detail pages differs, a consistent main content template skeleton can be generated.

[0087] For the first pagination node set within the first main content area, the outer structure tags and node category attributes of the first pagination node set can be retained, while the number of page number links and page number text content within the first pagination node set can be ignored. The first pagination node set can include multiple `` tags, `` tags, or `<button>` tags. These multiple nodes can be used to represent page numbers, previous page, next page, first page, last page, or jump pages. The outer container of the pagination node set can typically be a ``, ``, or `<nav>` node with class names such as `pagination`, `page`, `pager`, or `pages`.

[0088] As an optional implementation, the first set of pagination nodes can be represented as a `pagination` role node. This `pagination` role node can retain information such as the outer tag, class, and role of the pagination container, and is represented by `role=pagination` to indicate that it is a set of pagination nodes. This placeholder structure does not record the number of internal page number links, nor does it record the page number text content. For example, a pagination bar containing 7 links: "Previous Page, 1, 2, 3, 4, 5, Next Page", can be represented in the template skeleton as a `pagination` role node under the outer `div` or `nav`.

[0089] For the second pagination node set in the second main content area, the same processing method as for the first pagination node set can be used. The outer structure tags and node category attributes of the second pagination node set can be retained, while the number of page number links and page number text content within the second pagination node set can be ignored. In this way, even if the first and second pages of the same list template have different numbers of page number links, the pagination structure can remain consistent.

[0090] For the set of nodes in the first main content area whose first merged subtree count meets the threshold, the first merged subtree count can be replaced with a first preset count marker. The count threshold can be 2, 3, 5, or other values. For example, when the number of li nodes with the same structure in a list area is greater than or equal to 2, this merge count can be replaced with many. This many can be used as the first preset count marker. li[merged=5], li[merged=10], and li[merged=20] can all be replaced with li[merged=many]. When the merge count is 1, merged=1 can be retained or the merged field can be omitted to distinguish between the "single structure" and "repeated structure" template design modes.

[0091] For the set of nodes in the second main content area whose number of the second merged subtrees meets the quantity threshold, the number of the second merged subtrees can be replaced with a second preset quantity marker. The first preset quantity marker and the second preset quantity marker can be the same. For example, both can be "many". Through this processing, when the number of list items changes from 6 to 12, the template skeleton can still remain consistent.

[0092] In some embodiments, a set of main text content nodes can be identified by text density, paragraph tag ratio, and link density. Text density can be the ratio of the number of text characters to the number of nodes in the node set. Paragraph tag ratio can be the ratio of the number of p nodes to the total number of child nodes. Link density can be the ratio of the number of a nodes to the total text length or the total number of nodes. When the text density reaches a text density threshold, the paragraph tag ratio reaches a paragraph ratio threshold, and the link density is lower than a link density threshold, the corresponding node set can be identified as a set of main text content nodes.

[0093] In some embodiments, pagination node sets can be identified through page number text patterns, adjacent link sequences, href page number parameter patterns, and pagination category name keywords. Page number text patterns can include consecutive numbers, previous page, next page, first page, last page, etc. Adjacent link sequences can represent multiple tags or <button> tags appearing consecutively under the same parent node. href page number parameter patterns can include parameters such as page, p, pageNo, and current changing with page number. Pagination category name keywords can include pagination, pager, page-nav, pages, etc. Through these identification methods, pagination controls can be distinguished from ordinary news list links.

[0094] By processing the main content node set, processing the pagination node set, and replacing the number of merged subtrees, the impact of changes in main content length, number of pages, and number of list items on the fingerprint hash value can be reduced, respectively. This consistency processing makes it easier for different content instances under the same template to generate the same fingerprint, while preserving information that can characterize the template structure, such as the main content container, list item structure, and pagination shell of the detail page.

[0095] In some embodiments, the method provided in this application may further include: navigation area-assisted diagnosis.

[0096] The first navigation area of ​​a first webpage can be determined based on the first DOM tree or the first merged tree. The second navigation area of ​​a second webpage can be determined based on the second DOM tree or the second merged tree. The navigation area can be a common area within a webpage used for site navigation. For example, the navigation area may include one or more of the following: a top navigation menu, a sidebar, breadcrumb navigation, category entry points, header navigation, or the site shell.

[0097] The first and second navigation areas can be determined by navigation keywords and area scores. Navigation keywords can include nav, menu, navbar, header-nav, header-menu, top-nav, side-nav, left-nav, breadcrumb, and crumb, etc. If an area also contains main content keywords such as content, article, and main, its navigation score can be lowered. If an area has high link density, short text content, and multiple links arranged at the same level, its navigation score can be higher.

[0098] After determining the first and second navigation regions, the first navigation hash value of the first navigation region and the second navigation hash value of the second navigation region can be obtained. The first and second navigation hash values ​​can be obtained using normalization, serialization, and hash calculation methods similar to those used for the main content region. Unlike the main content region, the hash values ​​of the navigation regions are used to output diagnostic information and are not used to change the first or second result.

[0099] When the first fingerprint hash value and the second fingerprint hash value are different, a second result can be output. If the first navigation hash value and the second navigation hash value are the same at this time, auxiliary diagnostic information can also be output. The auxiliary diagnostic information is used to indicate that the navigation areas of the first webpage and the second webpage are the same, but the main content areas are different.

[0100] refer to Figure 8 As shown, pages A and B are compared using their main content areas' main-hash-A and main-hash-B, respectively. The output is based on the main-hash (the fingerprint hash value of the main content area). An output of 1 indicates that pages A and B have the same DOM structure, while an output of 0 indicates that their DOM structures are different. The navigation (nav) area is used for auxiliary diagnosis and does not change the output.

[0101] like Figure 9 As shown, this disclosure provides a method for determining a webpage template structure, including: S201, Obtain the HTML source code of the pages to be processed. The pages to be processed here include the first and second webpages that need to be compared.

[0102] S202, Clean the HTML source code to remove scripts, styles and media noise.

[0103] S203, builds a DOM tree based on the cleaned HTML source code.

[0104] S204, merge the duplicate sibling subtrees in the DOM tree.

[0105] S205, Identify the main content area, navigation area, and exclusion area in the DOM tree.

[0106] S206, normalize the main content template corresponding to the main content area. The normalization process includes text folding, page folding, and merging of duplicate nodes.

[0107] S207, perform stable serialization processing on the normalized main content template, and generate main_hash (the hash fingerprint value of the main content area) based on the serialization result.

[0108] S208 compares the main_hash of the two pages and outputs the comparison result indicating whether the main content of the pages is consistent.

[0109] Among these, navigation fingerprints can serve as auxiliary inputs for region identification and template normalization. Auxiliary diagnostic information can be used to record region identification results, node merging results, and intermediate information during hash comparison.

[0110] For example, the first webpage is midu-content2.html, and the second webpage is midu-news.html. Both webpages belong to the same website and share the same common navigation area. Both the first and second navigation hash values ​​can be a8d110f5f1f07eae. However, the main content area of ​​the first webpage is the details page's body container, news-container, while the main content area of ​​the second webpage is the list page container, news-pc. The fingerprint hash values ​​of their main content areas are different. Therefore, the server outputs a second result along with auxiliary diagnostic information. This auxiliary diagnostic information indicates that the first and second webpages have the same navigation area but different main content areas.

[0111] Using navigation areas for diagnostic purposes can explain structural differences between pages on the same website. This is especially important within the same website, where multiple pages often reuse the same navigation bar. Including the navigation area in the final voting or overall page similarity assessment could lead to detail pages and list pages being mistakenly identified as belonging to the same template. This application uses the navigation area for diagnostic explanation and determines the first or second result using the fingerprint hash value of the main content area. This reduces the interference of common navigation on template structure determination.

[0112] The processing procedure of the embodiments of this application will be described below with reference to several specific samples.

[0113] In one embodiment, the first webpage is content.html, and the second webpage is content2.html. Both webpages are article detail pages from the same website. The title text and body paragraph content of the two webpages can be different, and the number of body paragraphs can also be different. The server obtains the original HTML source code of the two webpages respectively, and cleans the script tags, style tags, iframe tags, img tags, as well as event attributes, display style attributes, and resource address attributes. The server constructs the first DOM tree and the second DOM tree based on the cleaned HTML source code.

[0114] The server identifies the specified node in each of the two DOM trees and merges sibling subtrees with the same structural tags under that node. The server scores candidate regions based on content skeleton tags, type keywords, tag keywords, number of subtrees, number of merged subtrees, and specified keywords. In both web pages, regions containing the class "news-container" contain content skeleton tags such as h1 and p, and do not meet exclusion criteria such as footer, copyright, pop-up, or advertisement. Therefore, the main content region of both web pages can be identified as "news-container".

[0115] The server performs template normalization on the two news-container areas. Normalization preserves the outer structure tags, hierarchical relationships, parent-child relationships, sibling order, and node category attributes of the news-container, while removing the title text, body text, and event attributes. It replaces the link address, page identifier parameters, time parameters, random parameters, and frame generation class name with corresponding preset tags, and represents the body content node set as the content_body role node. After this processing, even if the number of body paragraphs differs between the two web pages, the skeleton of the two main content templates remains consistent.

[0116] The server performs serialization and hash calculation on the two main content template skeletons. The first fingerprint hash value of the first webpage and the second fingerprint hash value of the second webpage are both 606e1d75e1389ba0. Since the first and second fingerprint hash values ​​are the same, the server outputs a first result. The first result indicates that the DOM structure of the two webpages is the same. This embodiment illustrates that the present application does not use the similarity of the main text content as the criterion for judgment, but rather the consistency of the template skeleton of the main content area.

[0117] In one embodiment, the first webpage is content2.html, and the second webpage is news.html. The first webpage is a details page of the same website, and the second webpage is a list page of the same website. The two webpages may have the same or highly consistent common navigation area. The server cleans the original HTML source code of the two webpages respectively, constructs a first DOM tree and a second DOM tree, and merges duplicate sibling subtrees.

[0118] When the server scores candidate areas for the first webpage, areas with the class "news-container" contain h1 tags, publication timestamps, and body content nodes, and have high text density; therefore, they are identified as the primary content area. When the server scores candidate areas for the second webpage, areas with the class "news-pc" contain ul, li, or multiple list item structures, and have a high number of merged subtrees; therefore, they are identified as the secondary content area. The primary content areas of the first and second webpages correspond to different template skeletons.

[0119] If the server performs template normalization on the first main content area, it obtains the first fingerprint hash value: 606e1d75e1389ba0. After performing template normalization on the second main content area, it obtains the second fingerprint hash value: f4dfe38331b63b29. Because the first and second fingerprint hash values ​​are different, the server outputs a second result. This second result indicates that the DOM structures of the first and second web pages are different.

[0120] The server can also generate navigation hash values ​​for the two web pages. If both the first and second navigation hash values ​​are a8d110f5f1f07eae, the server can also output auxiliary diagnostic information. This auxiliary diagnostic information indicates that the navigation areas of the first and second web pages are the same, but their main content areas are different. This embodiment illustrates that, even when the common navigation on the same site is consistent, this application embodiment can still distinguish between detail page templates and list page templates based on the main content area, avoiding misjudgments due to identical common navigation.

[0121] In one embodiment, the first webpage can be a snapshot of the first page of a list template, and the second webpage can be a snapshot of the second page of the same list template. The number of list items and pagination links on the two webpages can differ. For example, the first page displays 10 list items, and the pagination bar displays pages 1 to 5; the second page displays 20 list items, and the pagination bar displays pages 1 to 10. If the complete DOM tree is directly compared, the two webpages will have structural differences due to the different number of li nodes and page number links.

[0122] The server merges duplicate sibling subtrees of two web pages. List items on the first page can be represented as `li[merged=10]`, and list items on the second page can be represented as `li[merged=20]`. During template normalization, when the number of merges meets a threshold, the server can replace both `li[merged=10]` and `li[merged=20]` with `li[merged=many]`. The server can also identify pagination node sets, retain the outer structure tags and node category attributes of the pagination container, and represent the number of internal page number links and page number text content as pagination role nodes.

[0123] After the above consistency processing, the main content template skeletons of the two web pages can be identical. The server performs serialization and hash calculation on the two main content template skeletons to obtain the same fingerprint hash value and outputs the first result. The first result indicates that the DOM structure of the two list pages is the same. This embodiment illustrates that changes in the number of list items and the number of pagination links do not necessarily lead to the same template page being determined to have a different DOM structure.

[0124] In one embodiment, the first webpage can be a details page of a first website, and the second webpage can be a details page of a second website. Both webpages are details pages in terms of business type, but their main content areas may differ in structural tags, hierarchical relationships, parent-child relationships, or node category attributes. For example, the main content area of ​​the first webpage may include a title node, a publication time node, and a set of body nodes, while the main content area of ​​the second webpage may include a title node, a summary node, an attachment table node, and a set of body nodes. If a newly added summary node or attachment table node changes the main content template skeleton, the first fingerprint hash value and the second fingerprint hash value generated by the server will be different. The server can output a second result. This second result indicates that the two webpages have different DOM structures, not that they have different business types.

[0125] In one embodiment, the first webpage and the second webpage can come from different websites. The first webpage can be a details page, and the second webpage can be a list page. If the main content area selected by the server in both webpages falls into the same type of list container skeleton, for example, both fall into a list container of type am-list-news, and the normalized main content template skeleton is the same, then the first fingerprint hash value and the second fingerprint hash value can be the same. The server can output the first result. This embodiment illustrates that the comparison in this application is of the main content area template skeleton, not the business name itself such as "details page" or "list page".

[0126] The following is an example of end-to-end integration. The server receives two web pages to be compared. The first web page is the article details page midu-content2.html from the same website. The second web page is the news list page midu-news.html from the same website. The server first obtains the original HTML source code of the two web pages respectively. The server removes the script, style, noscript, iframe, img, video, and audio tags from the two HTML source codes, and processes attributes such as onclick, onload, style, src, srcset, and poster. The server retains structural tags such as body, div, section, article, main, nav, p, ul, li, table, a, span, and h1 to h6, as well as node category attributes such as id, class, role, and title.

[0127] The server constructs a first DOM tree and a second DOM tree based on the cleaned HTML source code. The server identifies the `body` node in both the first and second DOM trees. Under the `body` node, the server searches for sibling subtrees with the same structural tags. For the first webpage, the server identifies the body area, which includes a title node, a publication time node, and a set of body content nodes. For the second webpage, the server identifies the list area, which includes column titles, multiple list items, and a set of pagination nodes. The server merges the sibling subtrees with the same structural tags and records the number of merged subtrees.

[0128] In some embodiments, the server scores candidate regions of two web pages based on content skeleton tags, type keywords, tag keywords, number of subtrees, number of merged subtrees, and specified keywords. In the first web page, the region with the class "news-container" contains content skeleton tags such as "h1" and "p," and has high text density. This region receives a high score and is identified as the first primary content region. In the second web page, the region with the class "news-pc" contains content skeleton tags such as "ul" and "li," and has multiple list items with the same structure. This region receives a high score and is identified as the second primary content region. The footer regions in both web pages match the excluded keywords and are not identified as primary content regions.

[0129] The server performs template normalization on the first main content area. It retains the structural tags, hierarchical relationships, parent-child relationships, sibling order, and node category attributes. The server removes the main text and event attributes. The server replaces link addresses and dynamic parameters with preset tags. The server performs consistency processing on the main content node set, ignoring the number of paragraphs and text content. The server performs the same type of template normalization on the second main content area. The server performs consistency processing on the number of merged subtrees corresponding to list items and on the pagination node set.

[0130] If the server performs serialization and hash calculation on the first main content template skeleton, obtaining the first fingerprint hash value 606e1d75e1389ba0, and performs serialization and hash calculation on the second main content template skeleton, obtaining the second fingerprint hash value f4dfe38331b63b29, and compares the two fingerprint hash values, and outputs the second result because they are different, the second result indicates that the DOM structures of the two web pages are different.

[0131] The server can also generate navigation hashes for two web pages. If both the first and second navigation hashes are a8d110f5f1f07eae, the server outputs auxiliary diagnostic information.

[0132] This end-to-end implementation demonstrates that the server does not simply compare the entire page's DOM. The server first determines the main content area and then generates a fingerprint hash value based on the template skeleton of that main content area. Having the same public navigation will not lead to two web pages with different main content templates being identified as having the same DOM structure. This process is adaptable to scenarios such as web page scraping quality inspection, government website monitoring, news site template identification, page template clustering, and structural change detection.

[0133] In some embodiments, the server may maintain a node record table, a candidate region table, and a comparison result table to implement the above-described webpage template structure determination method. The node record table may include node identifiers, parent node identifiers, structural tags, node category attributes, hierarchy, sibling order, number of child nodes, text density, link density, number of merged subtrees, and node paths. The candidate region table may include region identifiers, region root node identifiers, content skeleton tag sets, type keywords, tag keywords, basic score, adjusted score, whether exclusion conditions are met, region role, and region shell. The comparison result table may include a first webpage identifier, a second webpage identifier, a first main content region identifier, a second main content region identifier, a first fingerprint hash value, a second fingerprint hash value, a first navigation hash value, a second navigation hash value, result type, and diagnostic description.

[0134] Using the aforementioned data tables, the server can break down DOM tree construction, region determination, template normalization, and result output into reusable data processing steps. For example, the node record table can be written by the construction module. The candidate region table can be written by the determination module after reading the node record table. The comparison result table can be written jointly by the comparison module and the output module. In this way, in a web page scraping system, the server can first generate node record tables and candidate region tables for different web pages in batches, and then generate a comparison result table when comparing any two web pages, thereby reducing the number of times HTML source code is repeatedly parsed.

[0135] In some embodiments, the node path in the node record table can be represented by a path like / body / div[1] / main[1] / article[1] / p[3]. The region shell in the candidate region table can be represented by a combination of tag, class, id, and role, such as div.news-container or section.article-list. The diagnostic description in the comparison result table can record information such as "different hashes for the main content region", "same hashes for the navigation region", and "excluded regions include footer and toolbar". The above data structure is convenient for manual review and also convenient for downstream systems to directly read result_code to complete automatic processing.

[0136] In some embodiments, the server can cache the serialized string before hash calculation. If the same webpage needs to be compared with multiple other webpages, the server can reuse the main content template skeleton, serialized string, and fingerprint hash value of that webpage. This way, in template clustering or batch quality inspection scenarios, the server does not need to repeatedly perform main content region determination and template normalization processing, thereby improving batch comparison efficiency. Cached data can be stored using webpage identifier, webpage snapshot time, source code verification value, cleaning rule version, and normalization rule version as keys. If the cleaning rule or normalization rule changes, the old cache can be invalidated and recalculated based on the rule version.

[0137] In some embodiments, the server may output an exclusion list. The exclusion list may include one or more of the following: footer area, copyright area, floating window area, toolbar area, advertising area, pagination area, and sidebar area. The exclusion list may be output along with the first result or the second result. Human reviewers can use the exclusion list to determine if any common components have been incorrectly included in the main content area. If the candidate area scoring rules need adjustment, the exclusion conditions corresponding to the first or second specified keyword can be modified.

[0138] In some embodiments, the server can perform exception handling when processing the first and second web pages. If the HTML source code of the first or second web page is empty, it can output diagnostic information indicating that a DOM tree cannot be constructed, and output a second result or an indeterminate result. If the HTML source code contains missing tags, nesting errors, or non-standard tags, but can form a traversable DOM tree through fault-tolerant parsing, the server can continue to execute subsequent steps and record the parsing exception in the diagnostic information. If the server can construct the DOM tree but cannot determine the main content area, it can output a second result or output diagnostic information indicating that the main content area is missing. If the server cannot generate a fingerprint hash value, it can output diagnostic information indicating that hash generation failed.

[0139] In some embodiments, when the first or second webpage lacks a navigation area, the server may not generate a corresponding navigation hash value. In this case, the server can still output a first result or a second result based on the first and second fingerprint hash values. That is, the navigation area and navigation hash value are auxiliary information used for diagnostic purposes. Their absence does not affect the determination of the DOM architecture based on the fingerprint hash value of the main content area.

[0140] In some embodiments, the first and second results can be represented by a result code, a result label, and a reason text. For example, the first result may include result_code=1 and result_label=same DOM structure. The second result may include result_code=0 and result_label=different DOM structure. In some embodiments, the system also outputs auxiliary diagnostic information, which may be a diagnosis_label, used to indicate that the navigation areas are the same but the main content areas are different. The reason text may record whether the first fingerprint hash value, the second fingerprint hash value, the first navigation hash value, and the second navigation hash value are the same. Through this output format, the subsequent quality inspection platform can directly read the result_code to complete the automatic judgment, or it can display the reason text for manual review.

[0141] In some embodiments, the web page template structure determination apparatus can be deployed on a server. The construction module can invoke an HTML parser to construct a DOM tree. The determination module can perform subtree merging and region scoring processing. The acquisition module can perform template normalization, serialization, and hash calculation. The comparison module can compare the hash values ​​of a first fingerprint and a second fingerprint. The output module can output a first result, a second result, and auxiliary diagnostic information.

[0142] like Figure 10As shown in the figure, this disclosure provides a webpage template structure determination device, including: a construction module 1011, a determination module 1012, an acquisition module 1013, a comparison module 1014, and an output module 1015. The construction module is used to construct a first DOM tree and a second DOM tree based on the original HTML source code of a first webpage and a second webpage to be compared. The determination module is used to determine a first main content area of ​​the first webpage based on the first DOM tree, and to determine a second main content area of ​​the second webpage based on the second DOM tree. The acquisition module is used to acquire a first fingerprint hash value of the first main content area and a second fingerprint hash value of the second main content area. The comparison module is used to compare the first fingerprint hash value and the second fingerprint hash value. The output module is used to output a first result or a second result based on the comparison result.

[0143] In some embodiments, the construction module, determination module, acquisition module, comparison module, and output module can be implemented by program instructions executed by the same processor. Alternatively, these modules can be implemented by multiple service components. For example, the construction module can be implemented by a web page parsing service. The determination module can be implemented by a region identification service. The acquisition module can be implemented by a fingerprint generation service. The comparison module and output module can be implemented by a template determination service. These modules can exchange data through message queues, remote procedure call interfaces, or shared memory.

[0144] In some embodiments, the data output by the construction module may include node identifiers, parent node identifiers, structural tags, hierarchy, sibling order, node category attributes, and text density. The data output by the determination module may include candidate region identifiers, base scores, adjusted scores, excluded region identifiers, main content region identifiers, and navigation region identifiers. The data output by the acquisition module may include the main content template skeleton, serialized string, fingerprint hash value, and navigation hash value. The data output by the output module may include result codes, result tags, comparison reasons, and diagnostic information.

[0145] In some embodiments, a computer-readable storage medium may store a computer program. When executed by a processor, the computer program can implement the web page template structure determination method described above. The computer-readable storage medium may include a read-only memory, random access memory, flash memory, solid-state drive, magnetic disk, optical disk, or other media capable of storing program instructions.

[0146] In some embodiments, the computer program may include HTML source code acquisition instructions, HTML cleaning instructions, DOM tree construction instructions, sibling subtree merging instructions, attribute information extraction instructions, scoring instructions, exclusion condition processing instructions, main content area determination instructions, template normalization instructions, serialization instructions, hash calculation instructions, comparison instructions, navigation diagnostic instructions, and output instructions. After executing the above instructions, the processor can determine whether the DOM structures of the first and second web pages are the same based on the original HTML source code of the first and second web pages.

[0147] In some embodiments, the computer program may further include cache read instructions and cache write instructions. The cache read instructions can reuse historical calculation results when a valid main content fingerprint already exists on the webpage to be compared. The cache write instructions can write the webpage identifier, rule version, main content template skeleton, and fingerprint hash value to the cache after completing the main content region determination, template normalization, and hash calculation. In this way, the program in the storage medium can be adapted to scenarios such as batch webpage template clustering, periodic webpage monitoring, and large-scale webpage collection and quality inspection.

[0148] In some embodiments, the first result, the second result, and the auxiliary diagnostic information can also be reused by downstream business systems. The webpage collection quality inspection system can trigger a collection rule review task after receiving the second result. The webpage archiving system can group two webpages into the same template group after receiving the first result. The site monitoring system can indicate that the public navigation has not changed but the main content template has changed after receiving auxiliary diagnostic information. The template clustering system can group webpages with the same fingerprint hash value into the same cluster. The government website monitoring system can indicate that the page publishing system or collection rules may be adjusted when the main content template changes. It is understood that the webpage template structure determination method in this application embodiment does not depend on any of the specific downstream businesses mentioned above.

[0149] In some embodiments, the server can configure scoring thresholds, quantity thresholds, and normalization rules. Scoring thresholds can be used to determine whether a candidate region can be used as the main content region. Quantity thresholds can be used to determine whether the number of merged subtrees needs to be replaced with a preset quantity marker. Normalization rules can be used to configure which class names belong to the frame-generated class names, and which parameters belong to page identifier parameters, time parameters, or random parameters. Different configurations can be adopted for different business scenarios. For example, news sites can increase the text density weight of the main content node set. Government information disclosure sites can increase the weight of content skeleton tags such as table, ul, and li. E-commerce or product display sites can increase the weight of list item structures, price field shells, and product card containers.

[0150] In some embodiments, the different processing methods described above can be used in combination. For example, the server can simultaneously perform HTML source code cleaning, merging of duplicate sibling subtrees, scoring of the main content area, consistency of the main content node set, consistency of the pagination node set, replacement of the number of merged subtrees, navigation area diagnosis, and result caching. Some extended processing can also be disabled depending on the business scenario. For example, for detail pages without a pagination structure, the pagination node set consistency processing can be omitted. For batch clustering tasks that do not require diagnostic output, navigation hash values ​​can be omitted. The above combinations are only optional implementation methods. This application embodiment does not require all of the above extended processing to be enabled in every comparison.

[0151] It is understood that in the above embodiments, "first" and "second" are only used to distinguish different objects and are not used to limit the order or importance. "Multiple" can be understood as at least two. The execution order of the steps in the above embodiments can be adjusted when there is no explicit dependency. For example, HTML cleaning is usually performed before DOM tree construction, and the determination of the navigation area can be performed after the determination of the main content area, or in parallel with the determination of the main content area. As another example, the DOM tree construction, subtree merging, area scoring, template normalization, and hash calculation of the first and second web pages can be performed in parallel.

[0152] To achieve the above objectives, embodiments of this application also provide an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 111 and a memory 113 connected to the processor 111 via a communication bus 112; wherein, the memory 113 is used for an executable program; the processor 111 can implement the web page template structure determination method provided in any of the foregoing embodiments by executing the executable program.

[0153] Another embodiment of this application provides a computer-readable storage medium storing an executable program. When the executable program is executed by a processor 111, it can implement the aforementioned webpage template structure determination method, for example, as... Figures 1-3 One or more of the methods shown.

[0154] In some embodiments, the computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] Those skilled in the art, upon considering the specification and practicing the technical solutions disclosed in this application, will readily conceive of other embodiments of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the art not specifically described in this application. The specification and embodiments are considered exemplary only, and the scope of protection of this application is determined by the claims.

Claims

1. A method for determining the structure of a webpage template, characterized in that, include: Based on the original Hypertext Markup Language (HTML) source code of the first and second web pages to be compared, construct the first Document Object Model (DOM) tree and the second DOM tree; The first main content area of ​​the first webpage is determined based on the first DOM tree, and the second main content area of ​​the second webpage is determined based on the second DOM tree. Obtain the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area; The comparison result is obtained by comparing the hash value of the first fingerprint with the hash value of the second fingerprint. When the first fingerprint hash value and the second fingerprint hash value are the same, a first result is output, wherein the first result indicates that the DOM structure of the first webpage and the second webpage is the same; when the first fingerprint hash value and the second fingerprint hash value are different, a second result is output, wherein the second result indicates that the DOM structure of the first webpage and the second webpage is different.

2. The method according to claim 1, characterized in that, The step of determining the first main content area of ​​the first webpage based on the first DOM tree and determining the second main content area of ​​the second webpage based on the second DOM tree includes: Determine whether the specified node exists in the first DOM tree and the second DOM tree; the first DOM tree includes one or more first subtrees; the second DOM tree includes one or more second subtrees. When the specified node exists in the first DOM tree, multiple sibling first subtrees with the same structural tag under the specified node in the first DOM tree are merged to obtain a first merged tree, wherein the first merged tree includes the first merged subtree formed by merging the multiple first sibling subtrees. When the specified node exists in the second DOM tree, multiple sibling second subtrees with the same structural label under the specified node in the second DOM tree are merged to obtain a second merged tree, wherein the second merged tree includes the second merged subtree formed by merging the multiple sibling second subtrees. The first main content area of ​​the first webpage is determined based on the first merged tree, and the second main content area of ​​the second webpage is determined based on the second merged tree.

3. The method according to claim 2, characterized in that, The step of determining the first main content area of ​​the first webpage based on the first merged tree and determining the second main content area of ​​the second webpage based on the second merged tree includes: Based on the first merged tree, first attribute information of each content area in the first webpage is extracted, wherein the first attribute information includes at least one of the following: the content skeleton tag of the first webpage, the first type keyword, the first tag keyword, the number of first subtrees, the number of first merged subtrees, and the first specified keyword; the first specified keyword includes at least the keyword of the excluded area in the first webpage. The first content area in the first webpage is scored based on the first attribute information, and the first main content area is determined based on the scoring results. Based on the second merged tree, the second attribute information of the second webpage is extracted, wherein the second attribute information includes at least one of the following: the content skeleton tag of the second webpage, the second type keyword, the second tag keyword, the number of second subtrees, the number of second merged subtrees, and the second specified keyword; the second specified keyword includes at least the keyword in the excluded area of ​​the second webpage. The second content area in the second webpage is scored based on the second attribute information, and the second main content area is determined based on the scoring results.

4. The method according to claim 3, characterized in that, The step of scoring each content area in the first webpage based on the first attribute information and determining the first main content area based on the scoring results includes: Determine the first basic score for each content area in the first webpage based on the first attribute information; When a content area in the first webpage matches the exclusion condition corresponding to the first specified keyword, the first base score of the content area is reduced, or the content area is removed from the first candidate area. Based on the adjusted first basic score, the first main content area is determined from the first candidate area; Determine the second basic score for each content area in the second webpage based on the second attribute information; When a content area in the second webpage matches the exclusion condition corresponding to the second specified keyword, the second base score of the content area is reduced, or the content area is removed from the second candidate area; The second main content area is determined from the second candidate area based on the adjusted second base score.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area includes: The first main content area is subjected to template normalization processing to obtain the first main content template skeleton; The second main content area is template normalized to obtain the second main content template skeleton; The first main content template skeleton is serialized and hashed to obtain the first fingerprint hash value; The second main content template skeleton is serialized and hashed to obtain the second fingerprint hash value.

6. The method according to claim 5, characterized in that, The step of performing template normalization processing on the first main content area to obtain the first main content template skeleton includes at least one of the following: The information in the first main content area that represents the template structure, including structural tags, hierarchical relationships, parent-child relationships, sibling order, node category attributes, and the first merged subtree, is retained. Delete the text content and event attributes in the first main content area; Replace the link address, page identifier parameter, time parameter, random parameter and / or frame generation class name in the first main content area with the corresponding preset tag; The structural differences in the first main content area caused by changes in the number of main text paragraphs, list items, or pagination links are made consistent. The step of performing template normalization processing on the second main content area to obtain the second main content template skeleton includes at least one of the following: The structural tags, hierarchical relationships, parent-child relationships, sibling order, node category attributes, and information of the second merged subtree used to characterize the template structure are retained in the second main content area; Delete the text content and event attributes in the second main content area; Replace the link address, page identifier parameter, time parameter, random parameter and / or frame generation class name in the second main content area with the corresponding preset tag; The structural differences caused by changes in the number of main text paragraphs, list items, or pagination links in the second main content area are consistent.

7. The method according to claim 6, characterized in that, The process of unifying structural differences in the first main content area caused by variations in the number of main text paragraphs, list entries, or pagination links includes at least one of the following: For the first main content node set in the first main content area, retain the outer structure tag and node category attribute of the first main content node set, and ignore the number of paragraph nodes and text content in the first main content node set; For the first pagination node set in the first main content area, retain the outer structure tag and node category attribute of the first pagination node set, and ignore the number of page number links and page number text content in the first pagination node set; For the set of nodes in the first main content area whose number of first merged subtrees meets the number threshold, the number of first merged subtrees is replaced with a first preset number marker. The process of unifying structural differences in the second main content area caused by variations in the number of main text paragraphs, list entries, or pagination links includes at least one of the following: For the second main content node set in the second main content area, retain the outer structure tags and node category attributes of the second main content node set, and ignore the number of paragraph nodes and text content in the second main content node set; For the second pagination node set in the second main content area, retain the outer structure tag and node category attribute of the second pagination node set, and ignore the number of page number links and page number text content in the second pagination node set; For the set of nodes in the second main content area whose number of second merged subtrees meets the specified number threshold, the number of second merged subtrees is replaced with a second preset number marker.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The first navigation area of ​​the first webpage is determined based on the first DOM tree or the first merged tree; The second navigation area of ​​the second webpage is determined based on the second DOM tree or the second merge tree; Obtain the first navigation hash value of the first navigation region and the second navigation hash value of the second navigation region; When the first fingerprint hash value and the second fingerprint hash value are different, and the first navigation hash value and the second navigation hash value are the same, auxiliary diagnostic information is output. The auxiliary diagnostic information is used to indicate that the navigation areas of the first webpage and the second webpage are the same, but the main content areas are different.

9. The method according to any one of claims 1 to 4, characterized in that, The construction of a first Document Object Model (DOM) tree and a second DOM tree based on the original Hypertext Markup Language (HTML) source code of the first and second web pages to be compared includes: The original HTML source code of the first webpage is cleaned to obtain the first cleaned HTML source code; The first DOM tree is constructed based on the first cleaned HTML source code; The original HTML source code of the second webpage is cleaned to obtain the second cleaned HTML source code; The second DOM tree is constructed based on the second cleaned HTML source code; The cleaning process includes deleting at least one of unstructured tags and tag attributes. The unstructured tags include at least one of script tags, style tags, embedded frame tags, and media tags. The tag attributes include at least one of event attributes, display style attributes, and resource address attributes.

10. A webpage template structure determining device, characterized in that, include: The building module is used to construct a first Document Object Model (DOM) tree and a second DOM tree based on the original Hypertext Markup Language (HTML) source code of the first and second web pages to be compared. The determination module is used to determine the first main content area of ​​the first webpage based on the first DOM tree, and to determine the second main content area of ​​the second webpage based on the second DOM tree; The acquisition module is used to acquire the first fingerprint hash value of the first main content area and the second fingerprint hash value of the second main content area; The comparison module is used to compare the first fingerprint hash value with the second fingerprint hash value to obtain a comparison result; The output module is configured to output a first result when the first fingerprint hash value and the second fingerprint hash value are the same, wherein the first result indicates that the DOM structure of the first webpage and the second webpage are the same; and to output a second result when the first fingerprint hash value and the second fingerprint hash value are different, wherein the second result indicates that the DOM structure of the first webpage and the second webpage are different.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.