A webpage update identification method, device, medium and equipment

Abstract

The present application relates to the technical field of internet data collection and analysis, and particularly relates to a webpage update identification method and device, medium and equipment, which realizes accurate positioning of the same content block at different time points by constructing an index mapping relationship based on a positioning identifier, avoids the calculation overhead and matching error of full tree traversal matching, calculates the structural similarity and selects a target subtree pair with stable structure, effectively filters noise blocks such as advertising positions and carousel pictures, reduces the false positive rate caused by non-substantial updates, extracts deep link addresses and compares the content hash values of the link target pages, deepens the update identification granularity from the surface page to the deep content, realizes accurate detection of substantial content changes, and determines substantial updates when the content hash values corresponding to any one deep link address are inconsistent, so that the update determination is based on the actual changes of the deep content, and the identification accuracy and reliability are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of Internet data collection and analysis technology, and in particular to a method, apparatus, medium and device for identifying webpage updates. Background Technology

[0002] With the rapid development of internet technology, the frequency of website content updates is increasing. Accurately identifying whether a webpage has undergone substantial content updates is of great significance for application scenarios such as search engine index updates, website content monitoring, and public opinion analysis.

[0003] In existing technologies, common methods for identifying webpage updates mainly fall into two categories: one type determines whether a webpage has changed by comparing the structural differences of the Document Object Model (DOM) tree of two webpages at two different points in time. Specifically, after parsing the webpage into a DOM tree, the type of change at the subtree level (such as insertion, deletion, modification, or movement) is determined by comparing the hash values ​​or structural changes of nodes in the two DOM trees, and whether it is a valid update is determined based on preset change types and content weights. However, this method requires pre-setting content weights for each block of the webpage and cannot automatically distinguish between substantive content updates and non-substantive layout changes such as advertisements, carousels, and recommendation lists, resulting in recognition results that rely on manual configuration and have a high false positive rate.

[0004] Another approach determines content updates by extracting link addresses from web pages. Specifically, it extracts all link addresses from the target webpage's list page, assembles them into a link string, and performs a hash comparison. If the hash value changes, the webpage content is considered updated. While this method addresses link changes, it only compares the link address string itself and cannot further detect whether the content of the target page linked to has undergone substantial changes. Furthermore, this method also generates false positives when advertising links, dynamic recommendation links, etc., change, failing to effectively filter out noisy blocks.

[0005] In summary, existing technologies struggle to accurately distinguish between substantive content updates and non-substantive layout or advertising changes during webpage update identification, resulting in a high false alarm rate and impacting the crawling efficiency and accuracy of web crawler systems.

[0006] Therefore, how to accurately distinguish between substantial and non-substantive webpage updates and reduce the false alarm rate of update identification has become an urgent problem to be solved. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention provides a webpage update identification method, which includes the following steps: S1. Based on the preset block positioning rules, construct the first index mapping relationship between each first subtree in the first syntax tree and the positioning identifier, and the second index mapping relationship between each second subtree in the second syntax tree and the positioning identifier. The first syntax tree corresponds to the first webpage of the target webpage at the first time point, and the second syntax tree corresponds to the second webpage of the target webpage at the second time point before the first time point.

[0008] S2, determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship.

[0009] S3. Based on the structural similarity between the first and second subtrees in each initial subtree pair, select several target subtree pairs whose structural similarity is greater than a preset similarity threshold.

[0010] S4, extract several deep link addresses contained in the first subtree of each target subtree pair.

[0011] S5, calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point.

[0012] S6. When the two content hash values ​​corresponding to any deep link address are inconsistent, it is determined that the target webpage has undergone a substantial content update.

[0013] The present invention also provides a webpage update identification device, which includes: The index mapping relationship construction module is used to construct a first index mapping relationship between each first subtree in the first syntax tree and the positioning identifier, and a second index mapping relationship between each second subtree in the second syntax tree and the positioning identifier, according to the preset block positioning rules. The first syntax tree corresponds to the first webpage of the target webpage at the first time point, and the second syntax tree corresponds to the second webpage of the target webpage at the second time point before the first time point.

[0014] The initial subtree pair determination module is used to determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship.

[0015] The target subtree pair filtering module is used to filter out several target subtree pairs whose structural similarity is greater than a preset similarity threshold based on the structural similarity between the first and second subtrees in each initial subtree pair.

[0016] The link address extraction module is used to extract several deep link addresses contained in the first subtree of each target subtree pair.

[0017] The hash value calculation module is used to calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point.

[0018] The webpage update identification module is used to determine that the target webpage has undergone a substantial content update when there are two inconsistent content hash values ​​corresponding to any deep link address.

[0019] The present invention also provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described webpage update identification method.

[0020] The present invention also provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0021] This invention has at least the following beneficial effects: By constructing an index mapping relationship based on location identifiers, it achieves accurate positioning of the same content block at different time points, avoiding the computational overhead and matching errors of full tree traversal matching; by calculating structural similarity and filtering structurally stable target subtree pairs, it effectively filters noisy blocks such as ad slots and carousels, reducing the false alarm rate caused by non-substantive updates; by extracting deep link addresses and comparing the content hash values ​​of the linked target pages, it deepens the update identification granularity from the surface page to the deep content, achieving accurate detection of substantive content changes; by determining substantive updates when the content hash values ​​corresponding to any deep link address are inconsistent, it makes update determination based on actual changes in deep content, significantly improving the accuracy and reliability of identification. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a webpage update identification method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a webpage update identification device provided in Embodiment 2 of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the terms used to distinguish similar objects can be interchanged so that the invention can also be implemented in other embodiments besides the illustrated or described embodiments. Furthermore, the terms "including," "having," and any variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0026] Example 1 This first embodiment provides a method for identifying webpage updates, such as... Figure 1 As shown, the webpage update identification method includes the following steps: S1. Based on the preset block positioning rules, construct the first index mapping relationship between each first subtree in the first syntax tree and the positioning identifier, and the second index mapping relationship between each second subtree in the second syntax tree and the positioning identifier. The first syntax tree corresponds to the first webpage of the target webpage at the first time point, and the second syntax tree corresponds to the second webpage of the target webpage at the second time point before the first time point.

[0027] In this context, the target webpage is a specific webpage that is monitored and its substantive content updates are identified. It can be a key page with content update requirements (such as the website homepage, category list page, or detail index page). In incremental crawling or website content monitoring scenarios, the target webpage is the core object whose update status needs to be checked regularly.

[0028] The first time point is the current detection time or the end time of the detection period. The second time point is a historical moment before the current detection period, such as the end time of the previous detection period. The version of the target webpage corresponding to the first time point is called the first webpage, and the version corresponding to the second time point before the first time point is called the second webpage. These constitute the comparison benchmark for updating and identifying the target webpage.

[0029] A syntax tree (SLT) is a tree-like data structure generated from the HTML source code of a webpage after processing by a parsing tool. It represents each HTML element and its hierarchical relationship in the form of nodes, providing a standardized data foundation for programmatically accessing, traversing, and comparing the webpage structure. The root node typically corresponds to the `<html>` tag, which contains child nodes such as `` and ``. These child nodes, in turn, contain various element nodes that constitute the visible content of the webpage (e.g., ``, `<h1>`, ``, `<h3>`, `<h4>`, `<h5>`, `<h6>`, `<h7>`, `<h8>`, `<h1 ...1>`, `<h1<h1> 、 、 wait).

[0030] A subtree is a local tree structure with relatively independent semantics extracted from the syntax tree according to preset partitioning rules. Each subtree corresponds to a content block on a webpage (such as a navigation bar, news list, sidebar, footer, etc.). By partitioning into subtrees, the complex full-page syntax tree can be decomposed into several functionally independent block units, facilitating subsequent precise positioning and differentiated processing.

[0031] A location identifier is a string identifier used to uniquely identify the position of a subtree within a webpage. Because the layout of the same target webpage may be adjusted or its content updated at different points in time, the subtrees corresponding to the same content block in the syntax tree obtained at different times may be in different structural positions or have different internal node details. Location identifiers can transcend the time dimension, establishing a stable correspondence for the same content block in different versions of the syntax tree, enabling subsequent comparisons and updates to be performed within the same block.

[0032] The first and second index mapping relationships are key-value pair mapping sets constructed using location identifiers as keys and corresponding subtrees as values. By constructing the first and second index mapping relationships, the corresponding subtrees can be quickly found by matching location identifiers, achieving efficient pairing of the same content blocks in web pages at two different points in time.

[0033] Preset block positioning rules are a set of rules used to extract or generate positioning identifiers from the top-level nodes of a subtree. These rules include: prioritizing the use of the `id` attribute value of the top-level node as the positioning identifier; and generating a positioning identifier based on the tag name, `class` attribute value, and its hierarchical path in the syntax tree when the top-level node does not have a valid `id` attribute. Preset block positioning rules balance the uniqueness and stability of positioning identifiers, ensuring that the same content block can be assigned the same or highly similar positioning identifiers in web pages retrieved at different times.

[0034] In one specific embodiment, S1 includes the following steps: S11, divide the subtrees by taking the direct child element nodes under the labels in the first syntax tree and the second syntax tree as the unit, to obtain several first subtrees in the first syntax tree and several second subtrees in the second syntax tree.

[0035] S12, for any first subtree or second subtree, obtain the top-level node of the current subtree. If the current top-level node has a non-empty id attribute, extract the id attribute value as the positioning identifier corresponding to the current subtree.

[0036] S13. If the current top-level node does not have a non-empty id attribute, extract the tag name and class attribute value of the current top-level node, and construct the hierarchical path from the document top-level node of the corresponding syntax tree to the current top-level node, where the document top-level node is the node corresponding to the tag.

[0037] S14: Based on the hierarchical path, tag name, and class attribute value, generate the location identifier corresponding to the current subtree.

[0038] In a webpage, the visible content is typically contained within tags, and their direct child element nodes constitute the main layout blocks of the page. Dividing the page into subtrees based on the direct child element nodes under each tag naturally divides the page into several semantically independent blocks, conforming to the actual structural characteristics of webpage design. Correspondingly, each direct child element node is taken as the root node, and its nodes, along with all their descendants, are extracted to form an independent subtree.

[0039] In the HTML specification, the `id` attribute is globally unique, meaning that no two elements on the same page should have the same `id` value. Therefore, if the top-level node of a subtree has a valid, non-empty `id` attribute, that `id` value naturally constitutes a unique identifier for the subtree's position on the page, serving as an ideal positioning identifier.

[0040] When a subtree's top-level node lacks an `id` attribute, the subtree cannot be directly located using a unique identifier. In this case, it's necessary to utilize the top-level node's structural position information within the syntax tree to construct a recognizable location identifier. Correspondingly, starting from the top-level node, the process traces back upwards along the parent node pointers, recording the tag name of each ancestor node sequentially until the document's top-level node corresponding to the tag is reached. The recorded tag name sequence is then reversed to a top-down order and concatenated with a preset separator (such as a right angle bracket `>`) to form a hierarchical path. The tag name and `class` attribute value of the top-level node are extracted, and whitespace characters in the `class` attribute value are normalized. The hierarchical path, tag name, and `class` attribute value are combined according to a preset format to form a string identifier that comprehensively reflects the subtree's structural position and style characteristics, serving as the corresponding location identifier. Since the core content blocks of a webpage typically have relatively fixed hierarchical structures and style names, this location identifier exhibits high stability across webpages at different times. Even without an `id` attribute, it can effectively distinguish content blocks of different positions and styles on the page, providing a reliable basis for subsequent cross-time matching.

[0041] As described above, by dividing the subtree, determining the location identifier, and constructing the index mapping relationship, the subtrees at two time points can be efficiently retrieved and paired based on a unified location identifier key-value system, which improves the execution efficiency of subsequent subtree matching steps and reduces the computational overhead of performing a full traversal of the entire tree.

[0042] S2, determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship.

[0043] The location identifier is extracted or generated from the top-level node of each subtree based on preset block location rules. The same content block at different time points will be assigned the same location identifier. Therefore, if a certain location identifier exists in both the first index mapping relationship and the second index mapping relationship, it means that the content block corresponding to that location identifier exists in both the first and second time point web pages. The content block corresponding to this location identifier is the content block shared by the web pages at both time points.

[0044] The first and second subtrees corresponding to the same location identifier logically correspond to the state of the same content block at two different points in time. They are identified as the initial subtree pair, establishing a one-to-one correspondence based on the location identifier, which provides a pairing basis for subsequent structural similarity calculation and content update analysis.

[0045] As described above, by extracting the location identifier sets from the first and second index mapping relationships and taking their intersection, the common content blocks in web pages at two time points are quickly identified, avoiding interference from the updated identification results caused by the addition or deletion of blocks due to page layout changes, and reducing the false alarm rate of non-substantive updates. By determining the initial subtree pairs based on the intersection of location identifiers, the states of the same content block at two time points are accurately matched, providing a reliable basic unit for subsequent structural similarity calculation and deep content comparison.

[0046] S3. Based on the structural similarity between the first and second subtrees in each initial subtree pair, select several target subtree pairs whose structural similarity is greater than a preset similarity threshold.

[0047] Structural similarity is used to quantify the degree of structural similarity between two subtrees, and its value is a floating-point number between 0 and 1. A higher structural similarity indicates that the two subtrees are more similar in terms of node composition, hierarchical relationships, and attribute distribution; conversely, a lower structural similarity indicates that the two subtrees have significant structural differences. In this embodiment, structural similarity is used to determine whether the same content block has undergone significant structural changes between two time points, thereby distinguishing between structurally stable core content blocks and structurally volatile noise blocks.

[0048] The preset similarity threshold is a critical value used to determine whether the subtree structure is stable. Its value is typically a floating-point number between 0.8 and 0.9 (e.g., 0.85). When the structural similarity of the initial subtree pair is greater than this threshold, it indicates that the content block corresponding to the subtree pair has maintained high structural stability between two time points and can be identified as the target subtree pair and enter the subsequent deep content comparison process. Conversely, when the structural similarity is less than or equal to this threshold, it indicates that the structure of the subtree pair has changed significantly and should be marked as a noise block and excluded from the substantive content update judgment.

[0049] Target subtree pairs refer to initial subtree pairs that are identified as structurally stable after structural similarity screening. The content blocks corresponding to these subtree pairs maintain relatively consistent structural characteristics between two points in time, typically corresponding to the core content areas of a webpage (such as news lists, article text, product display areas, etc.). Target subtree pairs are the primary targets for subsequent deep link extraction and content hash comparison.

[0050] In one specific embodiment, S3 includes the following steps: S31, for the first subtree in any initial subtree pair, generate the first node hash value corresponding to each node based on the label name, attribute key-value pair and text content of each node in the current first subtree.

[0051] S32, traverse all nodes in the current first subtree to obtain the hash set of the first node corresponding to the current first subtree.

[0052] S33, for the second subtree in any initial subtree pair, generate the second node hash value corresponding to each node based on the label name, attribute key-value pair and text content of each node in the current second subtree.

[0053] S34, traverse each node in the current second subtree to obtain the hash set of the second node corresponding to the current second subtree.

[0054] S35, calculate the structural similarity of the current initial subtree pair using a preset similarity algorithm based on the hash set of the first node and the hash set of the second node.

[0055] For each node, firstly, its label name string is extracted; secondly, all its attribute key-value pairs are extracted, sorted in ascending order by attribute name, and concatenated to form an attribute string; finally, its text content is extracted and normalized to obtain a text string. The node's label name string, attribute string, and text string are concatenated into a single string in a preset order, and then input into a preset hash function (such as SHA-256) for calculation to obtain a fixed-length node hash value. The label name identifies the node type, the attribute key-value pairs reflect the node's style and functional characteristics, the text content embodies the semantic information carried by the node, and the node hash value comprehensively represents the node's structural attributes. Furthermore, the node hash values ​​of all nodes in a subtree are aggregated to form a node hash set.

[0056] The preset similarity algorithm can be either the Jaccard similarity coefficient or the tree edit distance algorithm, which will not be elaborated here.

[0057] In one specific embodiment, S3 further includes the following steps: S36, for any initial subtree pair, if the structural similarity of the current initial subtree pair is less than or equal to a preset similarity threshold, the current initial subtree pair is marked as a noise block within the time period corresponding to the first time point.

[0058] S37, count the number of times the current initial subtree pair is marked as a noise block within M consecutive time periods, where M is an integer greater than 1.

[0059] S38, when the ratio of the number of occurrences to M is greater than the preset noise confidence threshold, add the corresponding location identifier of the current initial subtree to the mask list.

[0060] Among them, initial subtree pairs with excessively low structural similarity indicate that their corresponding content blocks have undergone significant structural changes between two time points. Such changes usually stem from non-substantive factors such as ad rotation, recommended content updates, and dynamic script rendering, rather than substantial updates to core content. By marking them as noise blocks, they can be excluded from the judgment of substantial content updates within the current detection period, avoiding false positives caused by layout changes.

[0061] A single instance of a block being marked as noise may only be due to an occasional layout adjustment, insufficient to determine if the block is a permanent noise source. By recording noise marking over multiple consecutive time periods, the pattern and stability of the block's changes can be assessed more accurately. Specifically, a historical noise marking record for each initial subtree pair is maintained, its marking status is updated after each detection period, and the cumulative number of times a block is marked as noise over the most recent M periods is counted. M is an integer greater than 1, typically ranging from 3 to 5.

[0062] If a content block is frequently marked as a noise block within multiple consecutive time periods, it indicates that the block's structural changes are highly frequent and irregular, belonging to a typical dynamic layout region. Adding its location identifier to the masking list means that in subsequent update and recognition processes, the subtree corresponding to that identifier will be directly skipped, no longer participating in index mapping construction, structural similarity calculation, and content update judgment. This continuously optimizes the noise filtering capability of the recognition system, reduces unnecessary computational overhead, and improves long-term efficiency and accuracy.

[0063] The preset noise confidence threshold is a critical value used to determine whether to permanently block a content block, and it is usually set to 0.7. For example, when M=5, if a block is marked as a noise block more than 3 times within 5 periods, permanent blocking is triggered.

[0064] As described above, by calculating the structural similarity of each initial subtree pair and comparing it with a preset similarity threshold, the system effectively distinguishes between structurally stable core content blocks and structurally variable noise blocks, achieving intelligent filtering of page layout changes and avoiding interference from non-substantive updates such as ad carousels and dynamic recommendations on the recognition results. By statistically analyzing the frequency of the same initial subtree pair being marked as a noise block within multiple consecutive time periods, and adding its location identifier to the masking list when the frequency exceeds a preset noise confidence threshold, the system can automatically identify and permanently mask high-frequency noise blocks, achieving adaptive optimization of the noise filtering strategy and continuously improving the long-term operating efficiency and accuracy of the update recognition system.

[0065] S4, extract several deep link addresses contained in the first subtree of each target subtree pair.

[0066] Deep link addresses are hyperlinks (URLs) extracted from the first subtree of a target subtree pair, pointing to the next level content page of the target webpage. Unlike the target webpage itself (usually a list page, index page, or homepage), the pages pointed to by deep link addresses are usually specific content detail pages (such as news article pages, product detail pages, article reading pages, etc.). By comparing the content of the target pages pointed to by deep link addresses, it is possible to determine at a more granular level whether the target webpage has undergone substantial content updates.

[0067] In one specific embodiment, S4 includes the following steps: S41, for any target subtree pair, extract each element in the first subtree of the current target subtree pair. The value of the href attribute of the tag node.

[0068] S42, deduplicate all extracted href attribute values ​​to obtain several deep link addresses, where each deep link address corresponds to a deduplicated href attribute value.

[0069] in, Tag nodes are elements in the HTML syntax tree that correspond to hyperlink elements. The node object. In the HTML specification, The `<link>` tag is used to define hyperlinks. Its `href` attribute specifies the target URL address the link points to. This attribute value can be an absolute URL path or a relative URL path. During extraction and processing, relative URL paths will be converted to absolute URL paths to ensure the integrity and accessibility of the link address. The first subtree in the target subtree pair corresponds to the core content block of the target webpage at the first point in time after structural similarity filtering. The content block corresponding to the target subtree (such as news lists, product catalogs, etc.) usually contains a large amount of... Each tag node represents a link to a specific content page. By traversing these nodes and extracting their href attribute values, you can obtain all the deep link addresses associated with that content block.

[0070] Specifically, a depth-first or breadth-first traversal algorithm is used to determine the type of each node in the first subtree; details will not be elaborated here. If the current node is... For tag nodes, their href attribute values ​​are read; if the current node has child nodes, its child nodes are recursively traversed. For the extracted relative URL paths, they are concatenated and transformed according to the base URL of the target webpage to obtain the complete absolute URL path. Finally, the original set of all href attribute values ​​contained in the first subtree of the current target subtree pair is collected.

[0071] The original extracted href attribute value set may contain duplicates. Reasons for duplicates include, but are not limited to: the same content appearing multiple times in a list (such as a pinned recommendation and a regular list item), and the repeated appearance of functional links (such as a "next page" link appearing at both the top and bottom of the page). If duplicates are retained, subsequent crawling and content hash calculations of the same target page will be performed repeatedly, resulting in unnecessary network requests and computational overhead. Therefore, it is necessary to deduplicate the original href attribute value set.

[0072] Specifically, based on the standardized string of the href attribute value (ignoring superficial differences such as URL parameter order), duplicates are removed using a hash set or ordered set data structure. After deduplication, each unique href attribute value remaining in the set is a deep link address. Finally, the set of deep link addresses corresponding to the current target subtree is obtained, which serves as input data for subsequent crawling of target content and hash comparison.

[0073] As described above, by extracting deep link addresses only from target subtree pairs after structural similarity filtering, the extraction of invalid links from noisy blocks is avoided, thus improving the effectiveness of link extraction; by traversing all... By tagging nodes and extracting the href attribute value, a complete collection of all deep links in the core content block is achieved, avoiding missed updates. By deduplicating the extracted href attribute values, duplicate crawling and calculation of the same link are avoided, reducing computational overhead and improving recognition efficiency.

[0074] S5, calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point.

[0075] The content hash value is a fixed-length string calculated using a preset hash algorithm, uniquely representing the textual characteristics of the content to be hashed. If two pieces of content to be hashed are completely identical, their content hash values ​​are the same; if there are any differences in the text (including additions or deletions of characters, changes in punctuation, differences in spaces, etc.), their content hash values ​​are different. By comparing the content hash values ​​of the first and second target content of a link, it is possible to efficiently and accurately determine whether the content of the page pointed to by a deep link address has been substantially updated.

[0076] In one specific embodiment, S5 includes the following steps: S51, based on the preset link traversal depth N, crawl the page for each deep link address, and obtain the first link target content corresponding to each deep link address at the first time point, and the second link target content corresponding to the second time point.

[0077] S52, for each deep link address, extract the page title and the first paragraph of the main text of the target content of the current deep link address, and combine them to form the first content to be hashed.

[0078] S53, the first content to be hashed is calculated using a preset hash algorithm to obtain the content hash value of the first link target content.

[0079] S54, extract the page title and the first paragraph of the main text of the current deep link address in the target content of the second link, and combine them to form the second content to be hashed.

[0080] S55, the second content to be hashed is calculated using a preset hash algorithm to obtain the content hash value of the second link target content.

[0081] The preset link traversal depth N is the allowed depth of further link following when crawling target pages pointed to by deep link addresses. N is a non-negative integer. When N=0, only the page content directly pointed to by the deep link address is retrieved, and no further links within that page are crawled. When N=1, in addition to retrieving the directly pointed page content, the content of subpages pointed to by links within that page can also be crawled. In this embodiment, N is usually set to 0 or 1 to avoid excessive server pressure or triggering the website's anti-crawling mechanism due to excessive crawling. For news web pages, N=1 is sufficient to retrieve the entire content of the article's main text page; for forum web pages, N can be set to 2 to retrieve complete posts and replies.

[0082] The page pointed to by the deep link address is the next level content page of the target webpage, and its actual update status directly affects the overall update determination of the target webpage. Since webpage update identification is usually periodic, the second link target content at the second time point has been crawled and stored in the historical detection period. Therefore, in this embodiment, the second link target content can be directly retrieved from the historical storage; the first link target content can be obtained through real-time crawling.

[0083] Specifically, the process first queries the historical storage for the second target content corresponding to the current deep link address at the second time point. If the content exists in the historical storage, it is read directly; otherwise, it is marked as empty (e.g., the link is newly added). Then, a network request is initiated for the current deep link address to obtain the HTML content of the page corresponding to it at the first time point, which is used as the first target content. During the crawling process, the preset link traversal depth N is strictly followed. If N=0, only the page content directly returned by the deep link address is obtained, without parsing or following any links within the page. If the crawling fails (e.g., network timeout, page not found), the deep link address is marked as abnormal and will not participate in subsequent content hash comparisons.

[0084] Page title (usually located in) <title> The text within the HTML tags and the first paragraph of the main content are a high-level summary of the webpage content, effectively reflecting the core theme and information of the page. By extracting and combining these two parts, a concise and representative text to be hashed can be formed, avoiding the noise interference and computational overhead caused by including the entire page's HTML in the hash calculation.< / title>

[0085] Specifically, the HTML structure of the target content of the first link is parsed to locate... <title> Tags are used to extract text content as page titles; the body text container tags (such as...) are located.< / title> <article> 、 (etc.) or paragraph tags (such as...) Extract the first paragraph or several paragraphs of text as the first paragraph of the main text. Concatenate the page title and the first paragraph of the main text in a preset order (e.g., by connecting them with line breaks or specific delimiters) to form the first content to be hashed.

[0086] The preset hash algorithm can use common hash functions such as MD5, SHA-1, or SHA-256. SHA-256 is preferred because it offers higher collision resistance while maintaining computational efficiency. The first string to be hashed is input into the hash function, and the output is a fixed-length hash value string, which is the content hash value of the first link target content. This hash value will serve as the digital fingerprint of the page content corresponding to this deep link address at the first time point, used for subsequent comparison with the content hash value at the second time point.

[0087] Similarly, the target content of the second link and its content hash value are obtained.

[0088] As described above, by controlling the crawling range through the preset link traversal depth N, the crawling range is controlled, which satisfies the content comparison requirements while avoiding excessive crawling and achieving a balance between crawling efficiency and information sufficiency. By using a unified extraction rule and hash algorithm to process the content at two different time points, the calculation of the content hash value is ensured to be consistent, avoiding misjudgments caused by differences in processing methods.

[0089] S6. When the two content hash values ​​corresponding to any deep link address are inconsistent, it is determined that the target webpage has undergone a substantial content update.

[0090] Substantive content updates refer to updates to the deeper content associated with the core content blocks of a target webpage that involve incremental or changed information, rather than non-substantive changes such as webpage layout, ad carousels, dynamic recommendation slots, or timestamp changes. In this embodiment, the criterion for determining substantive content updates is that at least one deep link address has a content hash value for the first link target content at a first time point that is inconsistent with the content hash value for the second link target content at a second time point. This criterion focuses on the actual changes in the deep content of the webpage, effectively eliminating the interference of surface structure changes on the update identification results.

[0091] When a target webpage is determined to have undergone a substantial content update, a substantial update flag (such as a boolean value of True) is generated, and optionally, a list of deep link addresses that have been updated is output for the crawler system to perform incremental crawling operations. When the two content hash values ​​of all deep link addresses are consistent, it is determined that the target webpage has not undergone a substantial content update, a no-update flag (such as a boolean value of False) is generated, and monitoring continues in the next detection cycle.

[0092] As described above, by comparing the content hash values ​​of deeply linked target pages to determine the nature of page updates, the system achieves accurate detection of substantive content changes and effectively eliminates interference from non-substantive changes such as layout adjustments and ad rotations. By adopting the logic of "determining an update if the content hash value of any deeply linked address is inconsistent," the system ensures that any core content update on the target webpage can be captured in a timely manner, avoiding missed updates.

[0093] As described above, by constructing an index mapping relationship based on location identifiers, accurate positioning of the same content block at different time points is achieved, avoiding the computational overhead and matching errors of full tree traversal matching. By calculating structural similarity and filtering structurally stable target subtree pairs, noisy blocks such as ad slots and carousels are effectively filtered out, reducing the false alarm rate caused by non-substantive updates. By extracting deep link addresses and comparing the content hash values ​​of the linked target pages, the update recognition granularity is deepened from the surface page to the deep content, achieving accurate detection of substantive content changes. By determining substantive updates when the content hash values ​​corresponding to any deep link address are inconsistent, update judgment is based on actual changes in deep content, significantly improving the accuracy and reliability of recognition.

[0094] Example 2 This second embodiment provides a webpage update identification device, such as... Figure 2 As shown, it includes: The index mapping relationship construction module 21 is used to construct a first index mapping relationship between each first subtree in the first syntax tree and the positioning identifier, and a second index mapping relationship between each second subtree in the second syntax tree and the positioning identifier, according to the preset block positioning rules. The first syntax tree corresponds to the first webpage of the target webpage at the first time point, and the second syntax tree corresponds to the second webpage of the target webpage at the second time point before the first time point.

[0095] The initial subtree pair determination module 22 is used to determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship.

[0096] The target subtree pair filtering module 23 is used to filter out several target subtree pairs whose structural similarity is greater than a preset similarity threshold based on the structural similarity between the first subtree and the second subtree in each initial subtree pair.

[0097] Link address extraction module 24 is used to extract several deep link addresses contained in the first subtree of each target subtree pair.

[0098] The hash value calculation module 25 is used to calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point.

[0099] The webpage update identification module 26 is used to determine that the target webpage has undergone a substantial content update when there are two inconsistent content hash values ​​corresponding to any deep link address.

[0100] In one specific embodiment, the index mapping relationship construction module 21 includes: The subtree partitioning module is used to partition subtrees based on the direct child element nodes under the tags in the first syntax tree and the second syntax tree, resulting in several first subtrees in the first syntax tree and several second subtrees in the second syntax tree.

[0101] The location identifier extraction submodule is used to obtain the top-level node of the current subtree for any first subtree or second subtree. If the current top-level node has a non-empty id attribute, the id attribute value is extracted as the location identifier corresponding to the current subtree.

[0102] The information extraction submodule is used to extract the tag name and class attribute value of the current top-level node if the current top-level node does not have a non-empty id attribute, and construct the hierarchical path from the document top-level node of the corresponding syntax tree to the current top-level node, where the document top-level node is the node corresponding to the tag.

[0103] The location identifier construction submodule is used to generate the location identifier corresponding to the current subtree by combining the hierarchical path, tag name, and class attribute value.

[0104] In one specific implementation, the target subtree filtering module 23 includes: The first node hash value generation submodule is used to generate the first node hash value for each node in any initial subtree pair, based on the label name, attribute key-value pair and text content of each node in the current first subtree.

[0105] The first set acquisition submodule is used to traverse all nodes in the current first subtree and obtain the hash set of the first node corresponding to the current first subtree.

[0106] The second node hash value generation submodule is used to generate the second node hash value for each node in any initial subtree pair, based on the label name, attribute key-value pair and text content of each node in the current second subtree.

[0107] The second set acquisition submodule is used to traverse each node in the current second subtree and obtain the hash set of the second node corresponding to the current second subtree.

[0108] The structural similarity calculation submodule is used to calculate the structural similarity of the current initial subtree pair based on the hash set of the first node and the hash set of the second node using a preset similarity algorithm.

[0109] In one specific embodiment, the target subtree filtering module 23 further includes: The noise block determination submodule is used to mark any initial subtree pair as a noise block within the time period corresponding to the first time point if the structural similarity of the current initial subtree pair is less than or equal to a preset similarity threshold.

[0110] The frequency statistics submodule is used to count the number of times the current initial subtree pair is marked as a noise block within M consecutive time periods, where M is an integer greater than 1.

[0111] The mask list update submodule is used to add the corresponding location identifier of the current initial subtree to the mask list when the ratio of the number of times to M is greater than the preset noise confidence threshold.

[0112] In one specific embodiment, the link address extraction module 24 includes: The attribute value extraction submodule is used to extract each attribute value from the first subtree of any given target subtree pair. The value of the href attribute of the tag node.

[0113] The link address extraction submodule is used to deduplicate all extracted href attribute values ​​to obtain several deep link addresses, where each deep link address corresponds to a deduplicated href attribute value.

[0114] In one specific embodiment, the hash value calculation module 25 includes: The content acquisition submodule is used to crawl the page for each deep link address according to the preset link traversal depth N, and obtain the first link target content corresponding to each deep link address at the first time point, and the second link target content corresponding to the second time point.

[0115] The first submodule for obtaining the content to be hashed is used to extract the page title and the first paragraph of the main text of the target content of the current deep link address for each deep link address, and combine them to form the first content to be hashed.

[0116] The first content hash value calculation submodule is used to calculate the content hash value of the first content to be hashed using a preset hash algorithm.

[0117] The second submodule for obtaining the content to be hashed is used to extract the page title and the first paragraph of the main text of the target content of the current deep link address, and combine them to form the second content to be hashed.

[0118] The second content hash value calculation submodule is used to calculate the content hash value of the second content to be hashed using a preset hash algorithm.

[0119] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0120] Example 3 Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the webpage update identification method provided in the above embodiment.

[0121] Example 4 Embodiment 4 of the present invention provides an electronic device, which includes a processor and a non-transitory computer-readable storage medium as described in Embodiment 3 of the present invention.

[0122] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention. < / article>

Claims

1. A method for identifying webpage updates, characterized in that, The webpage update identification method includes the following steps: S1, according to the preset block positioning rules, construct a first index mapping relationship between each first subtree in the first syntax tree and the positioning identifier, and a second index mapping relationship between each second subtree in the second syntax tree and the positioning identifier. The first syntax tree corresponds to the first webpage of the target webpage at a first time point, and the second syntax tree corresponds to the second webpage of the target webpage at a second time point before the first time point. The preset block positioning rules are a set of rules used to extract or generate positioning identifiers from the top-level nodes of the subtrees, including: using the id attribute value of the top-level node of the subtree as the positioning identifier; when the top-level node does not have a non-empty id attribute, generating a positioning identifier based on the tag name, class attribute value, and the hierarchical path of the top-level node in the syntax tree; and assigning the same positioning identifier to the same content block in webpages obtained at different time points. S1 includes the following steps: S11, divide the subtrees by taking the direct child element nodes under the tags in the first syntax tree and the second syntax tree as units, to obtain several first subtrees in the first syntax tree and several second subtrees in the second syntax tree; S12, for any first subtree or second subtree, obtain the top-level node of the current subtree. If the current top-level node has a non-empty id attribute, extract the id attribute value as the positioning identifier corresponding to the current subtree. S13, if the current top-level node does not have a non-empty id attribute, then extract the tag name and class attribute value of the current top-level node, and construct the hierarchical path from the document top-level node of the corresponding syntax tree to the current top-level node, wherein the document top-level node is the node corresponding to the tag. S14, Based on the hierarchical path, the tag name, and the class attribute value, generate the location identifier corresponding to the current subtree; S2, determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship; S3, based on the structural similarity between the first and second subtrees in each initial subtree pair, select several target subtree pairs whose structural similarity is greater than a preset similarity threshold; S4, extract several deep link addresses contained in the first subtree of each target subtree pair; S5, calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point; S6. When the two content hash values ​​corresponding to any deep link address are inconsistent, it is determined that the target webpage has undergone a substantial content update.

2. The webpage update identification method according to claim 1, characterized in that, S3 includes the following steps: S31, For the first subtree in any initial subtree pair, generate the hash value of the first node corresponding to each node based on the label name, attribute key-value pair and text content of each node in the current first subtree; S32, traverse all nodes in the current first subtree to obtain the hash set of the first node corresponding to the current first subtree; S33, for the second subtree in any initial subtree pair, generate the hash value of the second node corresponding to each node based on the label name, attribute key-value pair and text content of each node in the current second subtree; S34, traverse each node in the current second subtree to obtain the hash set of the second node corresponding to the current second subtree; S35, based on the hash set of the first node and the hash set of the second node, calculate the structural similarity of the current initial subtree pair using a preset similarity algorithm.

3. The webpage update identification method according to claim 2, characterized in that, S3 also includes the following steps: S36, for any initial subtree pair, if the structural similarity of the current initial subtree pair is less than or equal to the preset similarity threshold, the current initial subtree pair is marked as a noise block within the time period corresponding to the first time point; S37, count the number of times the current initial subtree pair is marked as a noise block within M consecutive time periods, where M is an integer greater than 1; S38, when the ratio of the number of times to M is greater than the preset noise confidence threshold, the corresponding positioning identifier of the current initial subtree is added to the mask list.

4. The webpage update identification method according to claim 1, characterized in that, S4 includes the following steps: S41, for any target subtree pair, extract each element in the first subtree of the current target subtree pair. The value of the href attribute of the tag node; S42, deduplicate all extracted href attribute values ​​to obtain several deep link addresses, where each deep link address corresponds to a deduplicated href attribute value.

5. The webpage update identification method according to claim 1, characterized in that, S5 includes the following steps: S51, according to the preset link traversal depth N, crawl the page for each deep link address, and obtain the first link target content corresponding to each deep link address at the first time point, and the second link target content corresponding to the second time point. S52, for each deep link address, extract the page title and the first paragraph of the main text of the current deep link address in the first link target content, and combine them to form the first content to be hashed; S53, the first content to be hashed is calculated using a preset hash algorithm to obtain the content hash value of the first link target content; S54, extract the page title and the first paragraph of the main text of the current deep link address in the second link target content, and combine them to form the second content to be hashed; S55, the preset hash algorithm is used to calculate the hash value of the second content to be hashed, and the content hash value of the second link target content is obtained.

6. A webpage update identification device, characterized in that, The webpage update identification device includes: An index mapping relationship construction module is used to construct a first index mapping relationship between each first subtree in a first syntax tree and a positioning identifier, and a second index mapping relationship between each second subtree in a second syntax tree and a positioning identifier, according to preset block positioning rules. The first syntax tree corresponds to the first webpage of the target webpage at a first time point, and the second syntax tree corresponds to the second webpage of the target webpage at a second time point before the first time point. The preset block positioning rules are a set of rules used to extract or generate positioning identifiers from the top-level nodes of the subtrees, including: using the id attribute value of the top-level node of the subtree as the positioning identifier; when the top-level node does not have a non-empty id attribute, generating a positioning identifier based on the tag name, class attribute value, and the hierarchical path of the top-level node in the syntax tree; and assigning the same positioning identifier to the same content block in webpages obtained at different time points. The index mapping relationship construction module includes: The subtree partitioning submodule is used to partition the subtrees based on the direct child element nodes under the tags in the first syntax tree and the second syntax tree, to obtain several first subtrees in the first syntax tree and several second subtrees in the second syntax tree. The location identifier extraction submodule is used to obtain the top-level node of the current subtree for any first subtree or second subtree. If the current top-level node has a non-empty id attribute, the id attribute value is extracted as the location identifier corresponding to the current subtree. The information extraction submodule is used to extract the tag name and class attribute value of the current top-level node if the current top-level node does not have a non-empty id attribute, and construct the hierarchical path from the document top-level node of the corresponding syntax tree to the current top-level node, wherein the document top-level node is the node corresponding to the tag. The location identifier construction submodule is used to combine the hierarchical path, the tag name and the class attribute value to generate the location identifier corresponding to the current subtree; The initial subtree pair determination module is used to determine the first subtree and the second subtree corresponding to each positioning identifier that exists simultaneously in the first index mapping relationship and the second index mapping relationship as an initial subtree pair with a one-to-one positioning relationship. The target subtree pair filtering module is used to filter out several target subtree pairs whose structural similarity is greater than a preset similarity threshold based on the structural similarity between the first subtree and the second subtree in each initial subtree pair. The link address extraction module is used to extract several deep link addresses contained in the first subtree of each target subtree pair; The hash value calculation module is used to calculate the content hash value of the first link target content corresponding to each deep link address at the first time point, and the content hash value of the second link target content corresponding to the second time point; The webpage update identification module is used to determine that the target webpage has undergone a substantial content update when there are two inconsistent content hash values ​​corresponding to any deep link address.

7. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the webpage update identification method as described in any one of claims 1-5.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 7.

Images