Website map crawler method and device based on browser rendering

Abstract

The invention discloses a website map crawler method and device based on browser rendering, and relates to the computer network technology. The method comprises the steps that a current page of a target website is rendered through a browser rendering engine; after the page rendering is finished, simulating an interactive behavior of a user on the page, monitoring a DOM change event occurring in the page, and capturing a page dynamic path triggered by the interactive behavior of the user; constructing all captured page dynamic paths into an extensible path set of the current page, performing comprehensive priority scoring on each path to be accessed in the extensible path set, performing dynamic sorting on the path priorities according to a scoring result, and taking the path with the high priority as an access entry of next page rendering; and repeating the steps until a task termination condition is met. According to the invention, a set of website subpath capturing system with high coverage rate, high precision and high controllability is constructed.

Description

Technical Field

[0001] The present invention relates to computer network technology, and in particular to a website map crawler method and device based on browser rendering. Background Art

[0002] Currently, most existing website sub-path extraction tools rely primarily on parsing static HTML content. For example, these tools read and analyze a website's robots.txt and sitemap.xml files, or use regular expressions, XPath, CSSSelector, and other techniques to traverse and match the webpage's DOM structure to extract visible link information. While these methods may meet basic requirements for traditional static websites, they exhibit significant shortcomings when applied to modern websites, particularly single-page applications (SPAs) built with front-end frameworks like Vue and React, or scenarios that rely heavily on AJAX for dynamic loading.

[0003] On the one hand, static parsing methods cannot restore the actual page structure after JavaScript scripts are executed. A large amount of page content is dynamically generated by front-end logic during browser runtime, including but not limited to data blocks loaded asynchronously through methods such as fetch, axios, and XMLHttpRequest, or structural nodes rendered after component mounting. If relying solely on the static DOM, no matter how complex the page, the system will only be able to capture the raw HTML output, resulting in the inability to extract a large amount of actual path information, seriously hindering the comprehensive modeling and analysis of the website structure.

[0004] On the other hand, although some tools have begun to introduce headless browsers (such as Puppeteer, Playwright, and Selenium) to execute JavaScript logic in pages, most implementations still have the following problems: (1) The rendering process lacks perception capabilities. It is unable to dynamically adjust the waiting strategy based on indicators such as page loading status, number of asynchronous requests, and DOM change frequency. Instead, it often uses a fixed timeout, resulting in low crawling efficiency or content omissions. (2) Lack of user behavior simulation. For content that requires actions such as clicking "load more," scrolling the page, hovering the menu, switching tabs, etc. to trigger, existing solutions are difficult to accurately simulate or systematically cover, resulting in path discovery being heavily dependent on static links; (3) Insufficient intelligence in path scheduling and value assessment. Existing solutions have difficulty accurately identifying high-value links, and have not established a feedback loop for crawling results, making it impossible to achieve dynamic optimization of crawling results and efficient resource allocation.

[0005] (4) Structural changes and path evolution are uncontrollable. The lack of an effective recording and comparison mechanism for the structural state of the page during the interaction process makes it impossible for the crawler system to determine whether the rendering is complete, and it is also impossible to avoid repeated crawling or path omissions.

[0006] In summary, current mainstream path extraction methods, when faced with highly dynamic and interactively complex website structures, generally face bottlenecks such as incomplete rendering, incomplete path recognition, missing behavior triggers, and insufficient scheduling intelligence. There is an urgent need to introduce a new website sub-path crawler system with structure perception, behavior drive, path value feedback, and scheduling optimization capabilities to achieve breakthroughs. Summary of the Invention

[0007] In view of the above-mentioned defects or deficiencies in the prior art, the present invention provides a method and device for crawling a website map based on browser rendering, which can solve the technical problems mentioned in the background technology.

[0008] One aspect of the present invention provides a website map crawler method based on browser rendering, comprising the following steps: rendering the current page of a target website through a browser rendering engine; after the page rendering is completed, simulating the user's interactive behavior on the page, monitoring DOM change events occurring in the page, and capturing the page dynamic path triggered by the user's interactive behavior; constructing all captured page dynamic paths into an extensible path set of the current page, performing a comprehensive priority score on each path to be accessed in the extensible path set, dynamically sorting the path priorities according to the score results, and using the high-priority path as the access entrance for the next round of page rendering; repeating the above steps until the task termination condition is met.

[0009] On the other hand, the present invention also provides a website map crawler device based on browser rendering, including: a rendering module, configured to render the current page of the target website through the browser rendering engine; a path acquisition module, configured to simulate the user's interactive behavior on the page after the page rendering is completed, monitor the DOM change events occurring in the page, and capture the page dynamic path triggered by the user's interactive behavior; a path priority sorting module, configured to construct all captured page dynamic paths into an extensible path set of the current page, perform a comprehensive priority score on each path to be visited in the extensible path set, dynamically sort the path priorities according to the scoring results, and use the high-priority path as the access entrance for the next round of page rendering; a repeat execution module, configured to repeat the above steps until the task termination condition is met.

[0010] The present invention provides a method and device for crawling a website map based on browser rendering, which has the following beneficial effects: (1) The present invention provides a rendering environment perception and adaptive rendering control mechanism that can dynamically detect the target website's dependency on the client environment during the browser rendering process and adaptively adjust the simulated environment configuration based on the perception results to ensure the integrity and consistency of the page loading and rendering process. For example, by detecting indicators such as page loading status, number of asynchronous requests, and DOM change frequency, the waiting strategy can be dynamically adjusted to improve crawling efficiency and avoid missing crawled content.

[0011] (2) The present invention simulates the operation process of real users to dynamically activate the lazy loading content and deep path resources in the page.

[0012] (3) The present invention proposes a dynamic evaluation and scheduling mechanism for path priority based on multi-dimensional feature fusion, which comprehensively considers the path depth, DOM structure complexity, and page importance index, and combines the system resource occupancy status and task execution feedback results to give a comprehensive priority score to each page path. The path priority is adjusted in real time according to the comprehensive priority score, and then the sorting strategy of the path priority scheduling queue is implemented to achieve priority scheduling of high-value paths and delayed processing of low-yield paths, thereby improving the efficiency of website crawling and the ability to identify high-value paths.

[0013] (4) To ensure the structural stability and accuracy of the crawled path, the present invention introduces a rendering result difference comparison mechanism based on structure perception. After rendering the same page multiple times, the system compares the DOM tree structures generated by different rendering snapshots, detects their structural differences, key node offsets, and jump path changes, and thus identifies the dynamic nature and stability of the page content. Paths with highly stable structures are marked as "stable paths" and are preferentially included in the structure diagram. Page paths with large structural differences are marked as "dynamic paths" and the triggering conditions and context are recorded for subsequent dynamic behavior analysis and risk assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 This is a flowchart of a browser-rendered website map crawler method provided by an embodiment of the present application; Figure 2 This is a schematic diagram of the structure of a website map crawler device based on browser rendering provided by one embodiment of the present application; Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0017] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0018] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0019] It should be noted that the directional terms such as "upper," "lower," "left," and "right" described in the embodiments of the present invention are described from the perspectives shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. Furthermore, in the context, it should be understood that when an element is referred to as being formed "on" or "under" another element, it can be formed not only directly "on" or "under" the other element, but also indirectly "on" or "under" the other element through an intermediate element.

[0020] See also Figure 1 One embodiment of the present application provides a website map crawler method based on browser rendering, comprising the following steps: Step S101: Render the current page of the target website through the browser rendering engine.

[0021] Specifically, the browser rendering engine is started, the homepage of the target website is loaded and the complete rendering process is performed to ensure that all dynamic content that relies on JavaScript can be correctly parsed and presented. Furthermore, the page loading status, the number of asynchronous requests and the frequency of DOM changes are detected during the page rendering process, and the chromedp.Sleep() function is used to automatically adjust the time to wait for the page to load according to the complexity of the page loading, thereby ensuring the integrity and consistency of the page loading and rendering process. For example: when the DOM loading is slow, the waiting time for the page to load is extended, and when the resource failure rate is high, the waiting time for the page to load is shortened. By automatically adjusting the waiting time for the page to load, the crawling efficiency can also be improved to avoid omission of crawled content.

[0022] Step S102: After the page is rendered, simulate the user's interactive behavior on the page, monitor the DOM change events occurring in the page, and capture the dynamic path of the page triggered by the user's interactive behavior.

[0023] This step simulates the user's real interactive behavior in the browser to trigger potential page paths, thereby breaking through the limitations of traditional static crawlers on the acquisition of dynamically loaded content.

[0024] Specifically, through a control module embedded in the browser rendering environment, page elements with potential interactive triggering behaviors are automatically identified, such as "Load More" buttons, infinite scroll containers, drop-down menus, hover trigger areas, carousel components, etc., and the chromedp library is used to simulate user interaction behaviors such as clicking page buttons, expanding dynamic areas, scrolling page views, and mouse hovering, driving the event listeners registered through JavaScript in the page to execute jump logic, thereby revealing path information that traditional static analysis cannot capture. Furthermore, the event listener can be a page change listener, which is implemented by dynamically injecting a listening script during the page rendering process. The page change listener is used to monitor and capture DOM change events generated by JavaScript scripts, such as DOM structure changes, asynchronous resource loading, and path generation triggered by user interaction, and extract all potential access paths dynamically generated by JavaScript. The above mechanism significantly enhances the system's ability to identify dynamic content and interactive structures, and improves the integrity and coverage of path extraction.

[0025] In step S103, all captured dynamic paths of the page are constructed into an extensible path set of the current page, and each path to be accessed in the extensible path set is scored with a comprehensive priority. The path priorities are dynamically sorted according to the scoring results, and the high-priority paths are used as access entrances for the next round of page rendering.

[0026] Specifically, the dynamic paths captured by the listener are deduplicated, cleaned, and standardized, and uniformly converted into directly accessible absolute URL formats to form an extensible path set for the current page.

[0027] Furthermore, running JavaScript rendering on the server consumes a lot of CPU and memory, so the system needs to crawl as many high-value website sub-paths as possible within limited server resources, and high-value pages should be crawled and recorded first. To achieve this goal, this step performs a comprehensive priority score on each path to be visited in the extensible path set, and adjusts the priority of path crawling based on the comprehensive priority score, specifically including: The system uses the goquery library to parse the rendered page's DOM structure and calculate the nesting depth of each link node within the DOM tree. The closer a path is to the top-level node in the DOM structure, the shallower its structure, the lower the barrier to entry, and the more important the page. Therefore, a higher structural hierarchy score (or higher score weight) is assigned to that page path to reflect its centrality within the website.

[0028] During page loading, the system uses Chromium-based DOM (chromedp) to simulate user behaviors, such as scrolling and clicking, and capture new DOM elements generated by these interactions. The system identifies these dynamically generated paths and determines their dynamic content ratio. Pages with a high dynamic content ratio are often closely linked to user behavior and have a high interaction value. Therefore, these page paths are assigned a higher dynamic content ratio score (or a higher score weight).

[0029] The system uses the GoQuery parsing library to extract link text, URL structure, and contextual information. Using keyword dictionary matching and natural language processing techniques, it determines whether the link contains core business keywords, such as "product," "list," and "detail." A high semantic score indicates that the page content is more closely aligned with the website's core business focus, thus assigning a higher semantic path score (or a higher score weight) to that page path.

[0030] The system analyzes whether the path contains dynamic parameters, such as token, timestamp, session and other parameters, and identifies whether the page path is introduced by other pages in an iframe nested manner (iframe is an HTML tag that allows another web page to be nested and loaded inside a web page, thereby realizing the function of displaying external web page content on the main page). If there are dynamic parameters, it means that the page may have problems such as high repeated crawling. After removing the dynamic parameters, if the page is already in the crawling queue or has been crawled, the page confidence score will be greatly reduced. If the page is iframe content, usually sub-functional modules such as advertising, payment, and customer service, it means that it lacks complete business logic and is not a core page of the website. Therefore, the structural confidence score of the page path is lowered (or the score weight is lowered) to reduce the priority of the page.

[0031] The system queries the path's historical crawl history. If the page path has been previously embedded in an iframe, it compares the page structure and content to determine whether updates have occurred, thereby estimating the page's update frequency. Furthermore, the system evaluates the stability and accessibility of the page path based on its historical crawl success rate, thereby determining the crawl success rate. Page paths with high update frequencies and high crawl success rates are typically important pages on the website, so these page paths are assigned a higher importance score (or a higher score weight).

[0032] The system sums (or weighted sums) these multiple scores to generate a comprehensive priority score for each page path. The paths to be crawled are then prioritized based on the scores, with those with the highest comprehensive scores being prioritized for the next round of page rendering and task scheduling, ensuring that system resources are prioritized for crawling high-value pages. When the system detects that the number of crawled sub-paths has reached the system's preset upper limit, it halts all concurrent tasks for that website, freeing up computing resources for structural analysis and path scoring for the next website.

[0033] Furthermore, it also includes a step of dynamically adjusting the path score: when a page operation abnormality is detected, the score of the path corresponding to the page in the priority sequence is lowered; the detected abnormal page information is written into the log database, and it is regularly evaluated whether the abnormal page path has returned to normal in subsequent visits, and the path score is automatically corrected according to the recovery situation. For example: if the page abnormality information returns to normal, the comprehensive path score of the page path is increased or the original priority of the page path is restored.

[0034] Furthermore, it also includes a page tracking and comparison step: after the page is rendered, a local DOM tree of the current page is constructed, and DOM structure summary information is generated based on the tag level, number of child nodes, and relative position of each node in the DOM tree; the current DOM structure summary information is compared with the historical DOM structure summary information, and page structure changes are identified based on the similarity matching results. Specifically, if the DOM structure summary information is completely consistent, it is marked as "no change in DOM structure", the access time is recorded, and no repeated crawling is performed; if the DOM structure summary information has node changes or style level adjustments in some areas, it is marked as "minor changes in DOM structure", and the path priority is increased; if new interactive components are added to the DOM structure summary information or the link structure is rearranged, it is marked as "reconstruction of DOM structure", and the path is triggered again. Recursive scheduling.

[0035] After identifying a page structure change, the DOM structure summary, HTML content, screenshots, and page meta-information (e.g., title, URL, and first-discovery time) are stored in a structure version database for subsequent visualization and traceability of structural evolution. The comparison results of all DOM structure summaries serve as feedback input to the scheduling strategy, working in conjunction with the path priority scoring module to intelligently identify page updates and dynamically adjust path priorities.

[0036] Preferably, the DOM structure summary information may be generated by generating an MD5 hash value based on the DOM structure path and the tag frequency of the DOM structure, and using the MD5 hash value as the DOM structure summary information.

[0037] See also Figure 2 Another embodiment of the present invention further provides a browser-rendering-based website map crawler device 200, comprising: a rendering module 201, a path acquisition module 202, a path priority sorting module 203, and a repeated execution module 204. The device 200 is capable of executing the browser-rendering-based website map crawler method of the method embodiment.

[0038] Specifically, the website map crawler device 200 based on browser rendering includes: Rendering module 201, configured to render the current page of the target website through a browser rendering engine; The path acquisition module 202 is configured to simulate user interaction with the page after the page is rendered, monitor DOM change events occurring in the page, and capture the dynamic path of the page triggered by the user interaction; The path priority sorting module 203 is configured to construct an extensible path set for the current page from all captured dynamic paths of the page, perform a comprehensive priority score on each path to be accessed in the extensible path set, dynamically sort the path priorities based on the score results, and select the high-priority path as the access entry for the next round of page rendering; The repetitive execution module 204 is configured to repeat the steps in the above modules until the task termination condition is met.

[0039] It should be noted that the browser-rendering-based website map crawler device 200 provided in this embodiment corresponds to a technical solution that can be used to execute various method embodiments. Its implementation principle and technical effects are similar to those of the method and will not be repeated here.

[0040] See also Figure 3 Another embodiment of the present invention provides an electronic device 300. The electronic device 300 is used to implement the browser-rendering-based website map crawler method described in the method embodiment. The electronic device 300 in the embodiment of the present invention may include, but is not limited to, a PC, a server, a laptop, a smartphone, a tablet computer, and the like. Figure 3 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0041] like Figure 3 As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes to implement the methods of the embodiments of the present invention according to programs stored in read-only memory (ROM) 302 or programs loaded from storage device 308 into random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via bus 305. Input / output (I / O) interface 304 is also connected to bus 305.

[0042] Typically, the following devices may be connected to the I / O interface 304: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0043] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of the present invention is not limited to technical solutions formed by specific combinations of the above-mentioned technical features. It also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents, without departing from the above-mentioned disclosure. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A website map crawler method based on browser rendering, characterized in that: The steps include: Render the current page of the target website through the browser rendering engine; After the page is rendered, simulate the user's interactive behavior on the page, monitor the DOM change events that occur in the page, and capture the dynamic path of the page triggered by the user's interactive behavior; Construct all captured page dynamic paths into an extensible path set for the current page, perform a comprehensive priority score on each path to be accessed in the extensible path set, dynamically sort the path priorities based on the score results, and use the high-priority paths as the access entry for the next round of page rendering; Repeat the above steps until the task termination condition is met.

2. A website map crawler method based on browser rendering according to claim 1, characterized in that: Rendering the current page of the target website by using the browser rendering engine includes: During the page rendering process, the page loading status, number of asynchronous requests, and DOM change frequency are detected, and the waiting time for page loading to complete is automatically adjusted according to the complexity of the page loading.

3. A website map crawler method based on browser rendering according to claim 1, characterized in that: The step of performing a comprehensive priority scoring on each to-be-visited path in the scalable path set includes: Parse the DOM structure of the rendered page, calculate the nesting depth of each path tag in the DOM tree, and assign a higher structural level score to path tags with relatively shallow nesting depth; Identify dynamic paths on a page triggered by simulated user interactions, determine the proportion of dynamic paths on the page, and assign higher dynamic path scores to page paths with a high proportion of dynamic paths. Extract the link text, URL address, and contextual information of each accessed path on the page. Use natural language processing or a rule engine to evaluate the relevance of each accessed path to the core business. Paths with higher relevance are assigned a higher path semantic score. Identify whether the page path contains dynamic parameters, or whether the page is introduced by other pages in an iframe nested manner. If so, lower the structure confidence score of the page path; Identify the page update frequency and historical crawling success rate, and give higher importance scores to page paths with higher page update frequency and historical crawling success rate; The structural hierarchy score, dynamic proportion score, path semantic score, structural confidence score and importance score of each page path are summed to obtain the comprehensive priority score of each page path.

4. A website map crawler method based on browser rendering according to claim 3, characterized in that: It also includes the following steps for dynamic adjustment of path scores: When a page operation anomaly is detected, the score of the path corresponding to the page in the priority sequence is lowered; the detected abnormal page information is written to the log database, and regular evaluation is conducted to see whether the abnormal page path returns to normal in subsequent visits, and the path score is automatically corrected based on the recovery situation.

5. The method for crawling a website map based on browser rendering according to claim 1, characterized in that: It also includes page tracking comparison steps: After the page is rendered, the local DOM tree of the current page is constructed, and the DOM structure summary information is generated based on the tag level, number of child nodes and relative position of each node in the DOM tree; Compare the current DOM structure summary information with the historical DOM structure summary information, and identify page structure changes based on similarity matching results; If the DOM structure summary information is completely consistent, the access time is recorded and no repeated crawling is performed. If the DOM structure summary information has node changes or style level adjustments in some areas, the path priority is increased. If new interactive components are added to the DOM structure summary information or the link structure is rearranged, the path is re-recursively scheduled again. After identifying changes in the page structure, the DOM structure summary information, HTML content, screenshot snapshots and page meta information are stored in the structure version database for visual analysis and tracing of the page structure evolution process.

6. A website map crawler device based on browser rendering, characterized in that: include: A rendering module is configured to render the current page of the target website through a browser rendering engine; The path acquisition module is configured to simulate user interaction with the page after the page is rendered, monitor DOM change events that occur in the page, and capture the dynamic path of the page triggered by user interaction; A path priority sorting module is configured to construct all captured dynamic page paths into an extensible path set for the current page, perform a comprehensive priority score on each path to be accessed in the extensible path set, dynamically sort the path priorities based on the score results, and select the high-priority path as the access entry for the next round of page rendering; The repetitive execution module is configured to repeat the above steps until the task termination condition is met.

7. A website map crawler device based on browser rendering according to claim 6, characterized in that: The rendering module is configured to: During the page rendering process, the page loading status, number of asynchronous requests, and DOM change frequency are detected, and the waiting time for page loading to complete is automatically adjusted according to the complexity of the page loading.

8. The website map crawler device based on browser rendering according to claim 6, characterized in that: The path prioritization module is configured to: Parse the DOM structure of the rendered page, calculate the nesting depth of each path tag in the DOM tree, and assign a higher structural level score to path tags with relatively shallow nesting depth; Identify dynamic paths on a page triggered by simulated user interactions, determine the proportion of dynamic paths on the page, and assign higher dynamic path scores to page paths with a high proportion of dynamic paths. Extract the link text, URL address, and contextual information of each accessed path on the page. Use natural language processing or a rule engine to evaluate the relevance of each accessed path to the core business. Paths with higher relevance are assigned a higher path semantic score. Identify whether the page path contains dynamic parameters, or whether the page is introduced by other pages in an iframe nested manner. If so, lower the structure confidence score of the page path; Identify the page update frequency and historical crawling success rate, and give higher importance scores to page paths with higher page update frequency and historical crawling success rate; The structural hierarchy score, dynamic proportion score, path semantic score, structural confidence score and importance score of each page path are summed to obtain the comprehensive priority score of each page path.

9. The website map crawler device based on browser rendering according to claim 8, characterized in that: It also includes a path score dynamic adjustment device configured to: When a page operation anomaly is detected, the score of the path corresponding to the page in the priority sequence is lowered; the detected abnormal page information is written to the log database, and regular evaluation is conducted to see whether the abnormal page path returns to normal in subsequent visits, and the path score is automatically corrected based on the recovery situation.

10. The website map crawler device based on browser rendering according to claim 6, characterized in that: Also includes a page tracking comparison module, which is configured to: After the page is rendered, the local DOM tree of the current page is constructed, and the DOM structure summary information is generated based on the tag level, number of child nodes and relative position of each node in the DOM tree; Compare the current DOM structure summary information with the historical DOM structure summary information, and identify page structure changes based on similarity matching results; If the DOM structure summary information is completely consistent, the access time is recorded and no repeated crawling is performed. If the DOM structure summary information has node changes or style level adjustments in some areas, the path priority is increased. If new interactive components are added to the DOM structure summary information or the link structure is rearranged, the path is re-recursively scheduled again. After identifying changes in the page structure, the DOM structure summary information, HTML content, screenshot snapshots and page meta information are stored in the structure version database for visual analysis and tracing of the page structure evolution process.

Images