Webpage data extraction method, system, medium and device for rolling container

Abstract

The application provides a webpage data extraction method, system, medium and device for a scrollable container, which comprises the following steps: after the initialization of a browser environment is completed, a search operation acting on a target webpage is received, the target webpage is triggered to be loaded in response to the search operation, and the initial result is waited to be rendered; a full-container scanning and scrolling function is executed, all elements in the target webpage are traversed, at least one scrollable element is screened out from the all elements, and a scrolling operation is performed on the at least one scrollable element to trigger a lazy loading mechanism; candidate selectors are traversed, the number of target elements matched by each candidate selector is counted, and the optimal selector is screened out from the candidate selectors according to the number of matched elements; the number of target elements is monitored in real time until the stable round reaches a preset threshold and / or the number of scrolls reaches a maximum limit to stop scrolling, the data of each target element is extracted by using the optimal selector, and a structured data set is obtained. The method is used for realizing the collection of complete data of a complex nested scrolling page.

Description

Technical Field

[0001] This invention relates to the field of web page data acquisition and processing technology, and in particular to a method, system, medium, and device for extracting web page data from a scrolling container. Background Technology

[0002] Web data collection technology, as a crucial means of acquiring internet information, has been widely applied in fields such as business intelligence, market analysis, and user behavior research. With the rapid development of modern web application technologies, single-page application (SPA) architecture and dynamic content loading technology have become ubiquitous, resulting in richer and more diverse ways of presenting web page data. In the field of data collection, traditional static Hypertext Markup Language (HTML) parsing methods parse the HTML document structure, extracting information based on features such as tag names, class names, and attributes. This method is highly efficient and resource-saving, requiring no browser environment. With the increasing dynamism of web applications, fixed-delay scrolling data collection technology has emerged. This technology simulates user scrolling, pausing for a fixed time after scrolling a fixed distance to wait for data loading before saving and parsing the HTML. This effectively collects paginated data from traditional waterfall layouts and e-commerce products loaded page by page. Optical character recognition (OCR) technology extracts information by converting web page screenshots into images and using OCR engines to recognize text, and can be applied to specific scenarios such as CAPTCHA recognition. These technologies play important roles within their respective scopes, constituting the mainstream technology system in the current field of web data collection.

[0003] However, existing technologies exhibit specific limitations when dealing with modern web applications employing multi-layered nested scrolling container architectures: First, modern web applications (especially short video platforms) use complex multi-layered nested scrolling architectures, with the actual scrolling containers hidden within multiple DOM structures, making them difficult to identify directly. Second, the identification and location of scrolling containers are multi-level and non-intuitive, making it difficult to accurately locate the scrolling context that truly drives lazy loading through preset selectors or empirical rules. Third, pages like Douyin (TikTok) may have multiple selectors that can match search results, but the number of matches varies significantly. Choosing the wrong selector can lead to severely incomplete data collection. Traditional methods require manual trial and error to determine the selector, which is inefficient and prone to errors.

[0004] The above analysis shows that existing technologies cannot reliably, completely, and adaptively capture all data in a scrolling page without disrupting the page's interaction logic. There is an urgent need for a web page data extraction method for scrolling containers to overcome the collection bottleneck in complex nested scrolling scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, medium, and device for extracting web page data from scrolling containers, so as to achieve seamless recognition of multi-layered nested scrolling containers, adaptive triggering and termination of the lazy loading process, and high-accuracy extraction of target data, thereby solving the technical problems of difficult recognition of complex nested scrolling containers, ineffective triggering of lazy loading, and incomplete data collection.

[0006] In a first aspect, embodiments of the present invention provide a method for extracting web page data from a scrolling container, comprising the following steps: After the browser environment is initialized, a search operation is received for the target webpage. In response to the search operation, the target webpage is loaded, and the initial result rendering is completed. A full container scan scrolling function is executed to traverse all elements on the target webpage, select at least one scrollable element, and perform a scrolling operation on the at least one scrollable element, triggering a lazy loading mechanism. Candidate selectors are traversed, and the number of target elements matched by each candidate selector is counted. Based on the number of matched elements, the optimal selector is selected. The change in the number of target elements is monitored in real time until the number of stable rounds reaches a preset threshold and / or the number of scrolls reaches the maximum limit, at which point scrolling stops. The data of each target element is extracted using the optimal selector to obtain a structured data set.

[0007] Secondly, the present invention also provides a web page data extraction system for scrolling containers, which is used to implement the methods described in the above method embodiments.

[0008] Thirdly, embodiments of the present invention also provide a computer-readable storage medium comprising a program that, when run on an electronic device, causes the electronic device to perform any of the possible implementations of the first aspect described above.

[0009] Fourthly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program that can run on the processor, and when the program is executed by the processor, the electronic device performs the method as described in any of the embodiments of the first aspect above.

[0010] Fifthly, embodiments of the present invention also provide a computer program product that, when the program product is run on an electronic device, causes the electronic device to perform any of the possible implementation methods of the first aspect described above.

[0011] The beneficial effects of the webpage data extraction method, apparatus, medium, and device for scrollable containers provided in this invention are as follows: This invention triggers lazy loading without requiring precise identification of the actual scrollable container by scanning all scrollable elements on the page and scrolling them simultaneously; it intelligently determines whether data loading is complete and terminates scrolling by monitoring changes in the number of target elements in real time; and it automatically selects the optimal selector by statistically analyzing the number of matching elements from multiple selectors to ensure accurate data extraction. This invention employs a collaborative strategy of "full container scanning scrolling + real-time monitoring + intelligent selector determination," which solves the problems of difficulty in identifying complex nested scrollable containers, triggering lazy loading, and determining scrolling strategies, thus achieving complete data collection from complex nested scrolling pages. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic flowchart of a webpage data extraction method for a scrolling container provided by an embodiment of the present invention; Figure 2 A schematic diagram illustrating another method for extracting web page data from a scrolling container, provided in an embodiment of the present invention; Figure 3 A schematic diagram of a webpage data extraction system architecture for a scrolling container provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0014] In the description of embodiments of the present invention, the terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be a limitation of the invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present invention, “at least one” and “one or more” refer to one or more (including two). The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0015] References to "one embodiment" or "some embodiments" as used in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized. The term "connection" includes both direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0016] In embodiments of the present invention, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or implementation described as "exemplarily" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or implementations. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0017] Reference Figure 1 The diagram shown illustrates the steps of a webpage extraction method for a scrolling container provided in an embodiment of this specification. It should be understood that this method can be... Figure 3 The various modules of the system shown are executed. This method may include the following steps: S1, after the browser environment is initialized, receives a search operation applied to the target webpage, triggers the loading of the target webpage in response to the search operation, and waits for the initial result to be rendered.

[0018] Step S1 may specifically include the following steps S11 and S12.

[0019] S11: Initialize the browser environment. Initialization: Initialize the browser, set up the browser automation environment, create page instances, define core data structures, and prepare temporary storage resources.

[0020] Specifically, in step S11, setting up the browser automation environment can refer to: initializing a headless browser instance using the Playwright or Selenium framework, and configuring page load timeout, script execution timeout, and network wait strategies. Additionally, in step S11, creating a page instance can refer to creating a new Page object within the browser instance for subsequent navigation, interaction, and DOM manipulation. Furthermore, defining the core data structures can refer to declaring in-memory variables such as a candidate selector list, a scroll log dictionary, an element count list, and a complete dataset dictionary. Preparing temporary storage resources can refer to creating a temporary directory (e.g., tmp_dir) on the local file system for persistently storing the scroll log (scroll_log.json), selector statistics (selector_statistics.json), raw / cleaned data (full_data.json), and error log (err.log).

[0021] For example, one could launch a headless browser (Chromium®) based on Playwright, set page_load_timeout=30000ms, and create a Page instance; or one could load the Firefox® driver based on Selenium WebDriver, configure implicitly_wait=10s, and then obtain the current window's Page context; furthermore, this application could also complete environment initialization by passing headless=True and args=['--no-sandbox', '--disable-setuid-sandbox'] parameters through the BrowserType.launch() interface. This embodiment can obtain a stable, controllable, and reproducible automated browser execution environment based on any of the above methods, providing a unified runtime carrier for subsequent steps.

[0022] S12, Trigger Search and Preload Target Webpage: Receives a search operation applied to the target webpage and triggers the loading of the target webpage.

[0023] In step S12, the search operation can refer to: locating the search input box element in the target webpage, injecting keyword text into it, and triggering a submit event (such as clicking the search button, pressing the Enter key, or calling element.submit()). Triggering the loading of the target webpage can refer to: executing a navigation action (such as page.goto(url)) or, after form submission, causing the page to enter the content route corresponding to the target URL; waiting for the initial result rendering to complete can refer to: waiting for the target element corresponding to the specified CSS selector to be visible in the DOM and have a quantity greater than zero, or waiting for the page to reach the ready state (load / networkidle), supplemented by explicit waiting (such as page.wait_for_selector(selector, state='visible', timeout=5000)).

[0024] For example, visit the Douyin official website homepage, locate the input box with id=search-input, enter Mixue Ice Cream®️, and then click the class=search-btn button to trigger a search; or, send a keyboard Enter event to the [data-testid=search-input] element on the Kuaishou®️ search page to complete the search jump. It is evident that obtaining a page snapshot with a known initial state based on any of the above methods ensures that S3 selector judgment and S4 scroll monitoring have a clear starting point.

[0025] S2 executes the full container scan scrolling function, traverses all elements in the target webpage, selects at least one scrollable element, performs a scrolling operation on at least one scrollable element, and triggers the lazy loading mechanism.

[0026] Specifically, it may include the following steps S21 to S25.

[0027] S21, Configure scrolling parameters, including scroll step size, scroll delay, maximum number of scrolls, and stable scrolling threshold. For example, set the scroll step size to 600px, scroll delay to 2500ms, maximum number of scrolls to 100, stable scrolling threshold to 10, and mouse wheel event deltaY=600.

[0028] In this step, the scroll step size can refer to the number of pixels that are displaced vertically in each scroll operation, used to control the magnitude of a single scroll; the scroll delay can refer to the time interval after each scroll operation when waiting for data loading to complete; the maximum number of scrolls can refer to the upper limit of the number of full container scan scroll operations allowed to prevent infinite loops; the stable round threshold can refer to the counting threshold at which the data loading is determined to be complete and the scrolling is terminated when the number of target elements has not changed after a number of consecutive scrolls.

[0029] S22, Perform a full container scan scrolling operation: Scan all DOM elements on the page, filter out elements with scrolling capabilities, and perform synchronous scrolling operations on all filtered scrollable elements, as well as the page window and the main page.

[0030] In this step, scanning all DOM elements on the page can refer to calling `document.querySelectorAll('*')` or an equivalent API to obtain the entire set of nodes; filtering out elements with scrolling capabilities can refer to checking whether the calculated style of each element's `overflow / overflowY` is `auto` or `scroll`, and that `scrollHeight > clientHeight`, `scrollHeight > 0`, and `clientHeight > 0`; the page window and page body can refer to the `window` object and the `document.body` element, both of which are included in the scrolling range regardless of whether they meet the DOM scrolling property conditions; synchronous scrolling operation can refer to performing scrolling actions sequentially on all filtered scrollable elements, `window`, and `document.body` within the same logical cycle. Optionally, this embodiment can also calculate the distance of each element from the bottom before scrolling, only performing scrolling on elements with a distance greater than 200px, and reserving a 50px safety distance to prevent scrolling out of bounds.

[0031] S23: After each scrolling operation, wait for a preset time to allow the data to load completely, and use the optimal selector to count the number of target elements on the current page.

[0032] In this step, the pre-set waiting time can refer to executing a fixed millisecond delay (such as page.wait_for_timeout(2500)), or waiting for a more robust loading signal such as the page network being idle (networkidle) or the specified element appearing (page.wait_for_selector); using the optimal selector to count the number of target elements on the current page can refer to calling page.query_selector_all(optimal_selector).__len__() or an equivalent method to get the total number of nodes matched by the optimal selector in the current DOM.

[0033] S24, determine whether the number of target elements has increased: if the number has increased, reset the stable round counter; if the number has not increased, increment the stable round counter by 1.

[0034] In this step, whether the number of target elements increases can refer to comparing the number obtained in this statistical analysis with the number obtained in the previous statistical analysis; resetting the stable round counter can refer to assigning the counter variable a value of 0; incrementing the stable round counter by 1 can refer to performing an auto-increment operation on the variable.

[0035] Optionally, this embodiment may also introduce a sliding window mechanism to record the most recent N quantity sequences, and use trend analysis to replace single-point comparison to enhance noise resistance.

[0036] S25. Repeat steps S22-S24 until the number of stable rounds reaches the preset threshold or the number of rolling times reaches the maximum limit.

[0037] In this step, repeating steps S22-S24 can mean encapsulating S22-S24 into a single scroll-monitor loop, which is then repeatedly executed within a while or for structure. Optionally, this embodiment can also limit the number of stable cycles to a preset threshold and the number of scrolls to a maximum, achieving a two-condition termination.

[0038] S3: Iterate through the candidate selectors, count the number of elements matched by each selector, and select the optimal selector based on the number of matched elements.

[0039] In this step, the pre-defined candidate selector list can refer to a set of CSS selectors or XPath expressions pre-configured based on common structural features of the target webpage, such as class names (e.g., .search-result-card), attribute selectors (e.g., [data-e2e="search-result"]), and tag combinations (e.g., aTty6Dqz). Counting the number of target elements matched by each selector on the current page can refer to calling the page instance's querySelectorAll(selector).length method or equivalent API to perform a DOM query on each candidate selector and return the number of matched nodes. Selecting the selector with the most matches as the optimal selector can refer to comparing the number of matches corresponding to each selector and selecting the selector with the maximum value as the optimal selector.

[0040] S4 monitors the changes in the number of target elements in real time until the number of stable cycles reaches a preset threshold and / or the number of scrolls reaches the maximum limit, then stops scrolling. It uses the optimal selector to extract the data of each search result element to obtain a structured data set.

[0041] In this step, data cleaning can also be performed, which can refer to operations such as filtering null values, deduplication, regular expression cleaning of text, and removal of ad placeholders. After that, the cleaned structured data set is serialized into JSON format and written to full_data.json, while the rolling log, selector statistics, and error information are written to their respective local files.

[0042] The following text uses Douyin® data search result collection as a scenario, combined with Figure 2 The implementation process of the technical solution of the present invention is presented exemplarily, and the specific implementation steps are as follows: S41, after completing the environment initialization work such as browser environment setup, data structure definition and temporary resource preparation, triggers the search and preloads all data.

[0043] S411, Browser Environment Setup: Initialize the Playwright headless browser, create Browser and Page instances, set the page load timeout to 30 seconds, simulate the real Chrome browser operating environment, and ensure that the interaction and JavaScript scripts of the Douyin®️ page can be executed normally.

[0044] S412, Data Structure Definition: For example, declare the core data storage objects: ① Candidate selector list: ['.search-result-card', '.aTty6Dqz', '[data-e2e="search-result"]', '[data-e2e="feed-video"]', '[data-e2e="feed-item"]', '[data-e2e="feed-active-video"]']; ② Scroll log dictionary: scroll_log = {'scrolled_records': []}; ③ Element count list: count_records = []; ④ Full dataset dictionary: full_data = {'titles': [], 'authors': [], 'play_counts': [], 'comments': []}; S413, Temporary resource preparation: Create a temporary directory tmp_dir to store intermediate results and debugging files, including: scroll_log.json (scroll log), selector_statistics.json (selector statistics), full_data.json (full dataset), and err.log (error log).

[0045] S42, Search Trigger and Initial Results Loading.

[0046] S421, access the Douyin® official website through the goto method of the Page instance, wait for the page to load, locate the search box element through query_selector, enter the keyword "Mixue Ice Cream®", then locate the search button and click it to trigger the loading of the search results page.

[0047] S422, Initial Results Waiting: Wait 3000ms to ensure the initial 20 search results are fully rendered. Verify successful loading of initial results by locating the .search-result-card element.

[0048] S423, DOM Diagnosis: The DOM diagnostic function is called to analyze the properties of #waterFallScrollContainer and .route-scroll-container on the page. The results show that #waterFallScrollContainer has scrollHeight=2107 and clientHeight=2107, indicating no scroll distance; .route-scroll-container has scrollHeight=0 and clientHeight=0, indicating it is a hidden element. Neither element has effective scrolling capability. The diagnostic information is recorded in err.log.

[0049] S43, Intelligent Optimal Selector This step determines the optimal selector for the Douyin search results for "Mixue Ice Cream" by traversing candidate selectors and counting the number of matches, as detailed below: S431. Candidate Selector Traversal: Traverse the 6 preset candidate selectors, execute the JavaScript script through the Page instance, and count the number of elements matched by each selector in turn; S432. Statistical Results Record: The statistical results show that '.search-result-card' matches 20 elements, '.aTty6Dqz' matches 16 elements, '[data-e2e="search-result"]' matches 0 elements, '[data-e2e="feed-video"]' matches 0 elements, '[data-e2e="feed-item"]' matches 2 elements, and '[data-e2e="feed-active-video"]' matches 1 element; S433. Determining the optimal selector: Select '.search-result-card' with the most matches as the working selector, and save the statistics to selector_statistics.json.

[0050] S44, full container scan scrolling and real-time monitoring.

[0051] This step uses a loop mechanism of "full container scrolling - real-time monitoring - status judgment" to trigger lazy loading and ensure that all data is loaded, as detailed below: S441, parameter configuration: set scroll step size 600px, scroll delay 2500ms, maximum number of scrolls 100, stable scroll threshold 10 times, mouse wheel event deltaY=600; S442, Looping and Monitoring: Initialize last_count=20, stable_rounds=0, enter the loop: a. Full container scan scroll: Execute the full container scan scroll function, traverse all elements on the page, select 1 scrollable element (child-route-container route-scroll-container IhmVuo1S), perform the scroll operation, scroll position from 0→600px, and record the scroll information to scroll_log; b. Event triggering: Dispatch mouse wheel events on the document to enhance the lazy loading triggering effect; c. Delayed waiting: Pause for 2500ms to wait for data loading to complete; d. Counting: Use the optimal selector '.search-result-card' to count the number of elements. The current count is 30, and it is recorded in count_records; e. Status judgment: If the current quantity (30) > the previous quantity (20), reset stable_rounds=0, update last_count=30, and output the log "New content detected! Increased from 20 to 30"; Repeat the above steps. As the number of scrolls increases, the number of elements gradually increases to 150+. When the 35th scroll is reached, the number of elements stabilizes at 152. After 10 consecutive scrolls, no new elements are added, and stable_rounds=10, triggering the termination condition. The log "No new elements after 10 consecutive scrolls, it is considered that the bottom has been reached" is output, and the loop terminates. S443, Result Verification: The final number of loaded elements is 152, which is a significant increase from the initial 20, indicating that the loading was successful.

[0052] S45, Data Extraction and Persistence.

[0053] This step completes the extraction, cleaning, and storage of all data, as detailed below: S451. Full Data Extraction: Execute a custom JavaScript script through the Page instance, use the optimal selector to locate 152 search result elements, and extract data such as video title, author nickname, number of views, and number of comments for each element in turn to form a structured data set; S452. Data Cleaning: Filter out 2 empty header data entries, and finally obtain 150 valid data entries; S453. Persistent storage: Save 150 valid data entries to full_data.json, save the scroll log to scroll_log.json, with no error messages and err.log empty; S454. Resource Release: Close the Playwright browser instance and release resources.

[0054] Another example is that this application can be based on Playwright to launch a Chromium headless browser, access the Kuaishou® search page, enter "Chinese intangible cultural heritage", and wait for the initial results to be rendered; load the candidate selector list ['[data-e2e=search-result]', '.search-result-card', '[data-e2e=feed-video]', '[data-e2e=feed-item]'], count the number of matches as [0, 20, 0, 2], and select .search-result-card as the optimal selector; scan all DOM elements and identify .window, .body, and .route-scroll-container as scrollable bodies; scroll them synchronously and dispatch wheel events with deltaY=600; wait 2500ms after each round of scrolling and count the number of matches for .search-result-card; terminate when there are no new additions for 10 consecutive rounds; finally, extract the title, author, number of views, and details link of 152 video cards, clean them, and save them as full_data.json.

[0055] As can be seen from the above methods, this invention, through a full-container scanning scrolling strategy, can cover all potential scrolling contexts on the page without pre-identifying specific scrolling containers, significantly improving lazy loading trigger coverage. By monitoring changes in the number of target elements in real time and dynamically adjusting scrolling termination conditions, it avoids premature stopping leading to data omissions or excessive scrolling causing performance waste. Through a selector intelligent judgment mechanism, it automatically selects the optimal selector based on actual matching results, overcoming the shortcomings of weak generalization ability and high maintenance cost of manually preset selectors. The three elements work together to form a closed-loop adaptive acquisition framework, ensuring the integrity, accuracy, and robustness of data extraction under complex nested scrolling pages.

[0056] Reference Figure 3 The diagram shown is an architectural schematic of a webpage extraction system 300 for scrolling containers provided in an embodiment of the present invention, including: The target webpage preloading module 310 is used to receive a search operation applied to the target webpage after the browser environment has been initialized, trigger the loading of the target webpage in response to the search operation, and wait for the initial result rendering to be completed. The full container scanning scrolling module 320 is used to execute the full container scanning scrolling function, traverse all elements on the target webpage, select at least one scrollable element, and perform scrolling operations on at least one scrollable element to trigger the lazy loading mechanism. The intelligent selection module 330 is used to traverse candidate selectors, count the number of target elements matched by each candidate selector, and select the optimal selector based on the number of matched elements. The real-time monitoring and data extraction module 340 is used to monitor the changes in the number of target elements in real time until the stable round reaches a preset threshold and / or the number of scrolls reaches the maximum limit and stops scrolling. It uses the optimal selector to extract the data of each target element and obtain a structured data set.

[0057] In one possible embodiment, the target webpage preloading module is further configured to: diagnose the DOM structure of the target webpage and obtain the properties of potential scrolling containers in the target webpage before executing the full container scan scrolling function.

[0058] In one possible embodiment, the method for the full container scanning scrolling module to filter out at least one scrollable element includes: obtaining element attributes, determining whether the element attributes are scrollable attributes, and verifying whether the element scroll height is greater than the visible height. If both are true, the filtered element is determined to be a scrollable element.

[0059] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0060] The above method will be illustrated with specific examples below.

[0061] In other embodiments of the present invention, an electronic device is disclosed that can integrate the above-described system, such as... Figure 4 As shown, the hardware components include: one or more processors 401; memory 402; display 403; one or more application programs (not shown); and one or more computer programs 404. These components can be connected via one or more communication buses 405. The one or more computer programs 404 are stored in the memory 402 and configured to be executed by the one or more processors 401. The one or more computer programs 404 include instructions.

[0062] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in the above-described method embodiments. Specific beneficial effects can be found in the above-described method embodiments.

[0063] The present invention also provides a computer program product that, when executed by a computer, implements the method described in the above-described method embodiments. Specific beneficial effects can be found in the above-described method embodiments.

[0064] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0065] In the various embodiments of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0066] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0067] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A method for extracting webpage data from a scrolling container, characterized in that, include: After the browser environment is initialized, it receives a search operation applied to the target webpage, triggers the loading of the target webpage in response to the search operation, and waits for the initial result to be rendered. The full container scan scrolling function is executed, which traverses all elements on the target webpage, selects at least one scrollable element, performs a scrolling operation on the at least one scrollable element, and triggers the lazy loading mechanism. Iterate through the candidate selectors, count the number of target elements matched by each candidate selector, and select the optimal selector based on the number of matched elements. The system monitors the changes in the number of target elements in real time until the number of stable cycles reaches a preset threshold and / or the number of scrolls reaches the maximum limit, at which point scrolling stops. The system then uses the optimal selector to extract the data for each target element, resulting in a structured dataset.

2. The method according to claim 1, characterized in that, Before executing the full container scan scroll function, the following is also included: Diagnose the DOM structure of the target webpage and obtain the properties of potential scrolling containers in the target webpage.

3. The method according to claim 2, characterized in that, Methods for filtering out at least one scrollable element include: By obtaining the element's attributes, we determine whether the element's attributes are scrollable and verify whether the element's scroll height is greater than the visible height. If both are true, then the filtered elements are determined to be scrollable elements.

4. The method according to claim 3, characterized in that, Performing a scroll operation on at least one scrollable element triggers the lazy loading mechanism, including: For the page window, the main page, and all selected scrollable elements, perform multi-container collaborative scrolling operations simultaneously, triggering the lazy loading mechanism. By monitoring changes in the number of target elements in real time, the system can intelligently determine whether all data has been loaded.

5. The method according to any one of claims 1 to 4, characterized in that, Methods for determining the optimal selector include: The candidate selector with the most matching elements is selected as the optimal selector.

6. A webpage data extraction system for scrolling containers, characterized in that, include: The target webpage preloading module is used to receive a search operation applied to the target webpage after the browser environment has been initialized, trigger the loading of the target webpage in response to the search operation, and wait for the initial result rendering to be completed. The full container scanning scrolling module is used to execute the full container scanning scrolling function, traverse all elements on the target webpage, select at least one scrollable element, and perform scrolling operations on at least one scrollable element, triggering the lazy loading mechanism. The intelligent selector determination module is used to traverse candidate selectors, count the number of target elements matched by each candidate selector, and select the optimal selector based on the number of matched elements. The real-time monitoring and data extraction module is used to monitor the changes in the number of target elements in real time until the stable rounds reach a preset threshold and / or the number of scrolls reaches the maximum limit and stops scrolling. The optimal selector is used to extract the data of each target element to obtain a structured data set.

7. The system according to claim 6, characterized in that, The target webpage preloading module is also used before executing the full container scan scrolling function to: Diagnose the DOM structure of the target webpage and obtain the properties of potential scrolling containers in the target webpage.

8. The system according to claim 7, characterized in that, The methods for the full container scanning scrolling module to filter out at least one scrollable element include: By obtaining the element's attributes, we determine whether the element's attributes are scrollable and verify whether the element's scroll height is greater than the visible height. If both are true, then the filtered elements are determined to be scrollable elements.

9. A computer-readable storage medium storing a program therein, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

10. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a program that can run on the processor, and when the program is executed by the processor, causes the electronic device to perform the method as described in any one of claims 1 to 5.

Images