Webpage data collection method and device, and storage medium

Abstract

The embodiment of the application discloses a webpage data collection method and device and a storage medium, relates to the technical field of computers, and comprises the following steps: acquiring a collection task for webpage data; calling a first collection template to split the collection task, so that the collection task is split into multiple subtasks; determining the task type of each subtask; for each subtask, calling a target collection template in multiple preset collection templates according to the task type of the subtask; and performing a collection operation of the subtask according to the target collection template, wherein a second collection template of the multiple preset collection templates is used for collecting webpage data of a menu page corresponding to each subtask, and a third collection template is used for collecting webpage data of multiple content pages corresponding to each menu page. The above scheme can deeply mine the webpage content of multiple layers of nesting, automatically and efficiently extract data in multiple levels of webpages, and improve the collection efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a web page data acquisition method, apparatus, and storage medium. Background Technology

[0002] Web scraping is a technique for extracting useful information from various web pages on the internet. With the rapid development of the internet, a large amount of valuable information is distributed across various websites. Common Python libraries for web scraping include Requests, Scrapy, Selenium, pyppeteer, Requests-HTML, and RoboBrowser. While Requests is simple and easy to use, it is not suitable for large-scale or high-concurrency data scraping tasks. Scrapy, while powerful, has a high learning curve, complex exception handling, and limited extensibility. Selenium, while capable of simulating real user behavior, is slow and resource-intensive, making it unsuitable for quickly scraping large amounts of data. Pyppeteer supports dynamic content scraping and highly customizable page operations, but it is resource-intensive and has complex asynchronous programming syntax. Requests-HTML combines the functionality of Requests and HTML parsing, but is relatively weak in handling dynamic content. RoboBrowser provides browser-like APIs, but is relatively weak in handling complex pages and dynamic content. Summary of the Invention

[0003] The purpose of this application is to provide a web page data acquisition method, apparatus, and storage medium to solve the technical problem that data acquisition operations cannot be performed efficiently in the prior art.

[0004] To achieve the above objectives, the first aspect of this application provides a web page data collection method, the method comprising:

[0005] Obtain data collection tasks for web pages;

[0006] The first data collection template is invoked to split the data collection task into multiple subtasks;

[0007] Determine the task type for each subtask, where different task types are used to collect different types of web page data;

[0008] For each subtask, the target acquisition template from multiple preset acquisition templates is invoked based on the task type of the subtask;

[0009] The collection operation of the sub-task is performed according to the target collection template. The multiple preset collection templates include a second collection template and a third collection template. The second collection template is used to collect web page data of the menu page corresponding to each sub-task, and the third collection template is used to collect web page data of multiple content pages corresponding to each menu page.

[0010] In the embodiments of this application, determining the task type of each subtask includes: for each subtask, collecting first data results using the first parameters configured in the first collection template, and determining whether the subtask needs to collect data for the menu page based on the first data results; if it is determined that the subtask needs to collect data for the menu page, determining the task type of the subtask as a menu page collection task; if it is determined that the subtask does not need to collect data for the menu page, determining the task type of the subtask as a content page collection task.

[0011] In the embodiments of this application, for each subtask, calling the target collection template among multiple preset collection templates according to the task type of the subtask includes: for each subtask, if the task type of the subtask is a menu page collection task, calling the second collection template; and for each subtask, if the task type of the subtask is a content page collection task, calling the third collection template.

[0012] In the embodiments of this application, the collection operation of the sub-task according to the target collection template includes: for each sub-task, when the third collection template is called, performing a simulated trigger operation according to the third collection template to traverse and access all content pages corresponding to the sub-task; for each content page, collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0013] In the embodiments of this application, the collection operation of the sub-task according to the target collection template includes: when the second collection template is called, performing a simulated trigger operation according to the second collection template to traverse and access all menu pages corresponding to each sub-task; analyzing the format of the page elements of the menu page corresponding to each sub-task according to the second collection template to obtain the target page elements in the menu page corresponding to each sub-task with the target format, each target page element including access links to each content page included in the menu page; calling the third collection template, performing a simulated trigger operation on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0014] In the embodiments of this application, invoking the third collection template and performing simulated triggering operations on each target page element according to the third collection template to access the content page corresponding to each target page element includes: for each subtask, if there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators according to the second collection template, each resource locator corresponding to each target page element of the subtask; generating a target link corresponding to each target page element according to each resource locator; invoking the third collection template and performing simulated triggering operations on each target link according to the third collection template to access the content page corresponding to each target link.

[0015] In the embodiments of this application, for each content page, collecting webpage data included in each content page according to the third collection template and storing the collected data in the corresponding file directory includes: for each content page, searching in the content page for a data format that matches each of the multiple second parameters of the third collection template; collecting webpage data of the matching data format included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, when the second parameter is an array, multiple webpage data of the matching data format included in each content page are collected.

[0016] In embodiments of this application, the method further includes: verifying a third parameter of the second collection template; confirming successful verification of the third parameter if the parameter type of the third parameter is an array and the array length of the third parameter is the same as the number of parameters included in the fourth parameter; wherein the third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure the corresponding resource locator for each target page element corresponding to each subtask; obtaining multiple operation parameters for simulating triggering operations; determining successful verification of multiple operation parameters if the data lengths of the multiple operation parameters are all the same; and stopping the collection operation of web page data if verification fails.

[0017] In embodiments of this application, the method further includes: for each subtask, combining the fifth and sixth parameters configured in the second acquisition template to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; marking the task checkpoint of the subtask after each subtask is completed; in the case of an interruption in the acquisition task, obtaining the interruption duration after the acquisition task is restarted; if the interruption duration is less than or equal to a preset duration threshold, determining the task checkpoint with the closest marked time among all task checkpoints, and executing the subtask corresponding to the task checkpoint with the closest marked time to continue the acquisition task; if the interruption duration is greater than the preset duration threshold, re-executing the acquisition task.

[0018] A second aspect of this application provides a web page data acquisition device, comprising:

[0019] The memory is configured to store instructions;

[0020] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the web page data acquisition method described above.

[0021] A third aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the web page data acquisition method described above.

[0022] The above technical solution designs the specific logic from receiving a webpage collection task to the final execution of data collection, covering the tasks of task breakdown, scheduling, and execution. Using data collection tools, data from multi-level webpages can be extracted automatically and efficiently, and the content of multi-layered nested webpages can be deeply mined, significantly improving the efficiency of data collection. Other features and advantages of this application's embodiments will be described in detail in the subsequent detailed implementation section. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0024] Figure 1 The illustration shows a flowchart of a web page data collection method according to an embodiment of this application;

[0025] Figure 2 The schematic diagram illustrates a web page data acquisition system according to an embodiment of this application;

[0026] Figure 3This illustration schematically shows a diagram of the collection of menu page and content page data according to an embodiment of this application;

[0027] Figure 4 This schematic diagram illustrates a structural block diagram of a web page data acquisition device according to an embodiment of this application;

[0028] Figure 5 The illustration shows a schematic diagram of the structure of a computer device according to an embodiment of the present application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0030] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0031] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0032] Figure 1 The illustration shows a schematic flowchart of a web page data collection method according to an embodiment of this application. Figure 1 As shown in the figure, this application provides a web page data collection method, which may include the following steps.

[0033] S102, Obtain the data collection task for web pages.

[0034] S104, call the first acquisition template to split the acquisition task into multiple sub-tasks.

[0035] S106, determine the task type of each subtask, where different task types are used to collect different types of web page data;

[0036] S108: For each subtask, call the target acquisition template from multiple preset acquisition templates according to the task type of the subtask.

[0037] S110, perform the sub-task collection operation according to the target collection template, wherein the multiple preset collection templates include a second collection template and a third collection template. The second collection template is used to collect web page data of the menu page corresponding to each sub-task, and the third collection template is used to collect web page data of multiple content pages corresponding to each menu page.

[0038] It's understandable that web page data can come from different sources and have different structures, including various types of content such as PDFs, images, tables, documents, and text. The data collection task can be defined by technical personnel, specifying the source of the web pages to be collected, the data types to be collected, and the time frame for data collection. A data collection template is a standardized and systematic data collection tool whose main function is to standardize the data collection process and ensure that the collected data conforms to specific formats and standards. Using a data collection template can reduce data entry errors and improve data quality and consistency. Furthermore, data collection templates can help users quickly and accurately collect the required data, improving work efficiency. Specifically, the first data collection template is used to obtain information about the data collection task. After obtaining the data collection task for web page data, the first data collection template can be used to break down the task into multiple sub-tasks based on the task information. Specifically, each sub-task contains specific web page information that needs to be collected; these web pages share similar web page elements. For example, if the data collection task specifies collecting food data from a certain food website, the task can be divided into sub-tasks, such as collecting food data related to a specific region from that food website. Furthermore, the task type of each subtask can be determined based on the information of each subtask. Different task types are used to collect different types of web page data. Task types include collecting data from menu pages and collecting data from content pages. It can be understood that a content page is a subpage of a menu page; users enter the corresponding content page by clicking links or buttons in the menu page. Content pages typically contain more detailed information, such as article details and product information. Therefore, different dimensions, rules, and methods of data collection are required for menu pages and content pages. The second and third collection templates are customized data collection templates designed for menu pages and content pages, respectively. The second collection template is used to collect web page data from the menu page corresponding to each subtask, and the third collection template is used to collect web page data from multiple content pages corresponding to each menu page. Specifically, the second collection template defines the operations to be performed before and after collecting data from the menu page, and the web page addresses to be collected from the page elements of the menu page for subsequent content page collection. The third collection template defines the operations to be performed before and after collecting data from the content page, and which elements to collect and save. Therefore, for each subtask, the target collection template from multiple preset collection templates can be called according to the task type of the subtask. Then, the collection operation of the sub-task can be executed according to the target collection template.

[0039] The above solution designs the specific logic from receiving a web page collection task to the final execution of data collection, covering the steps of task breakdown, scheduling, and execution. By using data collection tools, data from multi-level web pages can be extracted automatically and efficiently, and the content of multi-layered nested web pages can be deeply mined, which can significantly improve the efficiency of data collection.

[0040] In the embodiments of this application, determining the task type of each subtask includes: for each subtask, collecting first data results using the first parameters configured in the first collection template, and determining whether the subtask needs to collect data for the menu page based on the first data results; if it is determined that the subtask needs to collect data for the menu page, determining the task type of the subtask as a menu page collection task; if it is determined that the subtask does not need to collect data for the menu page, determining the task type of the subtask as a content page collection task.

[0041] It's understandable that the first parameter retrieves the task information for each subtask, i.e., the first data result. If the first data result indicates that the subtask needs to collect data for the menu page, then the task type of the subtask is determined to be a menu page data collection task. If the first data result indicates that the subtask does not need to collect data for the menu page, then the task type of the subtask is determined to be a content page data collection task. Specifically, the content of the first collection template is filled in according to the collection task, and the format is JSON (JavaScript Object Notation), a lightweight data exchange format that is easy for humans to read and write, and also easy for machines to parse and generate. JSON data consists of key-value pairs, where the key must be a string, and the value can be any JSON data type, such as: array, string, number, boolean, null.

[0042] In the embodiments of this application, for each subtask, calling the target collection template among multiple preset collection templates according to the task type of the subtask includes: for each subtask, if the task type of the subtask is a menu page collection task, calling the second collection template; and for each subtask, if the task type of the subtask is a content page collection task, calling the third collection template.

[0043] Specifically, the first data collection template includes multiple parameters used to collect information from different dimensions of the data collection task, and also includes secondary attribute parameters to support the configuration of multiple pieces of information. The second data collection template includes multiple parameters used to collect webpage data from different dimensions of the menu page of the subtask, and also includes secondary attribute parameters to support the configuration of multiple pieces of information. The third data collection template includes multiple parameters used to collect webpage data from different dimensions of the content page under the menu page, and also includes secondary attribute parameters to support the configuration of multiple pieces of information.

[0044] The first data collection template includes TASK_ID, TASK_NAME, TASK_DETAIL, CHECKPOINT_HOUR, and TASK_HEADERS. The secondary attributes under TASK_DETAIL include SUB_TASK_ID, SUB_TASK_NAME, SUB_TASK_TYPE, and SUB_TASK_URL. Specifically, TASK_ID is the ID number of the data collection task, a unique identifier used to distinguish different tasks. TASK_NAME is the task name, used to store the collection results. SUB_TASK_ID distinguishes different subtasks, stores collection results, and is used for endpoint continuation crawling. SUB_TASK_NAME distinguishes different subtask names and stores the data collection results of subtasks. SUB_TASK_TYPE specifies the task type of the subtask, used to determine whether data collection is required for menu pages. SUB_TASK_URL is the web address. CHECKPOINT_HOUR defines the valid time for resuming data collection at breakpoints; setting it to 0 disables the check for resuming data collection at breakpoints. TASK_HEADERS is used to collect webpage request headers and can be filled in optionally. Specifically, the first collection template is shown in Table 1 below.

[0045] Table 1 Parameters defined in the first acquisition template

[0046]

[0047] The second collection template includes MENU_PRE_COLLECT_ACT, MENU_POST_COLLECT_ACT, MENU_TITLE, MENU_NEXT_PAGE, MENU_COLLECT_CSS, and MENU_CUSTOM_URL. The parameters for the second-level attributes under MENU_NEXT_PAGE include css, disabled_attr, and next_symbol. The parameters for the second-level attributes under MENU_COLLECT_CSS include col_css and col_attr. Specifically, MENU_PRE_COLLECT_ACT defines the simulated trigger operation performed before the menu page collection. MENU_POST_COLLECT_ACT defines the simulated trigger operation performed after the menu page collection. MENU_TITLE is used to distinguish the menu page to which the content page belongs and to save the collection results for the menu page. css is used to collect the CSS selector syntax of the page element corresponding to the next page button in the menu page. disabled_attr is used to collect the disabled attribute name of the next page button in the menu page, used to determine whether there is a next page in the current menu page. `next_symbol` is used to collect the element text name of the "next page" button on the menu page, used to determine if there is a next page. `col_css` is used to collect the CSS selector syntax of page elements in the menu page. `col_attr` is used to collect the attributes that need to be extracted from the page elements of the menu page, for subsequent data collection on content pages. `MENU_CUSTOM_URL` is used in conjunction with `MENU_COLLECT_CSS`. When `MENU_COLLECT_CSS` collects more than one element in the menu page that is a webpage link, it uses `MENU_CUSTOM_URL` to construct a custom URL for subsequent content page collection. Specifically, the second collection template is shown in Table 2 below.

[0048]

[0049] The third collection template includes CONT_PRE_COLLECT_ACT, CONT_POST_COLLECT_ACT, CONT_TITLE, CONT_TBL_COLLECT_CSS, CONT_PIC_COLLECT_CSS, CONT_TXT_COLLECT_CSS, CONT_FILE_COLLECT_RE, and CONT_VERIFY. CONT_PRE_COLLECT_ACT defines the simulated trigger operation performed before content page collection. CONT_POST_COLLECT_ACT defines the simulated trigger operation performed after content page collection. CONT_TITLE is used to distinguish the menu page to which the content page belongs and to save the collection results of the content page. CONT_TBL_COLLECT_CSS is used to collect table data from the content page. The parameters of the second-level attributes under CONT_TBL_COLLECT_CSS include css, attr, extra_columns, file_name, and columns. The css attribute is used to collect the CSS selector syntax for each row of elements in the table. The attr attribute is used to collect the attributes to be extracted from the elements within the table. `extra_columns` is used to collect additional columns that need to be added to the table, and is used to uniformly write table data into the collection results. `file_name` is used to save the file name of the table, and is used to distinguish and unify multiple tables on the content page. `columns` is used to collect the names of the table columns. `CONT_PIC_COLLECT_CSS` is used to collect image data from the content page. The parameters of the second-level properties under `CONT_PIC_COLLECT_CSS` include `css`, `css_attr`, `file_name_css`, and `file_name_attr`. `css` is used to collect the CSS selector syntax for image elements. `css_attr` is used to collect the attributes where image links are located. `file_name_css` defines the CSS selector syntax for image name elements, and `file_name_attr` is used to collect the attributes where image names are located. `CONT_TXT_COLLECT_CSS` is used to collect text data from the content page. The parameters of the second-level properties under `CONT_TXT_COLLECT_CSS` include `css`, `css_attr`, and `file_name`. `css` is used to collect the CSS selector syntax for text content elements. The `css_attr` attribute is used to collect the attributes of the text content. `file_name` defines the filename for saving the text content. `CONT_FILE_COLLECT_RE` filters downloadable file content that matches the regular expression and downloads it, supporting configuration for multiple simultaneous downloads. The secondary attributes under `CONT_VERIFY` include parameters for `object`, `css`, and `attr`.This object configures which data collection items to validate. The CSS parameter is used for CSS validation; if left blank, it defaults to validating the number of elements on the webpage; if specified, it validates based on the attributes of the CSS elements. The attr parameter is used to validate the attributes of the elements, such as verifying whether the number of rows collected matches the number displayed on the webpage. Specifically, the third data collection template is shown in Table 3 below.

[0050]

[0051]

[0052] In the embodiments of this application, the collection operation of the sub-task according to the target collection template includes: when the second collection template is called, performing a simulated trigger operation according to the second collection template to traverse and access all menu pages corresponding to each sub-task; analyzing the format of the page elements of the menu page corresponding to each sub-task according to the second collection template to obtain the target page elements in the menu page corresponding to each sub-task with the target format, each target page element including access links to each content page included in the menu page; calling the third collection template, performing a simulated trigger operation on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0053] It's understandable that when the subtask type is a menu page collection task, the second collection template is called. Then, the menu page corresponding to the subtask is initialized, and the menu page link is accessed in the main thread. After receiving the webpage data of the menu page, a simulated trigger operation is performed on the menu page according to the parameter MENU_PRE_COLLECT_ACT in the second collection template, thereby traversing and accessing all menu pages corresponding to each subtask. The simulated trigger operation refers to simulating the user's automatic operations of inputting, clicking, and scrolling on the webpage. Since a menu page may have multiple pages, it is presented on the webpage with page numbers and previous / next page buttons on the side of the page. Therefore, during collection, each menu page needs to be accessed traversally. Based on the parameter MENU_NEXT_PAGE, the next page button of the webpage is analyzed, the current page number is determined, and it is possible to continue to the next page. It can be understood that the target format is the link format, and the target page elements include access links to each content page included in the menu page. The menu page contains links to multiple content pages that require menu access. Clicking each link on the menu page redirects to the content page to be collected. Therefore, the `MENU_COLLECT_CSS` parameter determines which page elements the menu page needs to retrieve content page information from. Further, a third-party data collection template is invoked, and a sub-thread receives the access links from the menu page collected by the main thread. Each sub-thread initializes the corresponding content page based on one access link. Within the sub-thread, content page links are accessed, and upon receiving webpage data, a simulated trigger operation is performed on the page based on `CONT_PRE_COLLECT_ACT`. Data is collected from the webpages included in each content page according to the parameters set in the third-party template, and the collected data is stored in the corresponding file directory. This solution utilizes multi-threading technology, allowing the data collection tool to process multiple webpages simultaneously, significantly improving the speed and efficiency of data collection without impacting the web server.

[0054] In the embodiments of this application, invoking the third collection template and performing simulated triggering operations on each target page element according to the third collection template to access the content page corresponding to each target page element includes: for each subtask, if there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators according to the second collection template, each resource locator corresponding to each target page element of the subtask; generating a target link corresponding to each target page element according to each resource locator; invoking the third collection template and performing simulated triggering operations on each target link according to the third collection template to access the content page corresponding to each target link.

[0055] As can be understood, a resource locator (RLO) is a standard address format used to locate resources on the internet. When there are multiple access links to content pages within a collection menu page, the page element corresponding to each link may not contain the complete URL information of the content page. Therefore, when there are multiple target page elements on the menu page corresponding to a subtask, a resource locator is generated for each target page element based on the MENU_CUSTOM_URL parameter of the second collection template. Further, the generated resource locator and the webpage attributes obtained from the MENU_COLLECT_CSS parameter are used to generate target links corresponding to each content page. This solution generates target links by combining resource locators with the webpage attributes of the menu page. These target links contain complete URL information, facilitating subsequent access and collection of content pages.

[0056] In the embodiments of this application, for each content page, collecting webpage data included in each content page according to the third collection template and storing the collected data in the corresponding file directory includes: for each content page, searching in the content page for a data format that matches each of the multiple second parameters of the third collection template; collecting webpage data of the matching data format included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, when the second parameter is an array, multiple webpage data of the matching data format included in each content page are collected.

[0057] Specifically, the second parameter refers to the parameters set by the third data collection template for collecting web page data in different data formats. These different data formats include text data, table data, image data, and files. For table data collection, the information provided by the second parameter CONT_TBL_COLLECT_CSS can be used to collect tables. If the second parameter is an array, multiple tables from a single content page can be collected and stored in the directory "Collection Date / TASK_ID-Table Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to their corresponding file_names. For text data collection, the information provided by the second parameter CONT_TXT_COLLECT_CSS can be used to collect text data from a content page. If the parameter is an array, multiple text files from a single page can be collected and stored in the directory "Collection Date / TASK_ID-Text Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to their corresponding file_names. For image collection, images on a webpage can be collected based on the information provided by the second parameter, CONT_PIC_COLLECT_CSS. If the parameter is an array, multiple images from a single page can be collected and stored in the directory "Collection Date / TASK_ID-Image Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to their corresponding file_names. For file collection, the content of files on a webpage can be collected based on the information provided by the second parameter, CONT_FILE_COLLECT_RE. Multiple files on a single page can be collected using regular expressions and stored in the directory "Collection Date / TASK_ID-File Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to their filenames.

[0058] In the embodiments of this application, the collection operation of the sub-task according to the target collection template includes: for each sub-task, when the third collection template is called, a simulated trigger operation is performed according to the third collection template to traverse and access all content pages corresponding to the sub-task; for each content page, the web page data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory. It can be understood that when the task type of the sub-task is to collect web page data of the content page, there is no need to consider the collection of menu page data, and the data collection method of the content page by calling the third collection template is the same as the aforementioned scheme, and will not be described again.

[0059] In embodiments of this application, the method further includes: verifying a third parameter of the second collection template; confirming successful verification of the third parameter if the parameter type of the third parameter is an array and the array length of the third parameter is the same as the number of parameters included in the fourth parameter; wherein the third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure the corresponding resource locator for each target page element corresponding to each subtask; obtaining multiple operation parameters for simulating triggering operations; determining successful verification of multiple operation parameters if the data lengths of the multiple operation parameters are all the same; and stopping the collection operation of web page data if verification fails.

[0060] It is understandable that the parameters of the template frame need to be validated before using the second data collection template. Specifically, after receiving the task template, since the web page data collection task template is fixed, only the interdependent relationships between parameters need to be validated. Check item 1: The third parameter MENU_COLLECT_CSS is an array type, and the array length must be equal to the number of parameters appearing in the fourth parameter MENU_CUSTOM_URL. The fourth parameter MENU_CUSTOM_URL must conform to the specified format. Check item 2: For parameters of simulated trigger operations, multiple operation parameters are also included. Operation parameters include parameters corresponding to input, click, scroll, and other operation behaviors. For example, the start date and end date of data collection from the website. If there are multiple "input" operations, and the value type of the "input" operations is an array, it is necessary to check whether the array lengths are consistent. If they are inconsistent, the arrangement of the web page data collection task needs to be readjusted. For example, if we collect historical temperature data from a public website, collecting data for nearly 20 years for each prefecture-level city (assuming a total of 293), and the website provides a button to filter by date, then the parameters for the "Input" operation are shown in Table 4 below.

[0061] Table 4 Operating Parameters

[0062]

[0063] After parameter validation, the task is split into smaller subtasks. The splitting logic is as follows: first, split according to the length of the array in `TASK_DETAIL`, and then split according to the `values` of the "input" operation parameters in the webpage operation module. For example, if the length of the `TASK_DETAIL` parameter array is `m`, and the length of the `values` of the "input" operation parameters is `n`, then the data collection task will be split into `m×n` subtasks. If each subtask is a menu page, and each menu page has 10 content pages, then ultimately `m×n×10` webpages will need to be collected.

[0064] In embodiments of this application, the method further includes: for each subtask, combining the fifth and sixth parameters configured in the second acquisition template to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; marking the task checkpoint of the subtask after each subtask is completed; in the case of an interruption in the acquisition task, obtaining the interruption duration after the acquisition task is restarted; if the interruption duration is less than or equal to a preset duration threshold, determining the task checkpoint with the closest marked time among all task checkpoints, and executing the subtask corresponding to the task checkpoint with the closest marked time to continue the acquisition task; if the interruption duration is greater than the preset duration threshold, re-executing the acquisition task.

[0065] It's understandable that task checkpoints can be created based on the combination of the fifth parameter TASK_ID and SUB_TASK_ID configured in the second acquisition template. Upon completion of each subtask, the task checkpoint of the subtask is marked, and the marking time is recorded. If the acquisition task is interrupted, the interruption duration after restarting is obtained. The interruption duration refers to the duration during which acquisition work is not performed after the acquisition task is interrupted. The preset duration threshold is a duration pre-set by technicians, which can be 3 hours, 24 hours, etc. If the interruption duration is less than or equal to the preset duration threshold, the task checkpoint with the closest marked time among all task checkpoints is determined, and the subtask corresponding to the task checkpoint with the closest marked time is executed to continue the acquisition task; if the interruption duration is greater than the preset duration threshold, the acquisition task is re-executed. This avoids unnecessary duplicate acquisition and effectively improves execution efficiency.

[0066] refer to Figure 2In one embodiment of this application, a web page data acquisition system is provided. After configuring a first acquisition template, a second acquisition template, and a third acquisition template for a web page acquisition task, the second and third acquisition templates are set in the data acquisition module of the web page data acquisition system. Then, the parameters in the templates are validated. Specifically, the third parameter of the second acquisition template is validated. If the parameter type of the third parameter is an array, and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, the third parameter is confirmed to be validated successfully. Multiple operation parameters for simulating trigger operations are obtained. If the data lengths of the multiple operation parameters are all the same, the multiple operation parameters are confirmed to be validated successfully; if validation fails, the web page data acquisition operation is stopped. Further, the first acquisition template is invoked to split the acquisition task according to the information of the acquisition task, so that the acquisition task is split into multiple sub-tasks before being issued. Further, the first data result is acquired using the first parameters configured in the first acquisition template. If the first data result indicates that the sub-task needs to acquire data for a menu page, then the task type of the sub-task is determined to be a menu page acquisition task. After invoking the second collection template for the menu page collection task, the webpage data of all menu pages in the subtask is collected, including content page access links within the menu pages. Further, the third collection template is invoked to collect the webpage data of the content pages. If the first data result indicates that the subtask does not require collection of data for menu pages, then the task type of the subtask is determined to be a content page collection task. In this case, the third collection template can be directly invoked to collect the webpage data of the content pages. The specific collection rules and process are not detailed here.

[0067] Data acquisition module: Reference Figure 3 The data acquisition module includes logic for collecting data from the menu page and logic for collecting data from the content page, specifically including the following steps.

[0068] Initialize menu page: Initialize the menu page corresponding to the subtask, access the menu page link in the main thread, and after receiving the web page data of the menu page, simulate the trigger operation of the menu page according to the parameter MENU_PRE_COLLECT_ACT in the second collection template, thereby traversing and accessing all menu pages corresponding to each subtask.

[0069] Analyzing the menu page: Since the menu page may have multiple pages, it is presented on the webpage as page numbers and previous / next page buttons on the side of the page. Therefore, during data collection, it is necessary to iterate through each menu page and, based on the parameter MENU_NEXT_PAGE, analyze the next page button of the webpage, determine the current page number, and whether it is possible to continue to the next page.

[0070] The menu page contains links to multiple content pages that need to be accessed. Clicking each link on the menu page redirects to the content page to be collected. Therefore, the `MENU_COLLECT_CSS` parameter determines which page elements need to be retrieved for the menu page's content page information. After collection, the `MENU_POST_COLLECT_ACT` parameter is used to simulate triggering operations on the page.

[0071] Content Page Generation: When multiple target page elements exist on the menu page corresponding to a subtask, a resource locator corresponding to each target page element is generated based on the MENU_CUSTOM_URL parameter of the second data collection template. Further, the generated resource locator and the webpage attributes obtained from the MENU_COLLECT_CSS parameter are used to generate target links corresponding to each content page.

[0072] Content page initialization: The third-party data collection template is invoked. Sub-threads receive access links from the menu page collected by the main thread. Each sub-thread initializes the corresponding content page based on an access link. Within a sub-thread, a content page link is accessed, and upon receiving webpage data, a simulated trigger operation is performed on the page based on CONT_PRE_COLLECT_ACT.

[0073] Table collection: Tables can be collected based on the information provided by the second parameter CONT_TBL_COLLECT_CSS. If the second parameter is an array type, multiple tables in a content page can be collected and stored in the directory "Collection date / TASK_ID-Table type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to the corresponding file_name.

[0074] Text collection: Based on the information provided by the second parameter CONT_TXT_COLLECT_CSS, text data of the content page can be collected. If the parameter is an array type, multiple texts in a page can be collected and stored in the directory "collection date / TASK_ID-text type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to the corresponding file_name.

[0075] Image capture: Based on the information provided by the second parameter CONT_PIC_COLLECT_CSS, images on a webpage can be captured. If the parameter is an array type, multiple images on a page can be captured and stored in the directory "capture date / TASK_ID-image type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to their corresponding file_name.

[0076] File collection: Based on the information provided by the second parameter CONT_FILE_COLLECT_RE, the file content of the webpage can be collected. Multiple files in a page are matched using regularization and collected, and stored in the directory "collection date / TASK_ID-file type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" according to the file name.

[0077] Webpage Operation Module: The parameters MENU_PRE_COLLECT_ACT / CONT_PRE_COLLECT_ACT / MENU_POST_COLLECT_ACT / CONT_POST_COLLECT_ACT, set through the second and third data collection templates, are used to simulate triggering operations during the data collection process from the menu page and content page. Specifically, the operation content includes the details shown in Table 5 below.

[0078] Table 5. Operation content of simulated triggering operation

[0079]

[0080] The ability to interact with page buttons before, during, and after data collection allows the tool to automate complex interactive processes such as clicking buttons and filling out forms, further enhancing the flexibility and functionality of data collection.

[0081] The data cleaning module processes collected titles, text, and tables, removing ambiguous characters and line breaks to ensure data consistency and readability. It also standardizes text formatting and handles special characters to improve data quality. The goal of improving data quality is to enhance the accuracy and reliability of data analysis, thereby better supporting business decisions and application needs.

[0082] The data validation module verifies the number of text entries, table rows, downloaded documents, and images collected to ensure consistency between the collected data and the content displayed on the webpage, thereby improving data integrity and reliability. This process helps identify and correct omissions and errors during data collection.

[0083] Data storage module: Data is stored according to the structure: Collection Date / TASK_ID-Collection Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME. This hierarchical storage method not only helps to clearly organize and manage data, but also improves data traceability and consistency through multiple ID identification and classification, making it easier for subsequent data verification and power outage resume data collection, providing a solid foundation for efficient data management and collection.

[0084] The breakpoint resume acquisition module: To efficiently utilize resources and ensure the continuity and integrity of data acquisition, a specific checkpoint is created based on the CHECKPOINT_HOUR parameter at the start of each subtask, according to the combination of TASK_ID and SUB_TASK_ID. Once a subtask completes its data acquisition and successfully passes data verification, the system marks the corresponding checkpoint. This means that when the task is executed again within the subsequent CHECKPOINT_HOUR time window, the completed portion will be intelligently skipped, avoiding unnecessary duplicate acquisition and effectively improving execution efficiency. For acquisition interruptions caused by abnormal situations (such as runtime errors or data verification failures), the module employs a detailed logging mechanism to record the reasons for the interruption of incomplete tasks, including specific error messages and details of failed verification. This not only facilitates rapid problem location and repair but also ensures the transparency and controllability of task management, providing valuable information support for subsequent data recovery and strategy adjustments. In summary, this mechanism, through refined management and intelligent scheduling, greatly enhances the stability and flexibility of the data acquisition process.

[0085] The cookie management module is responsible for periodically checking the status of cookies related to webpage interactions to ensure the continuity and validity of data collection. When an expired cookie is detected, this module automatically triggers the process of re-acquiring the cookie, seamlessly maintaining a valid session connection with the website server and ensuring that subsequent data requests can proceed normally. This mechanism avoids the problem of lost login status or interrupted data collection due to expired cookies, improving the stability and success rate of data collection tasks.

[0086] The webpage data collection template design in the above technical solution addresses specific data collection tasks by devising efficient methods for collecting webpage data. This includes defining collection rules, selecting appropriate collection tools and targets, and ensuring the accuracy and efficiency of the collection process. The data collection logic design covers the entire process from receiving a webpage data collection task to final storage. This includes task splitting, scheduling, execution, and storage, ensuring a smooth and efficient collection flow. The functional module design addresses this by individually designing each functional module within the overall logic, including data collection, data cleaning, data verification, and data storage. Each module's design ensures independence and scalability, thereby improving the overall system performance and reliability. This solution not only handles webpages from different sources and structures but also enables efficient data extraction, breakpoint resume collection, cookie management, multi-threaded concurrent collection, and fixed-level storage. Data cleaning and verification further enhance data quality, ensuring character conversion and data integrity. These features make data collection more comprehensive, flexible, and efficient, while also improving its stability and reliability. In practical applications, data acquisition tools can significantly improve the efficiency, accuracy, and stability of data collection, providing objective data support for subsequent data management, data analysis, and management decisions.

[0087] In the embodiments of this application, the web page data acquisition method can utilize AIGC (AI-Generated Content) technologies, such as natural language processing and computer vision, to overcome the challenges of dynamic web pages that are difficult for traditional web crawlers to handle. For example, it can automatically identify page elements (such as images and tables) rendered by JavaScript based on deep learning models, or understand the logical structure of web page content through semantic analysis to generate adaptive XPath rules. Such technologies significantly improve the ability to bypass dynamic loading and anti-crawling mechanisms (such as CAPTCHAs), for example, by using generative adversarial networks (GANs) to simulate human clicking behavior to bypass CAPTCHAs. AIGC can transform unstructured web page data into structured information. For example, it can use language models such as BERT to perform entity recognition, sentiment analysis, or knowledge graph construction on the acquired text, achieving an upgrade from "raw data" to "semantic data."

[0088] Figure 1 This is a flowchart illustrating a webpage data collection method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0089] Figure 4 A schematic block diagram of a controller according to an embodiment of this application is shown. Figure 4 As shown in the figure, this application provides a controller that may include:

[0090] The memory is configured to store instructions;

[0091] The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the aforementioned methods for controlling the boom.

[0092] Specifically, in this embodiment of the application, the processor can be configured as follows:

[0093] Obtain data collection tasks for web pages;

[0094] The first data collection template is invoked to split the data collection task into multiple subtasks;

[0095] Determine the task type for each subtask, where different task types are used to collect different types of web page data;

[0096] For each subtask, the target acquisition template from multiple preset acquisition templates is invoked based on the task type of the subtask;

[0097] The collection operation of the sub-task is performed according to the target collection template. The multiple preset collection templates include a second collection template and a third collection template. The second collection template is used to collect web page data of the menu page corresponding to each sub-task, and the third collection template is used to collect web page data of multiple content pages corresponding to each menu page.

[0098] In embodiments of this application, the processor may also be configured to:

[0099] Determining the task type for each subtask includes: for each subtask, collecting first data results using the first parameters configured in the first collection template, and determining whether the subtask needs to collect data for the menu page based on the first data results; if it is determined that the subtask needs to collect data for the menu page, the task type of the subtask is determined to be a menu page collection task; if it is determined that the subtask does not need to collect data for the menu page, the task type of the subtask is determined to be a content page collection task.

[0100] In embodiments of this application, the processor may also be configured to:

[0101] For each subtask, the target collection template from multiple preset collection templates is called according to the task type of the subtask, including: for each subtask, if the task type of the subtask is a menu page collection task, the second collection template is called; for each subtask, if the task type of the subtask is a content page collection task, the third collection template is called.

[0102] In embodiments of this application, the processor may also be configured to:

[0103] The collection operations of the sub-tasks according to the target collection template include: for each sub-task, when the third collection template is called, a simulated trigger operation is performed according to the third collection template to traverse and access all content pages corresponding to the sub-task; for each content page, the web page data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory.

[0104] In embodiments of this application, the processor may also be configured to:

[0105] The collection operations of the sub-tasks according to the target collection template include: when the second collection template is called, performing simulated trigger operations according to the second collection template to traverse and access all menu pages corresponding to each sub-task; analyzing the format of the page elements of the menu page corresponding to each sub-task according to the second collection template to obtain the target page elements in the target format in the menu page corresponding to each sub-task, and each target page element includes access links to each content page included in the menu page; calling the third collection template, performing simulated trigger operations on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0106] In embodiments of this application, the processor may also be configured to:

[0107] The process involves invoking a third data collection template and simulating trigger operations for each target page element to access the content page corresponding to each target page element. This includes: for each subtask, if there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators based on the second data collection template, with each resource locator corresponding to each target page element of the subtask; generating target links corresponding to each target page element based on each resource locator; and invoking the third data collection template and simulating trigger operations for each target link to access the content page corresponding to each target link.

[0108] In embodiments of this application, the processor may also be configured to:

[0109] For each content page, the webpage data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory. This includes: for each content page, searching for data formats that match each of the multiple second parameters in the third collection template; collecting webpage data of the matching data formats included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, if the second parameter is an array, multiple webpage data of the matching data formats included in each content page are collected.

[0110] In embodiments of this application, the processor may also be configured to:

[0111] The third parameter of the second data collection template is validated. If the third parameter is an array and its array length is the same as the number of parameters included in the fourth parameter, the validation of the third parameter is confirmed as successful. The third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure the corresponding resource locator for each target page element corresponding to each subtask. Multiple operation parameters used to simulate the trigger operation are obtained. If the data lengths of multiple operation parameters are all the same, the validation of multiple operation parameters is confirmed as successful. If the validation fails, the data collection operation on the web page is stopped.

[0112] In embodiments of this application, the processor may also be configured to:

[0113] For each subtask, the fifth and sixth parameters configured in the second acquisition template are combined to create a corresponding task checkpoint. The fifth parameter defines the task details of the acquisition task, and the sixth parameter defines the identifier of each subtask included in the acquisition task. Upon completion of each subtask, the task checkpoint of the subtask is marked. If the acquisition task is interrupted, the interruption duration after restarting is obtained. If the interruption duration is less than or equal to a preset duration threshold, the task checkpoint with the closest marked time among all task checkpoints is determined, and the subtask corresponding to the task checkpoint with the closest marked time is executed to continue the acquisition task. If the interruption duration exceeds the preset duration threshold, the acquisition task is re-executed.

[0114] This application also provides a machine-readable storage medium storing instructions that cause a machine to execute the web page data collection method described above.

[0115] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores data for web page data acquisition methods. The network interface A02 communicates with external terminals via a network connection. When the processor A01 executes the computer program B02, it implements a web page data acquisition method.

[0116] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0125] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for collecting web page data, characterized in that, The method includes: Obtain data collection tasks for web pages; The first acquisition template is invoked to split the acquisition task into multiple sub-tasks; Determine the task type for each subtask, where different task types are used to collect different types of web page data; For each subtask, the target acquisition template from multiple preset acquisition templates is invoked according to the task type of the subtask; The subtask collection operation is performed according to the target collection template, wherein the plurality of preset collection templates include a second collection template and a third collection template. The second collection template is used to collect web page data of the menu page corresponding to each subtask, and the third collection template is used to collect web page data of the plurality of content pages corresponding to each menu page. The step of performing the sub-task collection operation according to the target collection template includes: When the second collection template is invoked, a simulated trigger operation is performed according to the second collection template to traverse and access all menu pages corresponding to each subtask; The format of the page elements of the menu page corresponding to each subtask is analyzed according to the second collection template to obtain the target page elements in the menu page corresponding to each subtask with the target format. Each target page element includes an access link to each content page included in the menu page. The third collection template is invoked, and a simulated trigger operation is performed on each target page element according to the third collection template to access the content page corresponding to each target page element; The webpage data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory.

2. The webpage data collection method according to claim 1, characterized in that, The task type for determining each subtask includes: For each subtask, the first data result is collected using the first parameter configured in the first collection template, and the subtask is determined to be whether it needs to collect data for the menu page based on the first data result. If it is determined that the subtask needs to collect data from the menu page, then the task type of the subtask is determined to be a menu page data collection task. If it is determined that the subtask does not require data collection from the menu page, then the task type of the subtask is determined to be a content page data collection task.

3. The webpage data acquisition method according to claim 2, characterized in that, The step of calling the target acquisition template from multiple preset acquisition templates according to the task type of the subtask for each subtask includes: For each subtask, if the task type of the subtask is a menu page collection task, the second collection template is invoked; For each subtask, if the task type of the subtask is the content page collection task, the third collection template is invoked.

4. The webpage data acquisition method according to claim 3, characterized in that, The step of performing the sub-task collection operation according to the target collection template includes: For each subtask, when the third collection template is invoked, a simulated trigger operation is performed according to the third collection template to traverse and access all content pages corresponding to the subtask; For each content page, the webpage data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory.

5. The webpage data acquisition method according to claim 1, characterized in that, The process involves invoking the third data collection template and performing simulated triggering operations on each target page element to access the content page corresponding to each target page element, including: For each subtask, if there are multiple target page elements in the menu page corresponding to the subtask, multiple resource locators are generated according to the second collection template, and each resource locator corresponds to each target page element corresponding to the subtask. Generate target links corresponding to each target page element based on each resource locator; The third collection template is invoked, and a simulated trigger operation is performed on each target link according to the third collection template to access the content page corresponding to each target link.

6. The webpage data acquisition method according to claim 1, characterized in that, For each content page, the webpage data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory, including: For each content page, search the content page for a data format that matches each of the multiple second parameters of the third collection template; Based on each second parameter, the web page data of the matching data format included in each content page is collected, and the collected data is stored in the corresponding file directory; Specifically, for each second parameter, if the second parameter is an array, multiple webpage data in the matching data format are collected for each content page.

7. The webpage data acquisition method according to claim 5, characterized in that, The method further includes: The third parameter of the second collection template is verified. If the parameter type of the third parameter is an array and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, the verification of the third parameter is confirmed to be successful. The third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure the corresponding resource locator for each target page element corresponding to each subtask. Obtain multiple operation parameters used to simulate the trigger operation; If the data lengths of all the multiple operation parameters are the same, it is determined that the multiple operation parameters have been successfully verified. If a verification fails, stop collecting webpage data.

8. The webpage data collection method according to claim 1, characterized in that, The method further includes: For each subtask, the fifth and sixth parameters configured in the second acquisition template are combined to create a task checkpoint corresponding to the subtask. The fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task. Upon completion of each subtask, a task checkpoint for that subtask is marked. In the event that the data acquisition task is interrupted, the interruption duration after the data acquisition task is restarted is obtained; If the interruption duration is less than or equal to a preset duration threshold, determine the task checkpoint with the closest marked time among all task checkpoints, and execute the subtask corresponding to the task checkpoint with the closest marked time to continue the acquisition task. If the interruption duration exceeds the preset duration threshold, the data acquisition task will be re-executed.

9. A web page data acquisition device, characterized in that, include: The memory is configured to store instructions; A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the web page data acquisition method according to any one of claims 1 to 8.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the web page data acquisition method according to any one of claims 1 to 8.

Images