Webpage data collection method and device and storage medium

Abstract

The embodiment of the invention discloses a webpage data collection method and device and a storage medium, and relates to the technical field of computers, the method comprises the following steps: obtaining a collection task for webpage data; calling a first collection template to split the collection task so as to split the collection task into a plurality of sub-tasks; determining the task type of each subtask; for each subtask, calling a target acquisition template in a plurality of preset acquisition templates according to the task type of the subtask; the collection operation of the subtasks is executed according to the target collection template, a second collection template of the multiple preset collection templates is used for collecting webpage data of the menu page corresponding to each subtask, and a third collection template of the multiple preset collection templates is used for collecting webpage data of multiple content pages corresponding to each menu page. According to the scheme, the content of the multi-layer nested webpage can be deeply mined, the data in the multi-level webpage can be automatically and efficiently extracted, and the collection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a web page data collection method, device, and storage medium. Background Art

[0002] Web scraping is a technique for extracting useful information from various web pages on the internet. With the rapid development of the internet, a vast amount of valuable information is distributed across various websites. Among Python-based web scraping tools, common Python libraries include Requests, Scrapy, Selenium, Pyppeteer, Requests-HTML, and RoboBrowser. While the Requests library is simple and easy to use, it's not suitable for large-scale or highly concurrent data scraping tasks. While powerful, the Scrapy library requires a high level of familiarity, has relatively complex exception handling, and has limited scalability. While the Selenium library can simulate real user behavior, it's slow and resource-intensive, making it unsuitable for rapidly scraping large amounts of data. While the Pyppeteer library supports dynamic content scraping and highly customizable page operations, it consumes a lot of resources and has complex asynchronous programming syntax. The Requests-HTML library, while combining the functionality of Requests and HTML parsing, is relatively weak at handling dynamic content. While the RoboBrowser library provides a browser-like API, it's relatively weak at handling complex pages and dynamic content. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a web page data collection method, device and storage medium to solve the technical problem in the prior art that data collection operations cannot be performed efficiently.

[0004] To achieve the above objectives, the present application provides a webpage data collection method in a first aspect, the method comprising:

[0005] Get the collection task for web page data;

[0006] Invoking the first acquisition template to split the acquisition task into multiple subtasks;

[0007] Determine the task type of each subtask, where different task types are used to collect different types of web page data;

[0008] For each subtask, a target acquisition template from a plurality of preset acquisition templates is called according to the task type of the subtask;

[0009] The collection operation of the subtask is performed according to the target collection template, wherein the multiple preset collection templates include a second collection template and a third collection template, the second collection template is used to collect the web page data of the menu page corresponding to each subtask, and the third collection template is used to collect the web page data of multiple content pages corresponding to each menu page.

[0010] In an embodiment of the present application, determining the task type of each subtask includes: for each subtask, using the first parameter configured in the first acquisition template to acquire the first data result, and determining whether the subtask needs to be acquired for the menu page based on the first data result; when it is determined that the subtask needs to be acquired for the menu page, determining the task type of the subtask to be a menu page acquisition task; when it is determined that the subtask does not need to be acquired for the menu page, determining the task type of the subtask to be a content page acquisition task.

[0011] In an embodiment of the present application, for each subtask, calling a target acquisition template from multiple preset acquisition templates according to the task type of the subtask includes: for each subtask, when the task type of the subtask is a menu page acquisition task, calling the second acquisition template; for each subtask, when the task type of the subtask is a content page acquisition task, calling the third acquisition template.

[0012] In an embodiment of the present application, the acquisition operation of executing the subtask according to the target acquisition template includes: for each subtask, when calling the third acquisition template, performing a simulated trigger operation according to the third acquisition template to traverse and access all content pages corresponding to the subtask; for each content page, collecting the web page data included in each content page according to the third acquisition template, and storing the collected data in the corresponding file directory.

[0013] In an embodiment of the present application, executing the collection operation of the subtask according to the target collection template includes: when calling the second collection template, performing a simulated trigger operation according to the second collection template to traverse and access all menu pages corresponding to each subtask; analyzing the format of the page elements of the menu page corresponding to each subtask according to the second collection template to obtain the target page elements in the target format in the menu page corresponding to each subtask, and each target page element includes an access link to each content page included in the menu page; calling the third collection template, and performing a simulated trigger operation on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0014] In an embodiment of the present application, calling the third acquisition template, and performing a simulated trigger operation on each target page element according to the third acquisition template to access the content page corresponding to each target page element includes: for each subtask, when there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators according to the second acquisition template, each resource locator corresponding to each target page element corresponding to the subtask; generating a target link corresponding to each target page element according to each resource locator; calling the third acquisition template, and performing a simulated trigger operation on each target link according to the third acquisition template to access the content page corresponding to each target link.

[0015] In an embodiment of the present application, for each content page, the web page data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory, including: for each content page, searching the content page for a data format that matches each second parameter of the multiple second parameters of the third collection template; collecting the web page data of the matching data format included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, when the second parameter is an array type, multiple web page data of the matching data format included in each content page are collected.

[0016] In an embodiment of the present application, the method also includes: verifying the third parameter of the second acquisition template, and confirming that the third parameter verification is successful when the parameter type of the third parameter is an array type and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, wherein the third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure a corresponding resource locator for each target page element corresponding to each subtask; obtaining multiple operation parameters for simulating trigger operations; determining that the multiple operation parameters are successfully verified when the data lengths of the multiple operation parameters are the same; and stopping the collection operation of the web page data in the event of a verification failure.

[0017] In an embodiment of the present application, the method also includes: for each subtask, combining the fifth parameter and the sixth parameter configured according to the second acquisition template to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; when each subtask is executed, marking the task checkpoint of the subtask; when the acquisition task is interrupted, obtaining the interruption duration for restarting the acquisition task after the interruption; when the interruption duration is less than or equal to the preset duration threshold, determining the task checkpoint with the closest marking time among all task checkpoints, and executing the subtask corresponding to the task checkpoint with the closest marking time to continue the acquisition task; when the interruption duration is greater than the preset duration threshold, re-executing the acquisition task.

[0018] A second aspect of the present application provides a webpage data collection device, comprising:

[0019] a memory configured to store instructions;

[0020] The processor is configured to call the instructions from the memory and implement the web page data collection method when executing the instructions.

[0021] A third aspect of the present application provides a machine-readable storage medium having stored thereon instructions for enabling a machine to execute the web page data collection method described above.

[0022] Through the above technical solution, a specific logic is designed from receiving a webpage collection task to the final execution of data collection, covering the splitting, scheduling, and execution of the collection task. Utilizing the data collection tool, data from multi-level webpages can be automatically and efficiently extracted, and the content of multi-layered nested webpages can be deeply mined, significantly improving the efficiency of data collection. Other features and advantages of the embodiments of this application will be described in detail in the subsequent detailed description of the implementation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0024] Figure 1 The following schematically shows a flow chart of a web page data collection method according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a web page data collection system according to an embodiment of the present application is schematically shown;

[0026] Figure 3Schematic diagram of collecting menu page and content page data according to an embodiment of the present application;

[0027] Figure 4 The following schematically shows a structural block diagram of a web page data collection device according to an embodiment of the present application;

[0028] Figure 5 The schematic diagram shows the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0032] Figure 1 The following schematically shows a flow chart of a web page data collection method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a web page data collection method, which may include the following steps.

[0033] S102: Obtaining a collection task for web page data.

[0034] S104: Invoke the first collection template to split the collection task into multiple subtasks.

[0035] S106, determining a task type for each subtask, wherein different task types are used to collect different types of web page data;

[0036] S108 : For each subtask, call a target acquisition template from a plurality of preset acquisition templates according to the task type of the subtask.

[0037] S110, performing the collection operation of the subtask according to the target collection template, wherein the multiple preset collection templates include a second collection template and a third collection template, the second collection template is used to collect the web page data of the menu page corresponding to each subtask, and the third collection template is used to collect the web page data of multiple content pages corresponding to each menu page.

[0038] It is understood that web data can come from various sources and structures, including PDFs, images, tables, files, text, and other content types. Collection tasks can be defined by technical personnel, specifying the source of the web pages to be collected, the type of data to be collected, and the time of data collection. A collection template is a standardized and systematic data collection tool. Its primary function is to standardize the data collection process and ensure that the collected data conforms to specific formats and standards. Using collection templates can reduce data entry errors and improve data quality and consistency. Furthermore, collection templates can help users quickly and accurately collect the required data, improving work efficiency. Specifically, the first collection template is used to obtain collection task information. After obtaining a collection task for web data, the first collection template can be used to split the collection task into multiple subtasks based on the collection task information. Specifically, each subtask contains specific web page information to be collected, and the common feature of these web pages is that they have similar web page elements. For example, if the collection task specifies collecting food data from a specific food website, the collection task can be divided into subtasks, such as collecting food data related to a specific region on that food website. Furthermore, the task type of each subtask can be determined based on the information in each subtask. Different task types are used to collect different types of web page data. Task types include collecting data for menu pages and collecting data for content pages. It can be understood that a content page is a subpage of a menu page. Users access the corresponding content page by clicking a link or button on the menu page. Content pages typically contain more detailed information, such as article details and product information. Therefore, data collection for menu pages and content pages requires different dimensions, rules, and methods. The second and third collection templates are customized data collection templates designed for menu pages and content pages, respectively. The second collection template is used to collect web page data from the menu page corresponding to each subtask, while the third collection template is used to collect web page data from multiple content pages corresponding to each menu page. Specifically, the second collection template defines the operations required before and after collecting menu pages and the web page addresses collected from page elements on menu pages to facilitate subsequent content page collection. The third collection template defines the operations required before and after collecting content pages and the elements collected and saved. For each subtask, a target collection template from multiple preset collection templates can be called based on the subtask's task type. Then, the collection operation of the subtask can be performed according to the target collection template.

[0039] The above solution designs the specific logic from receiving a web page collection task to the final execution of data collection, covering the splitting, scheduling, execution and other links of the collection task. Using data collection tools, data from multi-level web pages can be automatically and efficiently extracted, and multi-layer nested web page content can be deeply mined, which can significantly improve the efficiency of data collection.

[0040] In an embodiment of the present application, determining the task type of each subtask includes: for each subtask, using the first parameter configured in the first acquisition template to acquire the first data result, and determining whether the subtask needs to be acquired for the menu page based on the first data result; when it is determined that the subtask needs to be acquired for the menu page, determining the task type of the subtask to be a menu page acquisition task; when it is determined that the subtask does not need to be acquired for the menu page, determining the task type of the subtask to be a content page acquisition task.

[0041] It can be understood that the first parameter can obtain the task information of each subtask, that is, the first data result. If the first data result is that the subtask needs to collect data for the menu page, then the task type of the subtask is determined to be a menu page collection task. If the first data result is that the subtask does not need to collect data for the menu page, then the task type of the subtask is determined to be a content page collection task. Specifically, the content of the first collection template is filled in according to the collection task, and the format is json (JavaScript Object Notation), which is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate. JSON data consists of key-value pairs, the key must be a string, and the value can be any JSON data type, such as: array, string, number, Boolean value, null.

[0042] In an embodiment of the present application, for each subtask, calling a target acquisition template from multiple preset acquisition templates according to the task type of the subtask includes: for each subtask, when the task type of the subtask is a menu page acquisition task, calling the second acquisition template; for each subtask, when the task type of the subtask is a content page acquisition task, calling the third acquisition template.

[0043] Specifically, the first collection template includes multiple parameters for collecting information of different dimensions for the collection task, and is also provided with parameters of secondary attributes to support the configuration of multiple pieces of information. The second collection template page includes multiple parameters for collecting web page data of different dimensions for the menu page of the subtask, and is also provided with parameters of secondary attributes to support the configuration of multiple pieces of information. The third collection template page includes multiple parameters for collecting web page data of different dimensions for the content page under the menu page, and is also provided with parameters of secondary attributes to support the configuration of multiple pieces of information.

[0044] Among them, the first collection template includes TASK_ID, TASK_NAME, TASK_DETAIL, CHECKPOINT_HOUR, and TASK_HEADERS. The parameters of the secondary attributes under TASK_DETAIL include SUB_TASK_ID, SUB_TASK_NAME, SUB_TASK_TYPE, and SUB_TASK_URL. Specifically, TASK_ID is the ID number of the collection task, which is the unique mark of the collection task and is used to distinguish different collection tasks. TASK_NAME is the task name of the collection task, which is used to save the collection results of the collection task. SUB_TASK_ID is used to distinguish different subtasks, to save the collection results, and to enable the endpoint resume crawling function. SUB_TASK_NAME is used to distinguish different subtask names and to save the data collection results of the subtasks. SUB_TASK_TYPE is used to collect the task type of the subtask, and to distinguish whether collection is required for the menu page. SUB_TASK_URL is the web page address. CHECKPOINT_HOUR is used to define the effective time of breakpoint resume. If it is set to 0, the breakpoint resume check will not be performed. TASK_HEADERS is used to collect web page request headers and can be optionally filled in. Specifically, the first collection template is shown in Table 1 below.

[0045] Table 1 Parameters defined in the first acquisition template

[0046]

[0047] The second collection template includes MENU_PRE_COLLECT_ACT, MENU_POST_COLLECT_ACT, MENU_TITLE, MENU_NEXT_PAGE, MENU_COLLECT_CSS, and MENU_CUSTOM_URL. The parameters of the secondary attributes under MENU_NEXT_PAGE include css, disabled_attr, and next_symbol. The parameters of the secondary attributes under MENU_COLLECT_CSS include col_css and col_attr. Specifically, MENU_PRE_COLLECT_ACT defines the simulation trigger operation performed before the menu page is collected. MENU_POST_COLLECT_ACT defines the simulation trigger operation performed after the menu page is collected. MENU_TITLE is used to distinguish the menu page to which the content page belongs, and is used to save the collection results of the menu page. css is used to collect the CSS selector syntax of the page element corresponding to the next page button in the menu page. disabled_attr is used to collect the disabled attribute name of the next page button of the menu page, and is used to determine whether there is a next page on the current menu page. Next_symbol is used to collect the element text name of the next page button on the menu page, which is used to determine whether there is a next page. Col_css is used to collect the CSS selector syntax of the page elements in the menu page. Col_attr is used to collect the attributes that need to be extracted from the page elements of the menu page for subsequent content page data collection. MENU_CUSTOM_URL is used for coordination. When MENU_COLLECT_CSS collects page elements on the menu page that are web page links and the number of elements exceeds 1, MENU_CUSTOM_URL is used to build a custom URL for subsequent content page collection. Specifically, the second collection template is shown in Table 2 below.

[0048]

[0049] The third collection template includes CONT_PRE_COLLECT_ACT, CONT_POST_COLLECT_ACT, CONT_TITLE, CONT_TBL_COLLECT_CSS, CONT_PIC_COLLECT_CSS, CONT_TXT_COLLECT_CSS, CONT_FILE_COLLECT_RE, and CONT_VERIFY. CONT_PRE_COLLECT_ACT defines the simulation trigger operation performed before the content page is collected. CONT_POST_COLLECT_ACT defines the simulation trigger operation performed after the content page is collected. CONT_TITLE is used to distinguish the menu page to which the content page belongs, and is used to save the collection results of the content page. CONT_TBL_COLLECT_CSS is used to collect the table data of the content page. Among them, the parameters of the secondary attributes under CONT_TBL_COLLECT_CSS include css, attr, extra_columns, file_name, and columns. The css is used to collect the CSS selector syntax of each row element in the table. The attr is used to collect the attributes that need to be extracted from the elements in the table. extra_columns is used to collect additional columns that need to be added to a table and to uniformly write table data into the collection results. file_name is used to save the table file name and is used to distinguish and unify multiple tables on a content page. columns is used to collect the names of the table columns. CONT_PIC_COLLECT_CSS is used to collect image data from content pages. The secondary attribute parameters under CONT_PIC_COLLECT_CSS include css, css_attr, file_name_css, and file_name_attr. css is used to collect the CSS selector syntax of image elements. css_attr is used to collect the location attributes of image links. file_name_css defines the CSS selector syntax of image name elements, and file_name_attr is used to collect the location attributes of image names. CONT_TXT_COLLECT_CSS is used to collect text data from content pages. The secondary attribute parameters under CONT_TXT_COLLECT_CSS include css, css_attr, and file_name. css is used to collect the CSS selector syntax of text content elements. The css_attr parameter is used to collect the attributes of the text content. The file_name parameter defines the file name where the text content is saved. The CONT_FILE_COLLECT_RE parameter filters and downloads downloadable files that match a regular expression, supporting the configuration of multiple simultaneous downloads. The secondary attributes under CONT_VERIFY include object, css, and attr.This object configures which collection items are validated. The css field is used for CSS validation. If left blank, the default is to validate the number of elements on the webpage. If filled in, validation is performed based on the attributes of the CSS element. The attr field is used to validate the attributes of the element, for example, to verify that the number of collected rows matches the number displayed on the webpage. Specifically, the third collection template is shown in Table 3 below.

[0050]

[0051]

[0052] In an embodiment of the present application, executing the collection operation of the subtask according to the target collection template includes: when calling the second collection template, performing a simulated trigger operation according to the second collection template to traverse and access all menu pages corresponding to each subtask; analyzing the format of the page elements of the menu page corresponding to each subtask according to the second collection template to obtain the target page elements in the target format in the menu page corresponding to each subtask, and each target page element includes an access link to each content page included in the menu page; calling the third collection template, and performing a simulated trigger operation on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0053] It can be understood that when the task type of the subtask is a menu page collection task, the second collection template is called. Then, the menu page corresponding to the subtask is initialized, and the menu page link is accessed in the main thread. After receiving the web page data of the menu page, the menu page is simulated triggered according to the parameter MENU_PRE_COLLECT_ACT in the second collection template, thereby traversing and accessing all menu pages corresponding to each subtask. The simulated trigger operation refers to the automatic operation of simulating a user to input, click, and scroll on the web page. Since a menu page may have multiple pages, the presentation on the web page is to have a page number display and previous and next page buttons on the side of the page. Therefore, when collecting, it is necessary to traverse and access each menu page. According to the parameter MENU_NEXT_PAGE, the next page button of the web page is analyzed, and the current page number and whether it is possible to continue to the next page are determined. It can be understood that the target format is the format of the link, and the target page element includes an access link to each content page included in the menu page. The menu page contains multiple content page links that require a menu. The presentation on the menu page is to jump by clicking on each access link, and the page that jumps to is the content page that needs to be collected. Therefore, according to the MENU_COLLECT_CSS parameter, it is determined from which page elements the menu page needs to obtain the content page information. Further, the third collection template is called, and the child thread receives the access links in the menu page collected by the main thread, and each child thread initializes the corresponding content page according to an access link. The content page link is accessed in the child thread, and after receiving the web page data, a simulated trigger operation is performed on the page according to CONT_PRE_COLLECT_ACT. The web page data included in each content page is collected according to the parameters set by the third collection template, and the collected data is stored in the corresponding file directory. The above scheme uses multi-threading technology, and the data collection tool can process multiple web pages at the same time, significantly improving the speed and efficiency of data collection without affecting the web server.

[0054] In an embodiment of the present application, calling the third acquisition template, and performing a simulated trigger operation on each target page element according to the third acquisition template to access the content page corresponding to each target page element includes: for each subtask, when there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators according to the second acquisition template, each resource locator corresponding to each target page element corresponding to the subtask; generating a target link corresponding to each target page element according to each resource locator; calling the third acquisition template, and performing a simulated trigger operation on each target link according to the third acquisition template to access the content page corresponding to each target link.

[0055] It can be understood that a resource locator refers to a standard address format used to locate resources on the Internet. When there are multiple access links to the content page in the collection menu page, the page element corresponding to the access link may not contain the complete URL information of the content page. Therefore, when there are multiple target page elements in the menu page corresponding to the subtask, a resource locator corresponding to each target page element is generated according to the MENU_CUSTOM_URL parameter of the second collection template. Furthermore, the generated resource locator is combined with the web page attributes obtained by the MENU_COLLECT_CSS parameter to generate a target link corresponding to each content page. The above scheme generates a target link by combining the resource locator with the web page attributes of the menu page. The target link contains complete URL information, which is convenient for subsequent access and collection of content pages.

[0056] In an embodiment of the present application, for each content page, the web page data included in each content page is collected according to the third collection template, and the collected data is stored in the corresponding file directory, including: for each content page, searching the content page for a data format that matches each second parameter of the multiple second parameters of the third collection template; collecting the web page data of the matching data format included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, when the second parameter is an array type, multiple web page data of the matching data format included in each content page are collected.

[0057] Specifically, the second parameter refers to the parameter set by the third collection template for collecting web page data in different data formats. Web page data in different data formats include text data, table data, image data, files, etc. For table collection, you can collect tables based on the information provided by the second parameter CONT_TBL_COLLECT_CSS. If the second parameter is an array type, you can collect multiple tables in a content page and store them in the "collection date / TASK_ID-table type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name. For text collection, you can collect text data from a content page based on the information provided by the second parameter CONT_TXT_COLLECT_CSS. If the parameter is an array type, you can collect multiple texts in a page and store them in the "collection date / TASK_ID-text type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name. For image collection, you can collect images from web pages based on the information provided by the second parameter CONT_PIC_COLLECT_CSS. If the parameter is an array type, you can collect multiple images from a page and store them in the "collection date / TASK_ID-image type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name. For file collection, you can collect file content from web pages based on the information provided by the second parameter CONT_FILE_COLLECT_RE. Collect multiple files from a page by regularizing and matching, and store them in the "collection date / TASK_ID-file type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the collected file names.

[0058] In an embodiment of the present application, executing a subtask's collection operation according to a target collection template includes: for each subtask, when calling a third collection template, performing a simulated trigger operation according to the third collection template to traverse and access all content pages corresponding to the subtask; for each content page, collecting the webpage data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory. It will be understood that if the subtask's task type is to collect webpage data for content pages, there is no need to consider the collection of menu page data. The method for collecting data from content pages by calling the third collection template is the same as the previously described scheme and will not be further described.

[0059] In an embodiment of the present application, the method also includes: verifying the third parameter of the second acquisition template, and confirming that the third parameter verification is successful when the parameter type of the third parameter is an array type and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, wherein the third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure a corresponding resource locator for each target page element corresponding to each subtask; obtaining multiple operation parameters for simulating trigger operations; determining that the multiple operation parameters are successfully verified when the data lengths of the multiple operation parameters are the same; and stopping the collection operation of the web page data in the event of a verification failure.

[0060] It can be understood that before using the second acquisition template for acquisition, the parameters of the template frame need to be verified. Specifically, after receiving the task template, since the web page acquisition task template is fixed, it is only necessary to verify the interdependent relationship between the parameters. Check item 1: The third parameter MENU_COLLECT_CSS is of array type. The array length must be equal to the number of parameters appearing in the fourth parameter MENU_CUSTOM_URL, and the fourth parameter MENU_CUSTOM_URL must be in the prescribed format. Check item 2: The parameters for the simulated trigger operation also include multiple operation parameters. The operation parameters include parameters corresponding to input, click, scroll and other operation behaviors. For example, the start date and end date of the input for data collection on the website. If there are multiple "input" operations, and the types of the values of the "input" operations are all arrays, it is necessary to check whether the array lengths are consistent. If they are inconsistent, the arrangement of the web page acquisition task needs to be readjusted. For example, suppose we collect historical temperature data from a public website, collect data for the past 20 years for each prefecture-level city (assuming there are 293 in total), and the website provides a button for filtering by date. The parameters for the "Input" operation are shown in Table 4 below.

[0061] Table 4 Operating parameters

[0062]

[0063] After parameter verification, the task is split. The specific splitting logic is as follows: first, split according to the length of the TASK_DETAIL array, and then split according to the values of the "input" operation parameter in the webpage operation module. For example, if the TASK_DETAIL parameter array length is m, and the "input" operation parameter value length is n, then the collection task will be split into m×n subtasks. If each subtask type is a menu page, and each menu page has 10 content pages, the final m×n×10 web pages will need to be collected.

[0064] In an embodiment of the present application, the method also includes: for each subtask, combining the fifth parameter and the sixth parameter configured according to the second acquisition template to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; when each subtask is executed, marking the task checkpoint of the subtask; when the acquisition task is interrupted, obtaining the interruption duration for restarting the acquisition task after the interruption; when the interruption duration is less than or equal to the preset duration threshold, determining the task checkpoint with the closest marking time among all task checkpoints, and executing the subtask corresponding to the task checkpoint with the closest marking time to continue the acquisition task; when the interruption duration is greater than the preset duration threshold, re-executing the acquisition task.

[0065] It can be understood that a task checkpoint can be created based on the combination of the fifth parameters TASK_ID and SUB_TASK_ID configured in the second acquisition template. When each subtask is completed, the task checkpoint of the subtask is marked, and the marking time is recorded during the marking. In the case of an interruption in the acquisition task, the interruption duration of the restart after the acquisition task is interrupted is obtained. The interruption duration refers to the duration of time during which the acquisition work is not performed after the acquisition task is interrupted. The preset duration threshold is the duration pre-set by the technician, which can be 3 hours, 24 hours, etc. When the interruption duration is less than or equal to the preset duration threshold, the task checkpoint with the closest marking time among all task checkpoints is determined, and the subtask corresponding to the task checkpoint with the closest marking time is executed to continue the acquisition task; when the interruption duration is greater than the preset duration threshold, the acquisition task is re-executed. Unnecessary repeated acquisition is avoided, and execution efficiency is effectively improved.

[0066] refer to Figure 2In one embodiment of the present application, a webpage data collection system is provided. After configuring a first, second, and third collection templates for a webpage collection task, the system installs the second and third collection templates in a data collection module of the webpage data collection system. The system then begins verifying the parameters in the templates. Specifically, the third parameter of the second collection template is verified. If the parameter type of the third parameter is an array and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, the third parameter verification is confirmed to be successful. Multiple operation parameters for simulating a trigger operation are obtained. If the data lengths of the multiple operation parameters are the same, the multiple operation parameters verification is determined to be successful. If the verification fails, the webpage data collection operation is terminated. Furthermore, the first collection template is invoked to split the collection task based on the collection task information, thereby splitting the collection task into multiple subtasks and issuing them. Furthermore, a first data result is collected using the first parameter configured in the first collection template. If the first data result indicates that the subtask requires collection of a menu page, the task type of the subtask is determined to be a menu page collection task. After invoking the second collection template for the menu page collection task, web page data for all menu pages in the subtask is collected, including links to content pages within the menu pages. Furthermore, the third collection template is invoked to collect web page data for content pages. If the first data result indicates that the subtask does not require collection of menu pages, the subtask's task type is determined to be a content page collection task. In this case, the third collection template can be directly invoked to collect web page data for content pages. The specific collection rules and process are not further detailed.

[0067] Data acquisition module: reference Figure 3 The data collection module includes the logic for collecting menu pages and the logic for collecting content pages, and specifically includes the following steps.

[0068] Initialize the menu page: Initialize the menu page corresponding to the subtask, access the menu page link in the main thread, and after receiving the web page data of the menu page, simulate the trigger operation on the menu page according to the parameter MENU_PRE_COLLECT_ACT in the second acquisition template, so as to traverse and access all menu pages corresponding to each subtask.

[0069] Analyze the menu page: Since the menu page may have multiple pages, the presentation on the web page is to display the page number and the previous and next page buttons on the side of the page. Therefore, when collecting data, it is necessary to traverse and access each menu page. According to the parameter MENU_NEXT_PAGE, analyze the next page button of the web page, determine the current page number and whether it is possible to continue to the next page.

[0070] Collecting menu pages: Menu pages contain multiple links to content pages that require a menu. Clicking each access link on the menu page redirects you to the content page you want to collect. Therefore, the MENU_COLLECT_CSS parameter determines which page elements the menu page needs to retrieve content page information from. After collection is complete, the MENU_POST_COLLECT_ACT parameter is used to simulate triggering actions on the page.

[0071] Generate content pages: If the menu page corresponding to the subtask has multiple target page elements, generate a resource locator corresponding to each target page element based on the MENU_CUSTOM_URL parameter of the second collection template. Furthermore, combine the generated resource locator with the webpage attributes obtained by the MENU_COLLECT_CSS parameter to generate a target link corresponding to each content page.

[0072] Initialize the content page: Call the third collection template. The child thread receives the access links from the menu page collected by the main thread. Each child thread initializes the corresponding content page based on the access link. The child thread accesses the content page link and simulates triggering the page according to CONT_PRE_COLLECT_ACT after receiving the web page data.

[0073] Table collection: You can collect tables based on the information provided by the second parameter CONT_TBL_COLLECT_CSS. If the second parameter is an array type, you can collect multiple tables in a content page and store them in the "collection date / TASK_ID-table type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name.

[0074] Text collection: The text data of the content page can be collected according to the information provided by the second parameter CONT_TXT_COLLECT_CSS. If the parameter is an array type, multiple texts in a page can be collected and stored in the "collection date / TASK_ID-text type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name.

[0075] Image collection: You can collect images from a web page based on the information provided by the second parameter CONT_PIC_COLLECT_CSS. If the parameter is an array type, you can collect multiple images from a page and store them in the "Collection Date / TASK_ID-Image Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the corresponding file_name.

[0076] File collection: Based on the information provided by the second parameter CONT_FILE_COLLECT_RE, the file content of the web page can be collected. Multiple files on a page are collected through regularization matching and stored in the "collection date / TASK_ID-file type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME" directory according to the collected file name.

[0077] Web page operation module: The parameters MENU_PRE_COLLECT_ACT / CONT_PRE_COLLECT_ACT / MENU_POST_COLLECT_ACT / CONT_POST_COLLECT_ACT set by the second collection template and the third collection template are used to simulate trigger operations in the process of collecting information for menu pages and content pages. Specifically, the operation content includes the following table 5.

[0078] Table 5 Operation contents of simulated trigger operation

[0079]

[0080] By operating page buttons before, during, and after data collection, this defined web page operation method enables the tool to automatically handle complex interactive processes, such as clicking buttons and filling out forms, further enhancing the flexibility and functionality of data collection.

[0081] Data Cleansing: This module processes collected titles, text, and tables, removing ambiguous characters and line breaks to ensure data consistency and readability. It also standardizes text formats and handles special characters to improve data quality. The goal of improving data quality is to increase the accuracy and reliability of data analysis, thereby better supporting business decisions and application needs.

[0082] Data Verification Module: Verifies the amount of collected text, table rows, downloaded documents, and images to ensure that the collected data is consistent with the content displayed on the webpage, thereby improving data integrity and reliability. This process helps to identify and correct omissions and errors in the data collection process.

[0083] Data storage module: Data is stored according to the structure: Collection Date / TASK_ID-Collection Type-TASK_NAME / SUB_TASK_ID-SUB_TASK_NAME. This hierarchical storage method not only helps to clearly organize and manage data, but also improves data traceability and consistency through multiple ID identification and classification, facilitating subsequent data verification and continued data collection after power outages, providing a solid foundation for efficient data management and collection.

[0084] Breakpoint Resume Module: To efficiently utilize resources and ensure the continuity and integrity of data collection, a checkpoint is created at the start of each subtask based on the CHECKPOINT_HOUR parameter, combining the TASK_ID and SUB_TASK_ID. Once a subtask completes its data collection and successfully passes data verification, the system marks the corresponding checkpoint as confirmed. This means that when the task is executed again within the following CHECKPOINT_HOUR window, the completed portion is intelligently skipped, avoiding unnecessary duplicate collection and effectively improving execution efficiency. For collection interruptions caused by abnormal conditions (such as runtime errors or data verification failures), the module implements a detailed logging mechanism that records the reasons for the interruption, including specific error messages and details of the verification failure. This not only facilitates rapid problem identification and remediation, but also ensures transparency and controllability in task management, providing valuable information for subsequent data recovery and policy adjustments. Overall, this mechanism significantly enhances the stability and flexibility of the data collection process through refined management and intelligent scheduling.

[0085] The cookie management module is responsible for regularly checking the status of cookies associated with web page interactions to ensure the continuity and effectiveness of data collection. When an expired cookie is detected, the module automatically triggers a cookie re-retrieval process, seamlessly maintaining an active session connection with the website server and ensuring that subsequent data requests proceed normally. This mechanism avoids login loss or data collection interruptions caused by cookie expiration, improving the stability and success rate of data collection tasks.

[0086] Through the web scraping template design described in the above technical solution, efficient web data collection is designed for specific scraping tasks. This includes defining scraping rules, selecting appropriate scraping tools and objects, and ensuring the accuracy and efficiency of the scraping process. Data scraping logic design encompasses the detailed logic from receiving a web scraping task to final storage. This encompasses every step, including task splitting, scheduling, execution, and storage, ensuring a smooth and efficient scraping process. Functional module design independently designs each functional module within the overall logic design, including data collection, data cleaning, data validation, and data storage. Each module is designed to ensure independence and scalability, thereby improving overall system performance and reliability. This solution not only handles web pages from diverse sources and structures, but also enables efficient data extraction, resumable scraping, cookie management, multi-threaded concurrent scraping, and fixed-level storage. Data cleaning and validation functions further enhance data quality, ensuring character conversion and data integrity. These features make data scraping more comprehensive, flexible, and efficient, while also improving its stability and reliability. In practical applications, data collection tools can significantly improve the efficiency, accuracy and stability of data collection, and provide objective data volume support for subsequent data management, data analysis and management decisions.

[0087] In an embodiment of the present application, the web page data collection method can utilize AIGC (AI generated content) technology, such as natural language processing and computer vision, to solve dynamic web page problems that are difficult for traditional crawlers to handle. For example, based on a deep learning model, the page elements (such as pictures and tables) rendered by JavaScript are automatically identified, or the logical structure of the web page content is understood through semantic analysis to generate adaptive XPath rules. Such technologies have significantly improved the ability to break through dynamic loading and anti-climbing mechanisms (such as verification codes), such as using generative adversarial networks (GANs) to simulate human click behavior to bypass verification codes. AIGC can convert unstructured web page data into structured information. For example, the collected text is subjected to entity recognition, sentiment analysis, or knowledge graph construction through language models such as BERT, realizing an upgrade from "raw data" to "semantic data".

[0088] Figure 1 FIG. 1 is a flow chart of a web page data collection method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0089] Figure 4 The following schematically shows a structural block diagram of a controller according to an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a controller, which may include:

[0090] a memory configured to store instructions;

[0091] The processor is configured to call instructions from the memory and implement the above-mentioned method for controlling the boom when executing the instructions.

[0092] Specifically, in the embodiment of the present application, the processor may be configured to:

[0093] Get the collection task for web page data;

[0094] Invoking the first acquisition template to split the acquisition task into multiple subtasks;

[0095] Determine the task type of each subtask, where different task types are used to collect different types of web page data;

[0096] For each subtask, a target acquisition template from a plurality of preset acquisition templates is called according to the task type of the subtask;

[0097] The collection operation of the subtask is performed according to the target collection template, wherein the multiple preset collection templates include a second collection template and a third collection template, the second collection template is used to collect the web page data of the menu page corresponding to each subtask, and the third collection template is used to collect the web page data of multiple content pages corresponding to each menu page.

[0098] In an embodiment of the present application, the processor may further be configured to:

[0099] Determining the task type of each subtask includes: for each subtask, using the first parameter configured in the first acquisition template to acquire the first data result, and determining whether the subtask needs to acquire data for the menu page based on the first data result; when it is determined that the subtask needs to acquire data for the menu page, determining the task type of the subtask to be a menu page acquisition task; when it is determined that the subtask does not need to acquire data for the menu page, determining the task type of the subtask to be a content page acquisition task.

[0100] In an embodiment of the present application, the processor may further be configured to:

[0101] For each subtask, the target acquisition template from multiple preset acquisition templates is called according to the task type of the subtask, including: for each subtask, when the task type of the subtask is a menu page acquisition task, the second acquisition template is called; for each subtask, when the task type of the subtask is a content page acquisition task, the third acquisition template is called.

[0102] In an embodiment of the present application, the processor may further be configured to:

[0103] The collection operation of executing subtasks according to the target collection template includes: for each subtask, when calling the third collection template, performing a simulated trigger operation according to the third collection template to traverse and access all content pages corresponding to the subtask; for each content page, collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0104] In an embodiment of the present application, the processor may further be configured to:

[0105] The collection operation of executing subtasks according to the target collection template includes: when calling the second collection template, performing a simulated trigger operation according to the second collection template to traverse and access all menu pages corresponding to each subtask; analyzing the format of the page elements of the menu page corresponding to each subtask according to the second collection template to obtain target page elements in the target format in the menu page corresponding to each subtask, and each target page element includes an access link to each content page included in the menu page; calling the third collection template, and performing a simulated trigger operation on each target page element according to the third collection template to access the content page corresponding to each target page element; collecting the web page data included in each content page according to the third collection template, and storing the collected data in the corresponding file directory.

[0106] In an embodiment of the present application, the processor may further be configured to:

[0107] Calling the third acquisition template, and performing a simulated trigger operation on each target page element according to the third acquisition template to access the content page corresponding to each target page element, including: for each subtask, when there are multiple target page elements on the menu page corresponding to the subtask, generating multiple resource locators according to the second acquisition template, each resource locator corresponding to each target page element corresponding to the subtask; generating a target link corresponding to each target page element according to each resource locator; calling the third acquisition template, and performing a simulated trigger operation on each target link according to the third acquisition template to access the content page corresponding to each target link.

[0108] In an embodiment of the present application, the processor may further be configured to:

[0109] For each content page, web page data included in each content page is collected according to the third collection template, and the collected data is stored in a corresponding file directory, including: for each content page, searching the content page for a data format that matches each second parameter of multiple second parameters of the third collection template; collecting web page data in the matching data format included in each content page according to each second parameter, and storing the collected data in the corresponding file directory; wherein, for each second parameter, when the second parameter is an array type, multiple web page data in the matching data format included in each content page are collected.

[0110] In an embodiment of the present application, the processor may further be configured to:

[0111] The third parameter of the second acquisition template is verified. When the parameter type of the third parameter is array type and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, it is confirmed that the third parameter verification is successful, wherein the third parameter is used to analyze the page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure a corresponding resource locator for each target page element corresponding to each subtask; multiple operation parameters for simulating trigger operations are obtained; when the data lengths of the multiple operation parameters are the same, it is determined that the multiple operation parameters are successfully verified; in the event of a verification failure, the collection operation of the web page data is stopped.

[0112] In an embodiment of the present application, the processor may further be configured to:

[0113] For each subtask, the fifth parameter and the sixth parameter configured according to the second acquisition template are combined to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; when each subtask is executed, the task checkpoint of the subtask is marked; in the case of an interruption in the acquisition task, the interruption duration of the restart after the interruption of the acquisition task is obtained; when the interruption duration is less than or equal to the preset duration threshold, the task checkpoint with the closest marking time among all task checkpoints is determined, and the subtask corresponding to the task checkpoint with the closest marking time is executed to continue the acquisition task; when the interruption duration is greater than the preset duration threshold, the acquisition task is re-executed.

[0114] An embodiment of the present application further provides a machine-readable storage medium having stored thereon instructions for enabling a machine to execute the above-mentioned web page data collection method.

[0115] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected via a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data of the web page data collection method. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a web page data collection method is implemented.

[0116] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0122] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0125] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A web page data collection method, characterized in that: The method comprises: Get the collection task for web page data; Invoking a first acquisition template to split the acquisition task into multiple subtasks; Determine the task type of each subtask, where different task types are used to collect different types of web page data; For each subtask, calling a target acquisition template from a plurality of preset acquisition templates according to the task type of the subtask; The collection operation of the subtask is performed according to the target collection template, wherein the multiple preset collection templates include a second collection template and a third collection template, the second collection template is used to collect web page data of the menu page corresponding to each subtask, and the third collection template is used to collect web page data of multiple content pages corresponding to each menu page.

2. The webpage data collection method according to claim 1, characterized in that: Determining the task type of each subtask includes: For each subtask, collect a first data result using the first parameter configured in the first collection template, and determine whether the subtask needs to collect data for the menu page based on the first data result; In the case where it is determined that the subtask needs to collect information for a menu page, determining that the task type of the subtask is a menu page collection task; When it is determined that the subtask does not need to collect data for the menu page, the task type of the subtask is determined to be a content page collection task.

3. The webpage data collection method according to claim 2, characterized in that: For each subtask, calling a target acquisition template from a plurality of preset acquisition templates according to the task type of the subtask includes: For each subtask, if the task type of the subtask is a menu page collection task, calling the second collection template; For each subtask, when the task type of the subtask is the content page collection task, the third collection template is called.

4. The webpage data collection method according to claim 3, characterized in that: The performing of the acquisition operation of the subtask according to the target acquisition template includes: For each subtask, when the third acquisition template is called, a simulated trigger operation is performed according to the third acquisition template to traverse and access all content pages corresponding to the subtask; For each content page, the web page data included in each content page is collected according to the third collection template, and the collected data is stored in a corresponding file directory.

5. The webpage data collection method according to claim 3, characterized in that: The performing of the acquisition operation of the subtask according to the target acquisition template includes: When the second acquisition template is called, a simulated trigger operation is performed according to the second acquisition template to traverse and access all menu pages corresponding to each subtask; Analyzing the format of the page elements of the menu page corresponding to each subtask according to the second acquisition template to obtain target page elements in a target format in the menu page corresponding to each subtask, each target page element including an access link to each content page included in the menu page; calling the third acquisition template, and performing a simulated trigger operation on each target page element according to the third acquisition template to access the content page corresponding to each target page element; The web page data included in each content page is collected according to the third collection template, and the collected data is stored in a corresponding file directory.

6. The webpage data collection method according to claim 5, characterized in that: The calling of the third acquisition template and performing a simulated trigger operation on each target page element according to the third acquisition template to access the content page corresponding to each target page element includes: For each subtask, if there are multiple target page elements on the menu page corresponding to the subtask, generate multiple resource locators according to the second acquisition template, each resource locator corresponding to each target page element corresponding to the subtask; Generate a target link corresponding to each target page element according to each resource locator; The third acquisition template is called, and a simulated trigger operation is performed on each target link according to the third acquisition template to access the content page corresponding to each target link.

7. The webpage data collection method according to claim 5, characterized in that: For each content page, collecting webpage data included in each content page according to the third collection template, and storing the collected data in a corresponding file directory includes: For each content page, searching the content page for a data format that matches each second parameter of the plurality of second parameters of the third acquisition template; Collecting web page data in a matching data format included in each content page according to each second parameter, and storing the collected data in a corresponding file directory; Wherein, for each second parameter, when the second parameter is of array type, a plurality of web page data in a matching data format included in each content page are collected.

8. The webpage data collection method according to claim 6, characterized in that: The method further comprises: Verifying the third parameter of the second acquisition template, and confirming that verification of the third parameter is successful if the parameter type of the third parameter is an array type and the array length of the third parameter is the same as the number of parameters included in the fourth parameter, wherein the third parameter is used to analyze page elements of the menu page corresponding to each subtask, and the fourth parameter is used to configure a corresponding resource locator for each target page element corresponding to each subtask; Get multiple operation parameters for simulating trigger operations; If the data lengths of the multiple operating parameters are all the same, determining that the verification of the multiple operating parameters is successful; In the event of a verification failure, the collection of web page data is stopped.

9. The webpage data collection method according to claim 1, characterized in that: The method further comprises: For each subtask, combining the fifth parameter and the sixth parameter configured in the second acquisition template to create a task checkpoint corresponding to the subtask, wherein the fifth parameter is used to define the task details of the acquisition task, and the sixth parameter is used to define the identifier of each subtask included in the acquisition task; When each subtask is completed, the task checkpoint of the subtask is marked; When the collection task is interrupted, obtain the interruption duration of the collection task after the interruption and restart; When the interruption duration is less than or equal to the preset duration threshold, determining the task checkpoint with the latest marking time among all task checkpoints, and executing the subtask corresponding to the task checkpoint with the latest marking time to continue the collection task; When the interruption duration is greater than the preset duration threshold, the collection task is re-executed.

10. A web page data collection device, characterized in that: include: a memory configured to store instructions; The processor is configured to call the instructions from the memory and implement the web page data collection method according to any one of claims 1 to 9 when executing the instructions.

11. A machine-readable storage medium, characterized in that The machine-readable storage medium stores instructions for enabling a machine to execute the webpage data collection method according to any one of claims 1 to 9.

Images