A page task script driven web information structured collection method and system

Abstract

The application discloses a page task script driven Web information structured collection method and system, which comprises the following steps: navigating to a search page, matching the page task script of the search page through the URL of the search page, executing the page task script of the search page, and loading the search result page; loading and executing the page task script of the search result page, collecting the root page URL of each item of the search result page, and newly creating a record in a page information storage table to save the root page URL of each item of the search result page; based on the root page URL, navigating to the root page of each item in sequence, loading and executing the page task script of the root page, collecting the information of each root page and each subpage URL, and saving the information of the root page and each subpage URL into the record of the corresponding item in the page information storage table. The application collects and transforms the related unstructured associated information distributed on multiple pages into the structured record in the database.

Description

Technical Field

[0001] This application relates to the field of Internet information collection and processing, and in particular to a method and system for structured collection of Web information driven by page task scripts. Background Technology

[0002] In today's era of big data, the internet offers a vast amount of publicly available information. Much of this information is published in web format, suitable for human reading but not for machine processing. Information gathering is the first step in information utilization, and many individuals and organizations have long established their own information gathering systems to acquire this publicly available information and structure it for further use.

[0003] Because each website provides different information and uses different webpage formats, it is necessary to build a customized information collector for each website, which is a time-consuming and labor-intensive task. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for structured web information collection driven by page task scripts. This system can easily and quickly establish a web information collector, collecting unstructured related information scattered across multiple pages and transforming it into structured records in a database.

[0005] The present invention adopts the following technical solution:

[0006] On the one hand, a method for structured web information collection driven by page task scripts includes:

[0007] S10, navigate to the search page, match the page task script of the search page through the URL of the search page, execute the page task script of the search page, and load the search results page;

[0008] S20, on the search results page, load and execute the page task script of the search results page, collect the root page URL of each item on the search results page, and create a new record in the page information storage table to save the root page URL of each item on the search results page; where each item corresponds to one record;

[0009] S30, based on the root page URL, navigates sequentially to the root page of each project, loads and executes the page task script of the root page, collects the information of each root page and the URL of each subpage, and saves the information of the root page and the URL of each subpage to the record of the corresponding project in the page information storage table; based on the subpage URL, navigates sequentially to the subpage of each root page, collects the subpage information and saves it to the record of the corresponding project in the page information storage table.

[0010] Preferably, in step S10, the user navigates to the search page via the starting link URL set in the task configuration file.

[0011] Preferably, the mapping relationship between each URL and the page task script is saved through the collection task configuration file.

[0012] Preferably, in step S10, executing the page task script for the search page to load the search results page specifically includes:

[0013] During the execution of the page task script on the search page, the search criteria are set, and the search button is clicked, causing the search results page to load.

[0014] Preferably, step S20 specifically includes:

[0015] S201, Collect relevant information and root page URL for each item on the current search results page, and create a new record in the page information storage table to save the above information, and save the root page URL to the "root page URL" field of the corresponding record;

[0016] S202, After collecting all project information, click the next page link to load the next search results page;

[0017] S203, repeat S201 to S202 until all search result pages have been processed.

[0018] Preferably, S30 specifically includes:

[0019] S301, Collect information of the current root page and save it to the "Page Information XML" field of the corresponding record in the page information storage table;

[0020] S302, Collect the URLs of each subpage of the current root page and save them to the "Subpage URL Collection" field of the corresponding record in the page information storage table;

[0021] S303, navigate to the subpage based on the subpage URL, collect the subpage information and save it to the "Page Information XML" field of the corresponding record in the page information storage table; then collect the URL of the secondary subpage, and use the URL of the secondary subpage to access and collect the secondary subpage; use a recursive approach until all the subpages that need to be accessed have been traversed.

[0022] S304, repeat S301 to S303 until all records corresponding to all items in the page information storage table have been processed.

[0023] Preferably, the web information structured collection method driven by page task scripts further includes:

[0024] The stored "Page Information XML" field is converted into a table in a relational database and stored there.

[0025] On the other hand, a web information structured collection system driven by page task scripts includes: a website, a web page collector, a set of page task scripts, page information XML, and a page information storage table;

[0026] The website consists of multiple web pages, including a search page, at least one search results page, at least one root page, and at least one subpage;

[0027] The page task script set includes the page task scripts corresponding to each webpage;

[0028] The page information XML is used to store the collected page information in XML format;

[0029] The page information storage table is used to store a project record in the form of a relational database table;

[0030] The web page crawler is used to implement the following methods:

[0031] Navigate to the search page, match the page task script of the search page through the URL of the search page, execute the page task script of the search page, and load the search results page;

[0032] On the search results page, load and execute the page task script for the search results page, collect the root page URLs of each item on the search results page, and create new records for the root page URLs of each item on the search results page in the page information storage table; where each item corresponds to one record;

[0033] Based on the root page URL, navigate sequentially to the root page of each project, load and execute the page task script of the root page, collect information of each root page and URL of each subpage, and save the root page information and URL of each subpage to the record of the corresponding project in the page information storage table; based on the subpage URL, navigate sequentially to the subpage of each root page, collect the subpage information and save it to the record of the corresponding project in the page information storage table.

[0034] Preferably, the web information structured acquisition system driven by the page task script further includes an acquisition task configuration file; the web page collector enters the search page of the target website based on the acquisition task configuration file, starting from the initial link, and traverses the entire website using the website link structure formed by the links between web pages to collect the required information; when the web page collector enters a web page, it matches the page task script corresponding to that type of web page from the page task script set according to the correspondence between web pages and task scripts recorded in the acquisition task configuration file and executes the script, performing task operations on the web page under the drive of the page task script.

[0035] Preferably, the web page collector consists of a web page analyzer and a page task script execution environment; the web page analyzer is used to load and parse web pages and provide page task scripts with functions for accessing page elements and performing simulated human operations on the page; the page task script execution environment is used to load and execute page task scripts in an interpreted manner, and provide the page task scripts with a web page analyzer call interface, through which the page task scripts send operation instructions to various elements on the web page.

[0036] The present invention has the following beneficial effects:

[0037] (1) The present invention can easily and quickly establish a web page collector and drive the execution of page task scripts to collect unstructured related information scattered on multiple pages and transform it into structured records in the database.

[0038] (2) Each webpage has a unique URL. Hyperlinks are created using URLs to organize scattered webpages into a website. Webpages with different functions in the website correspond to URLs with different patterns. This invention uses regular expressions to describe these patterns and performs pattern matching. The mapping relationship between the webpage type URL regular expression template and the corresponding page task script is stored in the collection task configuration file. Webpages of the same type use the same document structure and style, so only one task script needs to be written to process a type of webpage. After a webpage is loaded, the webpage collector matches the URL of the current webpage with the URL pattern stored in the configuration file, finds the page task script corresponding to the type of webpage, and can load and execute the page operation tasks set by the page task script. This allows for the convenient and rapid establishment of a webpage collector for any website, enabling rapid information collection.

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Attached Figure Description

[0040] Figure 1 This is a flowchart of a web information structured acquisition method driven by page task scripts according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the link structure of the mobile information disclosure website according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the search page layout according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the search results page layout according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the layout of the mobile phone model details page (root page) according to an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the CPU details page layout (subpage) according to an embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of structured mobile phone model information according to an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of the information collection framework according to an embodiment of the present invention;

[0048] Figure 9 This is a schematic diagram of mobile phone model information XML according to an embodiment of the present invention;

[0049] Figure 10 This is a schematic diagram of the web page crawler configuration according to an embodiment of the present invention;

[0050] Figure 11 This is a flowchart illustrating the process of collecting search results on an embodiment of the present invention.

[0051] Figure 12 This is a flowchart illustrating the overall data acquisition process in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0053] In the description of this invention, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the step identifiers S10, S20, S30, etc. are used only for convenience of description and do not indicate the execution order. The corresponding execution order can be adjusted.

[0055] See Figure 1 As shown, this method for structured web information collection driven by page task scripts includes:

[0056] S10, navigate to the search page, match the page task script of the search page through the URL of the search page, execute the page task script of the search page, and load the search results page;

[0057] S20, on the search results page, load and execute the page task script of the search results page, collect the root page URL of each item on the search results page, and create a new record in the page information storage table to save the root page URL of each item on the search results page; where each item corresponds to one record;

[0058] S30, based on the root page URL, navigates sequentially to the root page of each project, loads and executes the page task script of the root page, collects the information of each root page and the URL of each subpage, and saves the information of the root page and the URL of each subpage to the record of the corresponding project in the page information storage table; based on the subpage URL, navigates sequentially to the subpage of each root page, collects the subpage information and saves it to the record of the corresponding project in the page information storage table.

[0059] In this embodiment, in step S10, the user navigates to the search page via the starting link URL set in the task configuration file.

[0060] Furthermore, the mapping relationship between each URL and the page task script is saved by collecting task configuration files.

[0061] In this embodiment, step S10, executing the page task script for the search page to load the search results page, specifically includes:

[0062] During the execution of the page task script on the search page, the search criteria are set, and the search button is clicked, causing the search results page to load.

[0063] S20 specifically includes:

[0064] S201, Collect relevant information and root page URL for each item on the current search results page, and create a new record in the page information storage table to save the above information, and save the root page URL to the "root page URL" field of the corresponding record;

[0065] S202, After collecting all project information, click the next page link to load the next search results page;

[0066] S203, repeat S201 to S202 until all search result pages have been processed.

[0067] S30 specifically includes:

[0068] S301, Collect information of the current root page and save it to the "Page Information XML" field of the corresponding record in the page information storage table;

[0069] S302, Collect the URLs of each subpage of the current root page and save them to the "Subpage URL Collection" field of the corresponding record in the page information storage table;

[0070] S303, navigate to the subpage based on the subpage URL, collect the subpage information and save it to the "Page Information XML" field of the corresponding record in the page information storage table; then collect the URL of the secondary subpage, and use the URL of the secondary subpage to access and collect the secondary subpage; use a recursive approach until all the subpages that need to be accessed have been traversed.

[0071] S304, repeat S301 to S303 until all records corresponding to all items in the page information storage table have been processed.

[0072] Using the above method, the present invention can easily and quickly establish a web page collector and drive the execution of page task scripts to collect and transform unstructured related information scattered across multiple pages into structured records in a database.

[0073] The following will use a mobile information disclosure website as an example to provide a detailed explanation of the web information structure collection driven by page task scripts of the present invention.

[0074] See Figure 2 The diagram shown illustrates the link structure of the mobile phone information disclosure website in this embodiment. The main function of this website is to retrieve and display the technical parameters and related information of various mobile phone models on the market. This website is abstracted from a real website. Abstractly speaking, a website is a collection of web pages linked together by hyperlinks. The structure of web pages linked together can be broadly categorized into hierarchical and linear structures. Most websites use a combination of these two structures to construct their link structure.

[0075] See Figures 3 to 6As shown, the mobile phone information disclosure website's webpage layout includes search page layout, search results page layout, mobile phone details page layout (root page), and CPU details page layout (subpage). Information about a single mobile phone model is scattered across multiple webpages, which are organized into a tree structure via hyperlinks to ensure that all information pages for that model can be accessed through the root page. However, this information is scattered and unstructured, and cannot be directly used for data analysis.

[0076] Therefore, the web information structure collection method driven by page task scripts in this embodiment needs to collect all page information for each mobile phone model and organize it into a structured information item in a relational database. See the diagram of the structured mobile phone model information. Figure 7 As shown.

[0077] See the schematic diagram of the information acquisition framework in this embodiment. Figure 8 As shown in the diagram, the web page collector is the core of the entire framework. This collector gathers web page information and stores it in a semi-structured XML-based page information storage system. The "Collection Task Configuration File" stores the information and parameters required for the collection task, such as the task's starting link and the mapping relationship between web pages and page task scripts. Guided by the "Collection Task Configuration File," the web page collector enters the target website from the starting link, traversing the entire website using the website link structure formed by the links between web pages to collect the necessary information. When the web page collector arrives at a web page, it matches the corresponding task script from the "Page Task Script Collection" based on the mapping relationship between web pages and task scripts recorded in the task configuration file and executes the script. Driven by the page task script, it performs tasks such as information collection, link collection, link clicking, setting search conditions, and clicking the search button on that web page.

[0078] In this embodiment, the information collected from the webpage is first stored in a "page information storage" in XML format, which is semi-structured information. Then, the semi-structured XML page information storage can be converted into structured project information storage in a relational database (see...). Figure 7 (As shown). It should be noted that since modern relational databases all support the storage of Long Text data types, the entire XML document can be stored as a field in the relational database table (the page information storage table mentioned above).

[0079] Compared to relational databases, XML is a more flexible data format. According to relational database theory, a relation is a flat structure that does not allow nested components within attributes, making a single relation unsuitable for expressing hierarchical information. XML, on the other hand, allows for both flat, structured information through attribute fields and nested child elements within a single XML element, thus achieving hierarchical information expression. Therefore, XML offers greater flexibility and can fully adapt to the diverse and dynamic nature of information on different websites. See the XML diagram illustrating mobile phone model information in this embodiment. Figure 9 As shown in Table 1, the relationship between XML and mobile phone model information is illustrated. While XML offers greater flexibility in data organization than relational models, relational databases, supported by relational theory, possess powerful data retrieval and analysis capabilities that XML cannot match. Therefore, mainstream data analysis systems today are still built on relational databases, rather than directly on XML data storage. Converting XML data into relational database tables is quite simple; intuitively, it involves flattening the XML hierarchy. Modern mainstream relational databases such as MS SQL Server, MySQL, and Oracle all have XML data manipulation mechanisms, allowing users to easily import XML into the database or convert data in the database into XML format.

[0080] Table 1

[0081]

[0082] In summary, this embodiment primarily involves aggregating and collecting information scattered across multiple web pages into a single XML record. This task is accomplished by a web page crawler driven by a page task script. Figure 8 Each webpage in the website has a unique URL, and hyperlinks are created using these URLs to organize the scattered webpages into a website. Different functional webpages within the website correspond to different URL patterns. Referring to the mobile phone information disclosure website, webpages are categorized into at least four types based on their information type and format: the search website homepage, search results page, mobile phone information homepage, and various mobile phone attachment information pages. The page type and URL format comparison table (as shown in Table 2 below) illustrates the webpage types and their corresponding URL formats.

[0083] Table 2

[0084]

[0085] The following URL and regular expression mapping table (Table 3) shows how regular expressions describe these patterns and perform pattern matching. The mapping relationship between the URL regular expression templates for webpage types and the corresponding page task scripts is stored in the data collection task configuration file. Webpages of the same type often use the same document structure and style, so only one task script needs to be written to handle this type of webpage. After the webpage collector loads a webpage, it matches the URL of the current webpage with the URL patterns stored in the configuration file, finds the page task script corresponding to that type of webpage, loads and executes the page operation tasks set in the task script: collecting page element information, collecting links, clicking links, setting search conditions and clicking the search button, etc.

[0086] Table 3

[0087]

[0088] See Figure 10 As shown, in this embodiment, both webpage analysis and page task script execution are performed within the webpage collector. The webpage collector mainly consists of a webpage analyzer and a page task script execution environment. The main function of the webpage analyzer is to load and parse the webpage and provide the "page task script" with access to page elements (reading and setting page element attributes) and to perform simulated human operations on the page (clicking links, entering text in input boxes, clicking buttons). The main function of the page task script execution environment is to load and execute the "page task script" in an interpreted manner, and to provide the necessary services for the "page task script" to execute, namely, the webpage analyzer call interface. Through this interface, the "page task script" can send operation instructions to various elements on the webpage, such as clicking buttons, clicking links, setting text in input boxes, etc., and can also read all attributes, styles, and data of various elements on the webpage.

[0089] Specifically, webpage analyzers can be implemented using existing solutions, such as Selenium + browser (Chrome or Firefox). The webpage is loaded and run in the browser (including the execution of JavaScript within the page), ultimately forming a DOM (Document Object Model) tree. Visible components on the webpage (text, images, controls: input boxes, buttons, etc.) are all element nodes in the DOM tree. Selenium acts as the browser's access and control interface. Through Selenium, any element in the DOM tree can be accessed, its properties read or set, and simulated human-like commands can be sent to it (clicks, text settings).

[0090] The page task script can use a custom scripting language or a mature scripting language such as JavaScript or Python. The advantage of using a mature scripting language is that you don't need to develop your own script engine; there are already open-source script engines available on the market that can be integrated into the system. In this embodiment, the page task script runs in an isolated environment set up by the script engine. It cannot directly call the page element access functions provided by the browser access and control interface (Selenium). Therefore, this interface needs to be introduced and encapsulated into the script execution environment to form a "scripted web page analyzer call interface" for the page task script to call.

[0091] See Figure 11 The diagram illustrates the data collection process of a web scraper on a search results page. This embodiment uses a UML activity diagram. The specific operations for collecting page information and simulating human behavior on the page (such as clicking web page elements) are described in the task script. Below is the page task script corresponding to the search results page, presented in JavaScript.

[0092]

[0093] The page task script runs in an isolated environment called a "sandbox" built for it by the script engine, and therefore cannot directly call the web page analyzer's access interface. The aforementioned `script_agent` object encapsulates the web page analyzer's access interface into the script execution environment. Within the script, this object allows access to all the functions supported by the web page analyzer. Similarly, page information storage is constructed by the information collection framework and cannot be directly accessed within the task script; therefore, it is also introduced into the script environment through the `script_agent` object.

[0094] Since the information in a single structured information item is scattered across multiple hyperlinked subsets of web pages, the collector needs to traverse all web pages within each subset to aggregate this information into a single information entity. The page information is collected using the XML format described earlier to accommodate the diversity of web page information. In addition, the collection process requires some auxiliary information. Because XML is stored as an independent document in the file system, its retrieval performance is not high. Furthermore, modern relational databases support Long Text data types, meaning the entire XML document can be stored as a field in a relational database table. Therefore, this embodiment designs a page information storage table (Table 4) to store page information. The Information Item ID field uniquely identifies the information item; for the mobile phone information disclosure website, each mobile phone model corresponds to one information ID. The Root Page URL field stores the URL of the root page of the associated web page subset. The page information to be collected within this subset is collected as one information item record; for the mobile phone information disclosure website, the details page of each mobile phone model is the root page of that subset. The `Subpage URL Collection` field stores the URLs of all subpages. The crawler enters this subset of pages from the root page, processes the root page, and collects the URLs of all direct subpages linked from the root page. These URLs are then used to continue traversing all direct subpages of the root page, and further, to collect the URLs of subpages' subpages. This process is repeated recursively until all the required subpages within the subset are accessed. The `Page Information XML` field stores the information collected from the root page and its direct and indirect subpages in XML document format.

[0095] Table 4

[0096] Information Project ID Root page URL Subpage URL collection Page Information XML

[0097] See Figure 12 As shown, taking a mobile phone information disclosure website as an example, the overall workflow of the web scraper is as follows. This embodiment uses "5G" as the search keyword to collect parameters and other relevant information of 5G mobile phones on the market. The detailed process is as follows:

[0098] (1) The "Web Page Collector" navigates to the search page (using the starting link set in the "Collection Task Configuration File"), matches the URL, and executes the page task script corresponding to the search page (the mapping relationship between the URL regular expression pattern and the page task script is stored in the "Collection Task Configuration File"). The operation flow of the script settings is as follows:

[0099] (1.1) Set the keyword “5G” in the search box.

[0100] (1.2) Clicking the “Search” button will load the search results page.

[0101] (2) On the search results page, load and execute the corresponding page task script. The script's operation flow is as follows:

[0102] (2.1) Collect relevant information (model, feature summary, market reference price, etc.) for each mobile phone model on the current search results page and the URL of the mobile phone model details page link (the root page of the page subset), and create a new record in the page information storage table to save the above information.

[0103] (2.2) After collecting all mobile project information, click the next page link to load the next search results page.

[0104] (3) Repeat step (2) until all search result pages have been processed.

[0105] (4) The web crawler navigates sequentially to the details page of each mobile phone model (using the "root page URL" field of the page information storage table) and executes the corresponding page task script:

[0106] (4.1) Collect the information of the page (CPU model, screen size, running storage capacity, etc.) and update it to the "Page Information XML" field of the record corresponding to the mobile phone model in the "Page Information Storage Table".

[0107] (4.2) Collect subpage URLs (CPU details page URL, screen details page URL, etc.) and save them to the "Subpage URL Collection" field of the corresponding record.

[0108] (4.3) Navigate to the subpage according to the subpage URL, collect the subpage information and update the "Page Information XML" field in the corresponding record of the mobile phone model; then collect the URL of the secondary subpage, and use the URL to access and collect the secondary subpage; use a recursive approach until all the subpages that need to be accessed in the page subset have been traversed.

[0109] (5) Repeat step (4) until all records corresponding to all mobile phone models in the "Page Information Storage Table" have been processed.

[0110] In summary, this invention provides a framework for structured collection of page information based on page task scripts. The collection method and system using this framework can easily and quickly establish an information collector for any website, transforming the unstructured information of the web pages to be collected into structured information to meet the needs of information utilization.

[0111] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A method for structured web information acquisition driven by page task scripts, characterized in that, include: S10, navigate to the search page, match the page task script of the search page through the URL of the search page, execute the page task script of the search page, set the search conditions during the script execution, and click the search button to load the search results page; save the mapping relationship between each URL and the page task script through the collection task configuration file; S20, on the search results page, load and execute the page task script of the search results page, collect the root page URL of each item on the search results page, and create a new record in the page information storage table to save the root page URL of each item on the search results page; where each item corresponds to one record; S30, based on the root page URL, navigates sequentially to the root page of each project, loads and executes the page task script of the root page, collects the information of each root page and the URL of each subpage, and saves the information of the root page and the URL of each subpage to the record of the corresponding project in the page information storage table; based on the subpage URL, navigates sequentially to the subpage of each root page, collects the subpage information and saves it to the record of the corresponding project in the page information storage table. The search page, at least one search result page, at least one root page, and at least one subpage constitute a webpage, and multiple webpages constitute a website; the page task scripts corresponding to each webpage constitute a page task script set; the page information is stored in XML format through page information XML; the page information storage table stores one item record in the form of a relational database table; The specific implementation of the data collection task configuration file includes: the web page collector enters the search page of the target website based on the data collection task configuration file starting from the initial link, traverses the entire website using the website link structure formed by the links between web pages, and collects the required information; when the web page collector enters a web page, it matches the corresponding page task script from the page task script set according to the correspondence between web pages and task scripts recorded in the data collection task configuration file and executes the script, and performs task operations on the web page under the drive of the page task script; The web page collector consists of a web page analyzer and a page task script execution environment. The web page analyzer is used to load and parse web pages and provide page task scripts with functions such as accessing page elements and performing simulated human operations on the page. The page task script execution environment is used to load and execute page task scripts in an interpreted manner, and provides a web page analyzer call interface for page task scripts, through which page task scripts send operation instructions to various elements on the web page.

2. The method for structured web information acquisition driven by page task scripts according to claim 1, characterized in that, In step S10, the user navigates to the search page via the starting link URL set in the task configuration file.

3. The method for structured web information acquisition driven by page task scripts according to claim 1, characterized in that, S20 specifically includes: S201, Collect relevant information and root page URL for each item on the current search results page, and create a new record in the page information storage table to save the above information, and save the root page URL to the "root page URL" field of the corresponding record; S202, After collecting all project information, click the next page link to load the next search results page; S203, repeat S201~S202 until all search result pages have been processed.

4. The method for structured web information acquisition driven by page task scripts according to claim 3, characterized in that, S30 specifically includes: S301, collect the information of the current root page and save it to the "Page Information XML" field of the corresponding record in the page information storage table; S302, Collect the URLs of each subpage of the current root page and save them to the "subpage URL collection" field of the corresponding record in the page information storage table; S303, navigate to the subpage based on the subpage URL, collect the subpage information and save it to the "Page Information XML" field of the corresponding record in the page information storage table; then collect the URL of the secondary subpage, and use the URL of the secondary subpage to access and collect the secondary subpage; use a recursive approach until all the subpages that need to be accessed have been traversed. S304, repeat S301~S303 until all records corresponding to all items in the page information storage table have been processed.

5. The method for structured web information acquisition driven by page task scripts according to claim 4, characterized in that, Also includes: The stored "Page Information XML" field is converted into a table in a relational database and stored there.

6. A web information structured acquisition system driven by page task scripts, characterized in that, include: Website, web scraper, page task script set, page information XML, page information storage table, and task configuration file; The website consists of multiple web pages, including a search page, at least one search results page, at least one root page, and at least one subpage; The page task script set includes the page task scripts corresponding to each webpage; The page information XML is used to store the collected page information in XML format; The page information storage table is used to store a project record in the form of a relational database table; The web page crawler is used to implement the following methods: Navigate to the search page, match the page task script of the search page through the URL of the search page, execute the page task script of the search page, set the search conditions during the script execution, and click the search button to load the search results page; save the mapping relationship between each URL and the page task script through the collection task configuration file; On the search results page, load and execute the page task script for the search results page, collect the root page URLs of each item on the search results page, and create new records for the root page URLs of each item on the search results page in the page information storage table; where each item corresponds to one record; Based on the root page URL, navigate sequentially to the root page of each project, load and execute the page task script of the root page, collect information of each root page and URL of each subpage, and save the root page information and URL of each subpage to the record of the corresponding project in the page information storage table; based on the subpage URL, navigate sequentially to the subpage of each root page, collect the subpage information and save it to the record of the corresponding project in the page information storage table. The specific implementation of the data collection task configuration file includes: the web crawler enters the search page of the target website based on the data collection task configuration file, starting from the initial link, and traverses the entire website using the website link structure formed by the links between web pages to collect the required information; when the web crawler enters a web page, it matches the corresponding page task script from the page task script set according to the correspondence between web pages and task scripts recorded in the data collection task configuration file and executes the script, and performs task operations on the web page under the drive of the page task script; The web page collector consists of a web page analyzer and a page task script execution environment. The web page analyzer is used to load and parse web pages and provide page task scripts with functions such as accessing page elements and performing simulated human operations on the page. The page task script execution environment is used to load and execute page task scripts in an interpreted manner, and provides a web page analyzer call interface for page task scripts, through which page task scripts send operation instructions to various elements on the web page.

Images