Page data extraction method and device and computer device

Abstract

This application relates to a method, apparatus, and computer device for extracting page data. The method includes: obtaining report links from a webpage to form a candidate report list; comparing the candidate report list with a local historical report database to select new reports from the candidate report list; calling an automated testing tool to load and render the webpage content corresponding to the new report, generating a PDF file; calling an optical character recognition tool to parse the layout of the PDF file to convert it into a Markdown file; calling a multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file; adding a timestamp to the structured file to obtain a structured time-series file, and storing it in the historical report database for use in different analysis scenarios. This method can improve data acquisition efficiency.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for extracting page data. Background Technology

[0002] In the field of public health, the official websites of Centers for Disease Control and Prevention (CDC) regularly publish weekly disease surveillance reports, presented in webpage or PDF format. Research institutions, modeling teams, and public health departments typically extract key indicators from these reports when conducting epidemic trend prediction, pathogen surveillance assessment, and emergency response evaluation. These indicators include pathogen positivity rates, ILI (influenza-like illness) or SARI (severe acute respiratory infection) outpatient visits, surveillance weeks, and reference dates.

[0003] Traditional methods mostly rely on manually downloading reports page by page, manually copying tables from PDFs, and organizing them into Excel or CSV files before proceeding with the analysis. However, due to inconsistent file formats, complex table structures, easily changing layouts, and high file update frequency, manual methods are not only time-consuming and labor-intensive, but also prone to problems such as data omissions, extraction errors, and repetitive work. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for page data extraction that can accurately and efficiently acquire monitoring data, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for extracting page data, including:

[0006] Extract report links from web pages to create a candidate report list;

[0007] The candidate report list is compared with the local historical report database, and new reports are selected from the candidate report list;

[0008] The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, generating a PDF file.

[0009] The PDF file is parsed using an optical character recognition tool to convert it into a Markdown file;

[0010] The Markdown file is analyzed in a structured manner by calling the multi-agent parsing chain to obtain a structured file. A timestamp is added to the structured file to obtain a structured time series file, which is then stored in the historical report database for use in different analysis scenarios.

[0011] In one embodiment, the step of calling the automated testing tool to load and render the webpage content corresponding to the newly added report and generate a PDF file includes:

[0012] Configure the corresponding parameters for generating PDF files according to the file type of the newly added report; the parameters include at least one of rendering waiting parameters and PDF file page layout parameters.

[0013] The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, and a PDF file is generated according to the configured parameters.

[0014] In one embodiment, the step of invoking the multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file includes:

[0015] The multi-agent parsing chain is invoked to perform structured analysis on the Markdown file to obtain structured information; the structured information includes at least key fields, including the report's release date, monitoring week, monitoring system, and pathogen detection data;

[0016] The structured information is output as a CSV file to obtain a structured file.

[0017] In one embodiment, the step of performing structured analysis on the Markdown file to obtain structured information includes:

[0018] If the report corresponding to the Markdown file is a normal report, a rule-based parsing strategy is used to obtain the structured information;

[0019] If the report corresponding to the Markdown file is an anomaly report, then after parsing using a rule-based parsing strategy, a large language model is used to correct the parsing results to obtain the structured information.

[0020] In one embodiment, adding a timestamp to the structured file to obtain a structured time-series file and storing it in the historical report database includes:

[0021] Add a timestamp to the structured file according to the publication date of the report corresponding to the structured file to obtain a structured time-series file;

[0022] The structured time series file is stored in the historical report database, and the structured time series file is merged with the relevant structured time series file in the historical report database.

[0023] In one embodiment, the process of merging the structured time-series file with the relevant structured time-series files in the historical report database includes:

[0024] Based on the release date of the report corresponding to the structured time series file, match the week corresponding to the release date, and create a multi-level file directory by combining the reference date;

[0025] The structured time series files and related structured time series files in the historical report database are sorted uniformly, and duplicate files are removed based on unique keys.

[0026] In one embodiment, the method further includes:

[0027] The structured time series file is output through a standardized data output interface; the data output interface is compatible with various prediction model frameworks and supports the generation of visualization graphics.

[0028] In one embodiment, the method further includes:

[0029] By using workflow orchestration and scheduling tools, each task implementing the page data extraction method is defined as a directed acyclic graph to achieve automated scheduling of each task.

[0030] Secondly, this application also provides a page data extraction device, comprising:

[0031] The link retrieval module is used to retrieve report links from web pages and form a candidate report list.

[0032] The incremental detection module is used to compare the candidate report list with the local historical report database and filter out new reports from the candidate report list;

[0033] The file generation module is used to call automated testing tools to load and render the web page content corresponding to the newly added report, and generate a PDF file.

[0034] The file conversion module is used to call an optical character recognition tool to parse the layout of the PDF file and convert it into a Markdown file;

[0035] The document analysis module is used to call the multi-agent parsing chain to perform structured analysis on the Markdown file, obtain a structured file, add a timestamp to the structured file to obtain a structured time series file, and store it in the historical report database for use in different analysis scenarios.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Extract report links from web pages to create a candidate report list;

[0038] The candidate report list is compared with the local historical report database, and new reports are selected from the candidate report list;

[0039] The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, generating a PDF file.

[0040] The PDF file is parsed using an optical character recognition tool to convert it into a Markdown file;

[0041] The Markdown file is analyzed in a structured manner by calling the multi-agent parsing chain to obtain a structured file. A timestamp is added to the structured file to obtain a structured time series file, which is then stored in the historical report database for use in different analysis scenarios.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0043] Extract report links from web pages to create a candidate report list;

[0044] The candidate report list is compared with the local historical report database, and new reports are selected from the candidate report list;

[0045] The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, generating a PDF file.

[0046] The PDF file is parsed using an optical character recognition tool to convert it into a Markdown file;

[0047] The Markdown file is analyzed in a structured manner by calling the multi-agent parsing chain to obtain a structured file. A timestamp is added to the structured file to obtain a structured time series file, which is then stored in the historical report database for use in different analysis scenarios.

[0048] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0049] Extract report links from web pages to create a candidate report list;

[0050] The candidate report list is compared with the local historical report database, and new reports are selected from the candidate report list;

[0051] The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, generating a PDF file.

[0052] The PDF file is parsed using an optical character recognition tool to convert it into a Markdown file;

[0053] The Markdown file is analyzed in a structured manner by calling the multi-agent parsing chain to obtain a structured file. A timestamp is added to the structured file to obtain a structured time series file, which is then stored in the historical report database for use in different analysis scenarios.

[0054] The aforementioned page data extraction method, apparatus, computer equipment, computer-readable storage medium, and computer program product obtain report links from web pages to form a candidate report list; compare the candidate report list with a local historical report database to select new reports; call an automated testing tool to load and render the web page content corresponding to the new reports, generating a PDF file; call an optical character recognition tool to parse the layout of the PDF file to convert it into a Markdown file; call a multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file; add a timestamp to the structured file to obtain a structured time-series file, and store it in the historical report database for use in different analysis scenarios. Through the synergistic effect of various tools, automated access, batch downloading, and conversion of reports from web pages are achieved, replacing manual processing, significantly reducing manual work time, and improving data acquisition efficiency. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a page data extraction method in one embodiment;

[0057] Figure 2 This is a flowchart illustrating the page data extraction method in another embodiment;

[0058] Figure 3 This is a flowchart illustrating the page data extraction method in yet another embodiment;

[0059] Figure 4 This is a structural block diagram of a page data extraction device in one embodiment;

[0060] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more.

[0062] It is understandable that sentinel surveillance reports from disease control and prevention centers are primarily in webpage and PDF format, with diverse file formats, inconsistent layouts across versions, and complex, unstandardized table structures. Reports from different weeks are scattered across multiple links, making manual downloading and extraction extremely inefficient and prone to issues such as data omissions, table recognition errors, inconsistent week alignment, and misreading of dates spanning multiple years. Existing general-purpose web crawlers and PDF extraction tools cannot reliably parse specific structured fields such as dates, monitoring indicators, or weeks spanning multiple years from sentinel surveillance reports, and lack incremental updates, automatic aggregation, and compatibility with formats required by prediction models. Therefore, this application proposes a tool that can automatically synchronize surveillance reports from the official website of disease control and prevention centers and accurately convert the content into structured data. This addresses the problems of inconsistent weekly report link sources, frequent page structure changes, complex PDF table layouts, and the difficulty in maintaining accumulated data from multiple weekly reports over long periods.

[0063] In one embodiment, such as Figure 1 As shown, a method for extracting page data is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0064] Step S110: Obtain the report links from the webpage to form a candidate report list.

[0065] Specifically, after extracting the report links from the webpage, the extracted weekly report links are preprocessed, such as by filtering, standardization, and noise reduction, to obtain a list of candidate reports with a uniform format and parsable capability.

[0066] Taking the extraction of page data from the monitoring report page of a sentinel hospital as an example, a web crawler can be used to extract all available weekly report links from the monitoring report page of the sentinel hospital for preprocessing, forming a candidate report list. For example, the web crawler is Firecrawl (a cloud-based web crawling and data extraction API service). By calling Firecrawl's map interface, a full scan of the sentinel hospital's monitoring report page is performed, and all possible report links are extracted through the DOM map. Specifically, Firecrawl's map interface sends a request to the monitoring report page, loads and parses the webpage to generate a DOM map, traverses the anchor tags in the DOM map, extracts report links, and if there are multiple report links, a report link list is formed.

[0067] Step S120: Compare the candidate report list with the local historical report database, and select new reports from the candidate report list.

[0068] For example, the candidate report list is compared with the local historical report database (containing links to historical reports) to filter out new reports that have not yet been processed locally. The scheduling system (Airflow) automatically adds the links to new reports to the pending queue each time a task is executed, achieving incremental updates.

[0069] This step automatically compares the report link retrieved from the webpage with the historical report database each time, effectively identifying new reports. This enables incremental updates, avoids repeated crawling and parsing, and improves the system's computational efficiency and long-term stability.

[0070] Step S130: Call the automated testing tool to load and render the webpage content corresponding to the new report, and generate a PDF file.

[0071] For example, Playwright is used as the automated testing tool to control the browser engine and automate browser operations. Specifically, the steps for generating a PDF file using Playwright include: launching the browser through Playwright to create a new page; controlling the new page to access the link to the new report, i.e., the URL (uniform resource locator), and waiting for the page (including dynamic content) to finish loading; calling the page's PDF generation method, configuring parameters such as paper format, background retention, and margins, specifying the save path, and generating a high-quality PDF file. This process also supports concurrency control and exception retries to adapt to scenarios with frequent webpage adjustments and unstable resource loading.

[0072] Step S140: Use an optical character recognition tool to parse the layout of the PDF file and convert it into a Markdown file.

[0073] For example, an optical character recognition tool could be the MinerU API, capable of converting complex documents such as PDFs into structured formats like Markdown / JSON. Specifically, calling the MinerU API to perform layout parsing on PDF files accurately identifies and converts information such as text, paragraphs, date ranges, and complex table structures into Markdown representations, restoring table boundaries, row and column structures, and content correspondences, thus laying a stable structured foundation for subsequent semantic parsing.

[0074] For example, tables in PDFs may be "borderless tables," "dashed-line tables," or have blurred or missing borders due to scanning. The MinerU API uses layout analysis to accurately identify the physical boundaries of tables (such as which row is the header, which column is the data column, and the start and end positions of the table), and clearly indicates the boundaries in the Markdown table using vertical lines and separators to prevent content from being "overflowed."

[0075] For example, complex tables in PDFs (such as merged cells, tables spanning multiple rows and columns, and tables with multiple headers) have a "visual relationship" between their rows and columns in the original file, rather than structured data that is readable by machines. MinerUAPI parses this hierarchical relationship and restores it to the correct row and column correspondence logic in Markdown—for example, merged cells will be split into cells with the corresponding number of rows / columns, and multiple headers will be arranged hierarchically, ensuring that the "skeleton" of the table is consistent with the original PDF.

[0076] This step utilizes MinerU's table extraction capabilities and rule engine to achieve stable parsing, demonstrating good adaptability to changes in layout and complex tables. It also parses unstructured documents such as PDFs into Markdown format, preserving the original document's heading hierarchy, lists, tables, formulas, and other structures without requiring manual formatting. It ensures that every piece of data in the table corresponds one-to-one with the row and column positions in the original PDF, thus restoring the content correspondence.

[0077] Step S150: Call the multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file. Add a timestamp to the structured file to obtain a structured time series file, and store it in the historical report database for use in different analysis scenarios.

[0078] For example, the multi-agent parsing chain is a LangChain-based multi-agent parsing chain, in which multiple specialized intelligent agents work collaboratively to complete the parsing task of complex documents / data. Each agent is responsible for a sub-task in the parsing process (such as format recognition, content extraction, structure regularization, and knowledge verification). Through LangChain's agent scheduling, tool invocation, and memory mechanism, the division of labor and cooperation are achieved, ultimately outputting high-quality structured parsing results. The parsing results are output in a structured CSV format, and after adding a timestamp according to the report's publication date (reference_date), they are automatically archived to the corresponding pdf, md, and csv directories in the historical report database for continuous accumulation, long-term maintenance, and audit traceability.

[0079] The above-described page data extraction method involves: obtaining report links from web pages to form a candidate report list; comparing the candidate report list with a local historical report database to select new reports; using an automated testing tool to load and render the web page content corresponding to the new reports, generating a PDF file; using an optical character recognition tool to parse the layout of the PDF file and convert it into a Markdown file; and using a multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file. A timestamp is then added to the structured file to obtain a structured time-series file, which is stored in the historical report database for use in different analysis scenarios. Through the synergistic effect of these tools, automated access, batch downloading, and conversion of reports from web pages are achieved, replacing manual processing and significantly reducing manual work time and improving data acquisition efficiency.

[0080] In an exemplary embodiment, step S130 calls an automated testing tool to load and render the webpage content corresponding to the new report link, generating a PDF file, including: configuring corresponding parameters for generating a PDF file according to the file type of the new report; the parameters include at least one of rendering waiting parameters and page layout parameters of the PDF file; calling the automated testing tool to load and render the webpage content corresponding to the new report, and generating a PDF file according to the configured parameters.

[0081] Among them, rendering wait parameters include rendering wait time, and page layout parameters include paper size, page margins, and other parameters.

[0082] For example, different data sources result in different file types for the new reports. Therefore, different parameters can be configured for different file types. Then, the automated browser environment built by the automated testing tool can access the webpage corresponding to the new report, fully load the details page of the new report containing dynamically rendered content, and generate a PDF file according to the set paper size, margins, and rendering wait time parameters.

[0083] In this embodiment, by configuring the corresponding parameters for generating PDF files according to the file type of the newly added report, the high-quality presentation of web page content as PDF files is ensured.

[0084] In one exemplary embodiment, such as Figure 2 As shown, step S150 calls the multi-agent parsing chain to perform structured analysis on the Markdown file, obtaining a structured file, including:

[0085] Step S151: Call the multi-agent parsing chain to perform structured analysis on the Markdown file and obtain structured information; the structured information includes at least key fields, including the report's release date, monitoring week, monitoring system, and pathogen detection data;

[0086] Step S152: Output the structured information as a CSV file to obtain the structured file.

[0087] For example, a multi-agent parsing chain is invoked to perform structured analysis on the Markdown file, automatically identifying key fields such as the report's publication date, monitoring week, monitoring system (e.g., ILI, SARI), and pathogen monitoring data (e.g., pathogen name, detection volume, and positivity rate), as well as structured information such as tables and titles. The extracted structured information is then output as a CSV file.

[0088] This embodiment performs structured analysis on the Markdown file and converts it into a CSV file for output. CSV files are a universal lightweight data exchange format that can be imported and exported by most data processing, visualization, and database tools, making them more suitable for cross-scenario transfer.

[0089] In an exemplary embodiment, step S151 performs structured analysis on the Markdown file to obtain structured information, including: if the report corresponding to the Markdown file is a normal report, a rule-based parsing strategy is used to obtain structured information; if the report corresponding to the Markdown file is an abnormal report, after parsing using the rule-based parsing strategy, a large language model is enabled to correct the parsing result to obtain structured information.

[0090] For example, for reports with normal formatting, a rule-based parsing strategy can be used to obtain structured information. This can be achieved through methods such as table location, title recognition, and row / column matching. For reports with anomalies, such as cross-page tables, misaligned lines, or unstable extraction, after parsing using a rule-based strategy, a Large Language Model (LLM) enhancement mode is activated. This mode performs boundary correction, content completion, and format correction based on semantic relationships, and the corrected result is used as the obtained structured information.

[0091] In this embodiment, normal reports are directly parsed using a rule-based parsing strategy. For reports with anomalies, after parsing using the rule-based parsing strategy, the parsing results are further corrected using a large language model to improve the overall robustness and accuracy of the parsing.

[0092] In an exemplary embodiment, step S150 adds a timestamp to the structured file to obtain a structured time-series file and stores it in the historical report database, including: adding a timestamp to the structured file according to the publication date of the report corresponding to the structured file to obtain a structured time-series file; storing the structured time-series file in the historical report database; and merging the structured time-series file with the relevant structured time-series files in the historical report database.

[0093] Furthermore, the structured time series files are merged with the relevant structured time series files in the historical report database. This includes: matching the week corresponding to the release date of the report corresponding to the structured time series file, and creating a multi-level file directory based on the reference date; uniformly sorting the structured time series files and the relevant structured time series files in the historical report database, and removing duplicate files based on the unique key.

[0094] For example, after the structured extraction of this report is completed, the structured files are stored in the historical report database according to the publication date of the corresponding report. Specifically, a hierarchical directory can be automatically created according to the week corresponding to the publication date and the reference date, and the files are uniformly sorted and duplicate rows are removed based on the unique key. This ensures that all monitoring indicators maintain consistency and integrity across years, multiple version updates, and format adjustments. The update source is automatically identified to avoid duplicate formatting, manual input errors, and historical data overwriting issues. At the same time, the local historical report database will also be updated with the processing status to ensure the continuity and controllability of subsequent incremental analysis.

[0095] This embodiment automatically creates hierarchical directories by week and reference date, performs difference processing on new reports, and automatically merges the parsed structured files with historical data, performing deduplication and sorting during the merging process. This mechanism ensures the continuity and maintainability of long-term data while avoiding errors and delays caused by manual integration.

[0096] In one exemplary embodiment, the method further includes: outputting a structured time series file through a standardized data output interface; the data output interface is compatible with various prediction model frameworks and supports the generation of visualization graphics.

[0097] For example, a standardized data output interface can be provided, directly adapting to the predictive model framework and supporting the generation of visualization results such as trend charts, positivity rate change curves, and monitoring system comparison charts, facilitating rapid access in scenarios such as epidemic surveillance, predictive model training, sentinel operation evaluation, and automated assessment. Through a built-in interactive trend chart generation tool, it can be directly used for monitoring reports and predictive system visualization, improving the intuitive understanding of data users and the convenience of report generation.

[0098] In an exemplary embodiment, the method further includes: defining each task implementing the page data extraction method as a directed acyclic graph using a workflow orchestration and scheduling tool, so as to achieve automated scheduling of each task.

[0099] For example, the workflow orchestration and scheduling tool is Airflow. By using Airflow, tasks such as report link acquisition, web page rendering, PDF conversion, Markdown parsing, structured extraction and incremental archiving are linked together and defined as a Directed Acyclic Graph (DAG) to achieve automated task scheduling, including scheduling of various tools such as scheduling crawler tools, automated testing tools, optical character recognition tools and multi-agent parsing chains.

[0100] The entire process of extracting page data in this application is scheduled and connected by Airflow, enabling report link acquisition, webpage rendering, PDF conversion, Markdown parsing, structured extraction and incremental archiving to be automatically triggered and managed in a unified DAG, thus forming a stable, monitorable and scalable report data processing pipeline.

[0101] To more clearly illustrate the page data extraction method provided in the embodiments of this application, the following further explains this solution.

[0102] This application employs an Airflow-based, fully automated data synchronization and structured processing strategy, which can be used to achieve efficient collection, parsing, and incremental updates of sentinel hospital surveillance reports. (Reference) Figure 3The specific process is as follows:

[0103] (1) Obtaining new report links. The system first performs a full scan of the monitoring report pages of sentinel hospitals using a web crawler (such as the Firecrawl map API), automatically extracts all current report links, and compares them with the local historical report database to identify new reports that have not yet been processed, thus achieving incremental management. Based on this, the Airflow scheduler automatically triggers data acquisition tasks according to a preset cycle, adds new reports to the pending queue, avoids repeated crawling and parsing, and ensures stable execution of tasks through concurrency control and abnormal retry mechanisms.

[0104] (2) In the data acquisition stage, the system uses the automated testing tool (Playwright) to build an automated browser environment, fully render the webpage of each new report, and generate PDF files according to the set paper size, margins and rendering waiting time, so as to ensure that the table and text content are fully presented.

[0105] (3) Since the API of the optical character recognition tool (MinerU) that parses the PDF file does not support direct file upload from the local machine, an object storage service (OSS) is used as an intermediary. The generated PDF file is first uploaded to OSS, and the corresponding uniform resource locator (URL) and signature (access permission) of the PDF file are sent to the API of the optical character recognition tool (MinerU) for optical character recognition.

[0106] (4) PDF files are converted to Markdown format via the MinerU API. The system can automatically identify table boundaries, text content, and date areas, restoring the original structured information and laying a stable foundation for subsequent parsing. For reports with different formats and layouts, the system can choose between rule-based parsing or LLM enhancement mode. It uses LangChainAgent to automatically extract fields such as report date, monitoring week, monitoring type (ILI / SARI), pathogen name, detection volume, and positivity rate. When parsing complex tables or abnormal layouts, LLM enhancement mode can perform semantic correction to improve data accuracy.

[0107] (5) After parsing, the system archives the PDF, Markdown and CSV files into the hierarchical directory according to the reference date (reference_date), and performs incremental merging of the parsing results. It automatically compares, sorts and removes duplicates with the historical data table to ensure that the data is maintainable, complete and consistent in the long term.

[0108] Throughout the process, Airflow DAG is responsible for the unified scheduling of web scraping, PDF conversion, Markdown parsing, and incremental update tasks, enabling newly added weekly reports to be automatically processed and updated to standardized CSV output files, while also updating the local historical report database, achieving fully automated management across the entire chain.

[0109] The aforementioned processes can be modularized and automated, exhibiting strong adaptability in data source expansion and multi-modal processing. Furthermore, through modular design, each functional node, including web scraping, PDF generation, Markdown conversion, structured parsing, and incremental updates, can be independently configured and reused, enabling the system to adapt to monitoring reports from different organizations or in different formats. In addition, by adjusting the parsing strategy of the LangChai agent's prompts or the large language model enhancement mode, semantic-level parsing of tables and text information from different data sources and report formats can be achieved, realizing standardized field mapping and structured extraction. This eliminates the need to modify extensive data extraction code or frequently adapt to updates to data source website HTML tags, significantly reducing maintenance costs and development workload.

[0110] In terms of automated scheduling, this implementation fully utilizes the existing Airflow DAG framework, incorporating newly added data sources and reporting modules into a unified scheduling system. This enables scheduled data capture, incremental parsing, and data merging and updating without the need for additional complex scheduling logic. Compared to traditional single-data-source interfaces or hard-coded parsing methods, which typically only handle data from fixed sources, developers must manually modify the capture and parsing code if the data source webpage structure changes or new data sources are added, increasing operational complexity and the risk of errors. In contrast, this system, through modular and parameterized design, achieves flexible parsing and standardized output of multi-source data, maintaining data integrity, accuracy, and long-term maintainability. This provides a stable and efficient technical foundation for the construction of multi-module, multi-data-source monitoring systems, while significantly improving system scalability and adaptability.

[0111] Traditional manual methods require operators to download web pages one by one, generate PDFs, manually identify table content, and manually enter it into CSV files. The entire process is time-consuming and prone to data omissions and entry errors, making it difficult to ensure consistency across reports and multi-pathogen data. This implementation method, through automated modular design and incremental update mechanisms, achieves fully automated processing of the entire process, including automatic identification of report links, web page scraping, PDF to Markdown conversion, structured parsing, and CSV merging. This significantly improves data processing efficiency, ensures the integrity, accuracy, and traceability of monitoring data, and can directly adapt to the needs of epidemiological prediction models and visualization analysis, meeting the stability requirements of long-term monitoring and scientific research applications.

[0112] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0113] Based on the same inventive concept, this application also provides a page data extraction device for implementing the page data extraction method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more page data extraction device embodiments provided below can be found in the limitations of the page data extraction method described above, and will not be repeated here.

[0114] In one exemplary embodiment, such as Figure 4 As shown, a page data extraction device is provided, comprising:

[0115] Link acquisition module 410 is used to acquire report links in web pages and form a candidate report list;

[0116] The incremental detection module 420 is used to compare the candidate report list with the local historical report database and filter out new reports from the candidate report list;

[0117] The document generation module 430 is used to call the automated testing tool to load and render the web page content corresponding to the new report, and generate a PDF file.

[0118] The file conversion module 440 is used to call an optical character recognition tool to parse the layout of a PDF file and convert it into a Markdown file;

[0119] The document analysis module 450 is used to call the multi-agent parsing chain to perform structured analysis on Markdown files, obtain structured files, add timestamps to the structured files to obtain structured time-series files, and store them in the historical report database for use in different analysis scenarios.

[0120] In one embodiment, the file generation module 430 is configured to configure corresponding parameters for generating PDF files according to the file type of the new report; the parameters include at least one of rendering waiting parameters and page layout parameters of the PDF file; and call an automated testing tool to load and render the web page content corresponding to the new report, and generate a PDF file according to the configured parameters.

[0121] In one embodiment, the document analysis module 450 is used to call the multi-agent parsing chain to perform structured analysis on the Markdown document and obtain structured information; the structured information includes at least key fields, including the report's release date, monitoring week, monitoring system, and pathogen detection data; the structured information is output as a CSV file to obtain a structured document.

[0122] In one embodiment, the document analysis module 450 is further configured to: if the report corresponding to the Markdown file is a normal report, use a rule-based parsing strategy to obtain structured information; if the report corresponding to the Markdown file is an abnormal report, after parsing using the rule-based parsing strategy, enable a large language model to correct the parsing result and obtain structured information.

[0123] In one embodiment, the device further includes a file storage module, used to add a timestamp to the structured file according to the publication date of the report corresponding to the structured file to obtain a structured time-series file; store the structured time-series file in a historical report database; and merge the structured time-series file with the relevant structured time-series files in the historical report database.

[0124] In one embodiment, the file storage module is further configured to match the week corresponding to the publication date of the report corresponding to the structured time series file, and establish a multi-level file directory in conjunction with the reference date; uniformly sort the structured time series file and the related structured time series file in the historical report database, and remove duplicate files based on the unique key.

[0125] In one embodiment, the apparatus further includes a file output module for outputting structured time-series files through a standardized data output interface; the data output interface is compatible with various prediction model frameworks and supports the generation of visualization graphics.

[0126] In one embodiment, the apparatus further includes an automatic scheduling module, which defines each task implementing the page data extraction method as a directed acyclic graph using workflow orchestration and scheduling tools, so as to achieve automated scheduling of each task.

[0127] Each module in the aforementioned page data extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0128] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a page data retrieval method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0129] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0131] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0132] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0133] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0135] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of extracting page data, characterized by, The method includes: Extract report links from web pages to create a candidate report list; The candidate report list is compared with the local historical report database, and new reports are selected from the candidate report list; The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, generating a PDF file. The PDF file is parsed using an optical character recognition tool to convert it into a Markdown file; The Markdown file is analyzed in a structured manner by calling the multi-agent parsing chain to obtain a structured file. A timestamp is added to the structured file to obtain a structured time series file, which is then stored in the historical report database for use in different analysis scenarios.

2. The method of claim 1, wherein, The process of calling the automated testing tool to load and render the webpage content corresponding to the newly added report, and generating a PDF file, includes: Configure the corresponding parameters for generating PDF files according to the file type of the newly added report; the parameters include at least one of rendering waiting parameters and PDF file page layout parameters. The automated testing tool is invoked to load and render the webpage content corresponding to the newly added report, and a PDF file is generated according to the configured parameters.

3. The method of claim 1, wherein, The process of calling the multi-agent parsing chain to perform structured analysis on the Markdown file to obtain a structured file includes: The multi-agent parsing chain is invoked to perform structured analysis on the Markdown file to obtain structured information; the structured information includes at least key fields, including the report's release date, monitoring week, monitoring system, and pathogen detection data; The structured information is output as a CSV file to obtain a structured file.

4. The method of claim 3, wherein, The step of performing structured analysis on the Markdown file to obtain structured information includes: If the report corresponding to the Markdown file is a normal report, a rule-based parsing strategy is used to obtain the structured information; If the report corresponding to the Markdown file is an anomaly report, then after parsing using a rule-based parsing strategy, a large language model is used to correct the parsing results to obtain the structured information.

5. The method of claim 3, wherein, The step of adding timestamps to the structured file to obtain a structured time-series file and storing it in the historical report database includes: Add a timestamp to the structured file according to the publication date of the report corresponding to the structured file to obtain a structured time-series file; The structured time series file is stored in the historical report database, and the structured time series file is merged with the relevant structured time series file in the historical report database.

6. The method of claim 5, wherein, The step of merging the structured time-series file with the relevant structured time-series file in the historical report database includes: Based on the release date of the report corresponding to the structured time series file, match the week corresponding to the release date, and create a multi-level file directory by combining the reference date; The structured time series files and related structured time series files in the historical report database are sorted uniformly, and duplicate files are removed based on unique keys.

7. The method of claim 1, wherein, The method further includes: The structured time series file is output through a standardized data output interface; the data output interface is compatible with various prediction model frameworks and supports the generation of visualization graphics.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: By using workflow orchestration and scheduling tools, each task implementing the page data extraction method is defined as a directed acyclic graph to achieve automated scheduling of each task.

9. A page data extraction apparatus characterized by comprising: The device includes: The link retrieval module is used to retrieve report links from web pages and form a candidate report list. The incremental detection module is used to compare the candidate report list with the local historical report database and filter out new reports from the candidate report list; The file generation module is used to call automated testing tools to load and render the web page content corresponding to the newly added report, and generate a PDF file. The file conversion module is used to call an optical character recognition tool to parse the layout of the PDF file and convert it into a Markdown file; The document analysis module is used to call the multi-agent parsing chain to perform structured analysis on the Markdown file, obtain a structured file, add a timestamp to the structured file to obtain a structured time series file, and store it in the historical report database for use in different analysis scenarios. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images