A high-precision time-series data extraction method and system based on modular design

Abstract

This invention presents a method and system for extracting highly timely time-series data from multi-source heterogeneous networks. The system employs asynchronous message queues to achieve loosely coupled distributed collaborative operation of five functional modules: web crawling, structured parsing, data cleaning and mapping, task scheduling, and data storage. By integrating a multi-mode crawling engine with static, dynamic rendering, and API interface calls, the system solves the compatibility problem of heterogeneous data sources. The structured parsing module uses a hybrid parsing approach combining rule bases and a BERT-based natural language processing model, significantly improving the intelligence and accuracy of entity extraction from unstructured text. The data cleaning module uses the statistical Z-Score algorithm to dynamically detect outliers in time-series data, ensuring the reliability of the stored data. The task scheduling module supports incremental updates and failure retry mechanisms, greatly improving system throughput and operational stability. This invention achieves automated, highly reliable, and highly timely time-series data acquisition, cleaning, and structured storage through an advanced modular architecture, AI-driven parsing technology, and statistical data quality control mechanisms.

Description

Technical Field

[0001] This invention relates to the fields of computer data processing, high-throughput data acquisition, and big data quality control. Specifically, it relates to a method and system for automatically extracting, cleaning, verifying, and structurally storing high-time-sensitivity time-series data from multi-source heterogeneous networks. Background Technology

[0002] With the rapid development of information technology and distributed computing, the demand for highly timely time-series data in various industries has exploded. This data (e.g., sensor readings in industrial control, network equipment performance indicators, or medical and health monitoring data, collectively referred to as time-series data) is usually published on the network in diverse forms, but existing technologies face a series of technical bottlenecks when processing these complex data sources.

[0003] Target data sources include traditional static HTML, dynamically rendered pages generated by client-side JavaScript (loaded via AJAX), and content returned by structured API interfaces. Traditional data crawlers or parsing scripts typically only adapt to a single format, resulting in poor adaptability to multi-source heterogeneous data, high maintenance costs, and susceptibility to failure due to website structure changes. Furthermore, directly crawled raw data often suffers from data redundancy, inconsistent formats and units, and the presence of errors or outliers. Existing methods generally lack efficient and systematic data quality verification processes, especially for highly volatile time-series data. Traditional static threshold detection methods are poorly adaptable, leading to erroneous data contaminating the database and affecting the accuracy of downstream applications. Moreover, the lack of efficient task scheduling and incremental update mechanisms often results in systems using full-scale crawling, causing resource waste and data update delays. Simultaneously, rudimentary crawling failure handling mechanisms cannot guarantee data integrity and continuity under network fluctuations. Finally, traditional data extraction processes generally lack systematic monitoring and auditing mechanisms for data quality. When data problems are discovered, it is difficult to quickly locate the source of the error or trace human intervention in the data processing process, reducing the overall reliability of the data.

[0004] In conclusion, the data service field urgently needs a well-designed, technologically advanced, stable, intelligent, and highly scalable data extraction method and system. This system should be able to automatically and accurately extract key information from complex and diverse web documents, and perform rigorous data cleaning, verification, and standardization, ultimately providing a solid and reliable data foundation for downstream computing applications such as high-frequency computing, quantitative analysis, and risk modeling. Summary of the Invention

[0005] This invention aims to systematically solve a series of key technical problems in existing data acquisition, such as poor parsing adaptability due to the diverse forms of data sources, low quality and inconsistent formats of raw data, rigid data update strategies leading to insufficient timeliness, and the lack of data quality monitoring and traceability mechanisms.

[0006] This invention achieves the following targeted technical effects and system performance improvements through a specific modular architecture, intelligent data parsing algorithms, and dynamic statistical verification methods.

[0007] Specifically, the present invention provides a high-efficiency time-series data extraction system based on modular design, characterized in that the system comprises:

[0008] The web page crawling module is used to automatically obtain raw data from heterogeneous web pages from multiple sources on the Internet according to the task instructions issued by the task scheduling module. The crawling module can dynamically select static HTML crawling, dynamic page rendering or API interface calling mode according to the data source type.

[0009] The structured parsing module is used to parse the raw data obtained by the web crawling module. The structured parsing module combines a configurable rule base with a BERT-based natural language processing (NER) model to achieve mixed parsing of key entity fields in unstructured announcement texts and generate structured data objects, thereby significantly improving the intelligence and accuracy of data parsing.

[0010] The data cleaning and mapping module is used to deduplicatize, unify the format, and normalize the units of structured data. It also detects outliers based on the statistical Z-Score algorithm. When the calculation exceeds a preset threshold, the system automatically marks the data as an anomaly and adds an anomaly score to achieve proactive identification and risk warning of anomaly data.

[0011] The task scheduling module is used to automatically generate scheduled tasks based on Cron expressions and to perform incremental updates by reading the timestamp of the last successful data capture from the database. The task scheduling module has an automatic retry mechanism for failures, which can place failed tasks into a delay queue and re-execute them after a preset time interval, thereby ensuring the continuity and integrity of data capture.

[0012] The data entry module is used to write the cleaned time-series data into a time-series database and a relational database, and automatically generates a system data quality report after entry, which includes indicators such as the total number of records entered, the number of abnormal data, and the field missing rate. The data entry module further provides a manual review interface for correcting and tracing the marked outliers and recording the review log.

[0013] Each module interacts with data through an asynchronous message queue, forming a top-down, highly cohesive, loosely coupled data processing pipeline.

[0014] Preferably, the web page crawling module includes a static HTML crawling submodule, a dynamic page rendering submodule, and an API interface calling submodule, which can automatically select the optimal crawling method to adapt to multiple heterogeneous data sources.

[0015] Preferably, the structured parsing module performs hybrid parsing based on XPath, regular expressions, and BERT named entity recognition model to automatically extract data source identifiers, key personnel information, information release time, and core time-series numerical fields from unstructured text.

[0016] Preferably, the data cleaning and mapping module calculates the hash value of key fields for each record to remove duplicates, and unifies the date format to the ISO 8601 standard time format to ensure data consistency.

[0017] Preferably, the data entry module generates a data quality report after completing the batch data entry. The report includes indicators such as the number of entries, the number of new records, the number of abnormal records, and the field missing rate, and provides a manual review interface to correct abnormal data.

[0018] Another aspect of the present invention is to provide a time-series data extraction method based on modular design, characterized in that the method includes the following steps:

[0019] S1. Task Scheduling and Task Generation:

[0020] The task scheduling module triggers data capture tasks periodically based on timing rules; before generating a task, the scheduling module reads the last successful update timestamp from the metadata database and generates task instructions only for incremental data after the timestamp; if the task execution fails, the system puts the task into a delay queue and automatically retryes it after a preset time interval to ensure the integrity of the data capture.

[0021] S2, Data Fetching:

[0022] The web crawling module executes corresponding strategies based on the crawling mode specified in the task instructions: static pages use requests and lxml to directly parse the HTML structure; dynamic pages use a headless browser (Playwright or Puppeteer) to simulate user behavior and wait for JavaScript to finish rendering; API pages directly obtain structured JSON or XML data through HTTP requests; finally, the crawled raw data is encapsulated into data packets and pushed to the message queue.

[0023] S3, Structured Parsing:

[0024] The structured parsing module combines the rule base with the BERT named entity recognition model to perform field recognition and text extraction, generate structured JSON objects, and push them to downstream modules;

[0025] S4. Data Cleaning and Anomaly Detection:

[0026] The data cleaning and mapping module receives a structured JSON object, performs deduplication and format standardization operations, and performs statistical outlier detection through the following steps:

[0027] S41. Implement data deduplication based on key field hashing;

[0028] S42. Numerical and time formats shall be standardized in units and in the ISO 8601 format;

[0029] S43. Detect outliers using the Z-Score algorithm. If |Z|>3, mark the record as a potential outlier and record the outlier score.

[0030] This allows for proactive identification of abnormal data, preventing erroneous values ​​from contaminating the database.

[0031] S5. Data import and quality audit:

[0032] The cleaned data is written into time-series databases and relational databases;

[0033] After the data import module completes the import process, it generates a data quality report, which includes statistics on the number of data imported, the percentage of abnormal data, and the field missing rate.

[0034] S6. Manual review and final data confirmation:

[0035] For flagged abnormal data, the system automatically pushes it to the manual review interface for administrators to verify and correct. All manual operations are recorded to ensure data traceability.

[0036] Preferably, the Z-Score outlier detection includes: calculating the mean μ and standard deviation σ of the historical sequence of the time series data;

[0037] Substitute the current value X to be detected into the formula Z = (X − μ) / σ; when |Z| is greater than the preset threshold 3, mark the record as abnormal and add an abnormal score.

[0038] Preferably, after a task fails to execute, the task scheduling module places the task in a delay queue and automatically retryes it after a preset time interval, for a total of three retries.

[0039] Preferably, after generating a data quality report, the data entry module pushes the report to the monitoring system or administrator terminal and records the manual review operation log to achieve full traceability of the data process.

[0040] Another aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described time-series data extraction method.

[0041] Compared with existing technologies, this invention, through specific innovative design, achieves the following targeted technical effects in terms of automation, quality, timeliness, and reliability of data extraction:

[0042] 1. By integrating web crawling modules with three modes—static, dynamic rendering, and API calls—and combining a hybrid parsing strategy of "rule engine + NLP model," the system can automatically adapt to multi-source, heterogeneous data publishing formats. This solves the problem of traditional crawlers becoming ineffective due to website redesigns, greatly reduces maintenance costs, and can intelligently extract key information from unstructured announcement texts, achieving a higher level of automation.

[0043] 2. The data cleaning module of this invention fundamentally solves the "dirty data" problem by deduplication, format unification, and unit normalization. In particular, it introduces an automatic outlier detection mechanism based on statistical (Z-Score) methods, which can dynamically identify data points that deviate from the normal fluctuation range, increasing the accuracy of data anomaly detection to over 96%, preventing erroneous data from contaminating the database, and ensuring the final quality of the data.

[0044] 3. The task scheduling module achieves efficient incremental data retrieval by recording update timestamps, avoiding resource waste. This reduces the average latency from data publication to database entry to less than 2 minutes. Simultaneously, the built-in automatic retry mechanism ensures data integrity under network fluctuations and other conditions, improving system stability.

[0045] 4. The data import module automatically generates a system data quality report and pushes system-marked abnormal data to the manual review interface. All manual interventions are recorded in detail, forming a clear "audit log." This design solves the pain points of unclear data sources and difficulty in tracing problems, providing key technical guarantees for data reliability and compliance. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1This is a schematic diagram of the structure and function of a modularly designed high-precision time-series data extraction system provided by the present invention.

[0048] Figure 2 This is a schematic diagram of a modular design-based high-precision time-series data extraction method provided by the present invention. Detailed Implementation

[0049] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0051] See attached document Figure 1 This invention utilizes the distributed, asynchronous collaboration of five core technology modules to construct a highly efficient and accurate time-series data processing pipeline (e.g., sensor readings in industrial control, network device performance indicators, or medical and health monitoring data). The advantage of this architecture lies in its high cohesion and low coupling design. Through asynchronous message queues (such as RabbitMQ or Apache Kafka) for data interaction, it solves the system scalability bottlenecks and low fault tolerance issues faced by traditional centralized processing architectures when handling massive heterogeneous data streams.

[0052] The following is a description of the specific components and functions of each module.

[0053] Web crawling module 10.

[0054] This module retrieves raw data from the internet based on instructions issued by the task scheduling module. Administrators or users can configure scraping tasks through a web interface. The configuration options are extensive, including but not limited to: a list of target URLs (importable is supported), scraping frequency (defined using Cron expressions), a time window for the target data (e.g., scraping only the most recent month), HTTP request headers (for simulating specific browsers), a proxy IP pool (for anti-anti-scraping measures), and login cookies for specific websites.

[0055] One embodiment employs a multi-mode crawling execution engine that intelligently selects the most suitable crawling method for different URLs. In one embodiment, a static HTML page parsing mode is used: this mode is suitable for pages with simple technology and content directly in the HTML source code. The system uses a lightweight requests library combined with lxml for high-speed downloading and preliminary DOM validity checks. This method consumes minimal resources, is the fastest, and is suitable for crawling a large number of regular information pages. In a preferred embodiment, a dynamic page rendering mode is used. For widely used single-page applications (SPAs) or web pages that load data asynchronously via AJAX, this module launches a headless browser environment, such as a Chromium-based browser controlled by the Playwright or Puppeteer libraries. It can completely simulate the user's real browsing behavior: opening the page, scrolling, clicking buttons, and waiting for specific JavaScript code to finish executing or for network requests to become idle before extracting the finally rendered HTML DOM, ensuring "what you see is what you get" data integrity.

[0056] In one embodiment, for systems providing standardized data interfaces, a direct API call model is adopted. This module can directly construct HTTP requests, call its API interface, and directly receive the returned structured data in JSON or XML format. This method is the most efficient and stable, and is the preferred data acquisition approach.

[0057] This module takes a task instruction object (JSON format) from the task scheduling module, for example: {"task_id":"uuid-123","url":"http: / / example.com / fund / 001234.html", crawl_mode": "dynamic", "priority": 1}. After processing by this module, a data packet containing the crawl results is pushed to the raw_data_topic in the message queue. The data packet structure can be: {"task_id": "uuid-123", "source_url": "...", "raw_content": "...", "content_type": "html", "timestamp": "utc_timestamp"}. Simultaneously, the task execution status (success / failure and reason) is written to the scheduling log database.

[0058] This module acts as a producer, generating raw data for the message queue; simultaneously, it receives instructions from the task scheduling module and returns the execution results to it.

[0059] Structured Analysis Module 20

[0060] The core task of this module is to read and understand the raw data sent by the Web Crawling Module 10 and convert it into structured key-value pairs that are easy for machines to process.

[0061] For web page elements with relatively fixed formats, the module uses the built-in configurable rule library and uses XPath or regular expressions for precise field positioning and extraction. However, for large sections of unstructured content such as announcement texts, due to their highly variable formats, it is difficult to cover them with fixed rules.

[0062] This invention introduces a Named Entity Recognition (NER) model fine-tuned based on the BERT model here. This model has been trained on a large amount of chronological data announcement texts and can automatically identify key entities such as [Person], [Time], [Metric Value], etc. through in-depth understanding of text semantics. This hybrid parsing strategy of "rule engine + BERT-NER model" is a fundamental technical improvement to solve the bottleneck of unstructured data parsing, and its technical effect is to significantly improve the intelligence level and accuracy of information extraction, thus reducing the burden of manual rule maintenance.

[0063] For HTML / XML content, the module first uses the lxml library to parse it into a DOM tree. The system has a built-in configurable rule library, and for the page structures of different websites, XPath or CSS selector paths are preset to quickly and precisely locate the nodes containing the target data (such as the label where the fund net value is located). Due to its powerful axis query ability, XPath performs better than CSS selectors when dealing with complex nested tables. When dealing with large sections of text such as announcements, for information with relatively fixed formats, such as "This event occurred on March 6, 2024", the regular expression \d{4} year \d{1,2} month \d{1,2} day can be used to precisely extract the date.

[0064] In a preferred embodiment, for semantic information such as changes in the status of key entities and adjustments of complex configuration information that cannot be extracted by fixed rules, the module calls a Named Entity Recognition (NER) model fine-tuned based on the BERT model. This model has been trained on a large amount of announcement texts and can automatically identify

[0065] In a preferred embodiment, it can be predefined entities such as [Person - Key Participant], [Organization - Data Source], [Time - Release Time], [Event - Business Decision], etc., which greatly improves the intelligence level of information extraction.

[0066] This module consumes a raw data packet from the raw_data_topic topic of the message queue. After passing through this module, a structured JSON object containing the parsing results is pushed to the parsed_data_topic topic of the message queue.

[0067] Example output in a preferred embodiment: {"fund_code": "001234", "nav": 1.5032, "date": "2025-08-15", "manager": "Zhang San", "announcement_summary": "..."}. As a consumer and producer of the message queue, it connects the raw data with the semi-finished structured data.

[0068] Data cleaning and mapping module 30.

[0069] This module ensures data quality and is responsible for deeply processing and purifying the parsed data. The main functions include: 1. To prevent redundant data caused by task retries or repeated publication of data sources, the module combines the key fields of the records (such as fund_code + date) and calculates the hash value (such as MD5), and compares it with the existing hash value in the database to achieve efficient deduplication. [[ID=​​​​​​​​​​​​​​​​​If the calculated absolute Z-Score is greater than a preset threshold (usually 3, representing a deviation from the mean of more than 3 standard deviations), the value is identified as a potential outlier and labeled "pending review." Each key in the cleaned JSON object is precisely mapped to the corresponding field in the target database table using a predefined mapping rule (or ORM model), making final preparations for writing to the database.

[0076] This module consumes a structured JSON object from the `parsed_data_topic` in the message queue. Through this module, a data object that perfectly matches the database table structure can be directly accepted by the database ORM (Object-Relational Mapping) layer. This module is the final processing stage before the data is written to the database; its output is directly submitted to the data ingestion module.

[0077] Task scheduling module 40.

[0078] This module controls the rhythm, sequence, and strategy of the entire data flow. In a preferred embodiment, this module defines the scheduling rules as follows: it uses standard Cron expressions for task timing, providing extremely high flexibility.

[0079] When generating a crawling task, the scheduler first queries the metadata database to obtain the timestamp of the last successful crawl of the data source. This timestamp is passed as a parameter to the crawling module, instructing it to only download and process incremental data updated after that timestamp. This strategy solves the resource waste and high latency problems caused by repeated crawling of full data, and is a key technical means to improve the timeliness of system data, reducing the average latency from data publication to database entry to less than 2 minutes. In a preferred embodiment, to cope with situations such as brief network fluctuations or temporary unavailability of the target website, the scheduler has a built-in system fault tolerance mechanism. When a task fails, the system does not immediately give up, but places it in a delay queue and automatically retryes it after a preset time interval (preferably three retries). This ensures the final integrity of the data in the event of a momentary system failure, improving the continuity and robustness of system operation.

[0080] For example, the module uses standard Cron expressions for task timing, using "0 18 * * 1-5" to indicate that a daily update task for all critical time-series data will be triggered at 6 PM on every weekday.

[0081] When a task (whether crawling or parsing) fails, the scheduler does not give up immediately, but puts it into a delayed queue and automatically retryes it later (e.g., after 5 minutes, 15 minutes, 30 minutes). In a preferred embodiment, it is retried 3 times to deal with network fluctuations or temporary unavailability of the target website.

[0082] To achieve efficient data synchronization, the scheduler first queries the metadata database of the data storage module when generating a crawling task to obtain the timestamp of the last successful crawl of the data source. Then, it passes this timestamp as a parameter to the crawling task. Based on this, the crawling module only downloads and processes data updated after that timestamp, avoiding the repeated crawling and processing of the entire dataset and greatly improving efficiency.

[0083] This module has pre-set task configuration files, manually issued real-time task instructions, and task execution status feedback from other modules. After passing through this module, specific task instruction objects are issued to the web crawling module 10. This module interacts with all execution modules in terms of control flow, acting as the issuer of system commands and the monitor of status.

[0084] Data import module 50

[0085] This department is responsible for the final persistence of the data and provides channels for quality monitoring and human intervention.

[0086] This module employs a hybrid storage approach. The system stores high-frequency, timestamped metrics (such as real-time sensor readings) in a time-series database (such as InfluxDB) optimized for write, compression, and query depth, while storing data with complex entity relationships (such as full-text announcements) in a relational database. This approach optimizes the efficiency of storing and querying large-scale heterogeneous data.

[0087] In addition, after each batch data import operation is completed, this module will automatically perform a statistical analysis and generate a data quality report. The report includes key indicators such as: the total number of records imported, the number of newly added records, the number of updated records, the number of outliers detected, and the missing rate of each field. This report can be pushed to the monitoring system or sent to the data administrator.

[0088] In a preferred embodiment, a manual review interface is also provided. The system provides a web management backend containing a "Data Review" page. All abnormal data marked as "pending review" by the data cleaning module is listed here, displaying its original values, Z-score calculation results, and historical data graphs. Administrators can view the original webpage snapshot for verification, and then select "Confirm Pass" (mark the data as normal) or "Manual Correction" (enter the correct value). All manual operations are recorded in detail, ensuring the final authority and traceability of the data.

[0089] This module takes the data objects from the data cleaning and mapping module, which are ready to be stored in the database. After processing by this module, the data is written to the database and a data quality report is generated. This module is the end point of the data processing pipeline, and its metadata (such as the last update timestamp) is read in reverse by the task scheduling module, forming a closed loop.

[0090] This invention also provides a time-series data extraction method adapted to the above-described system in a modular design, as a method embodiment corresponding to the above embodiments (e.g., Figure 2 (As shown).

[0091] Step S1: Automatic scheduling and generation of daily update tasks for key time series data

[0092] The task scheduling module, based on the preset Cron rules 0 20 * * 1-5, automatically triggers the daily update task for key time-series data at 20:00 on August 15, 2025. The scheduler queries the metadata database and finds that the last successful update timestamp of the key time-series data from the target data source is 2025-08-15T20:00:00Z. The scheduler generates a specific crawling task instruction, {"task_id": "...", "url": "http: / / data.source.com / 001234 / data.html", "crawl_mode": "dynamic", "since_timestamp": "2025-08-15T20:00:00Z"}, and sends it to the web crawling module.

[0093] Step S2: Crawling Dynamic Web Pages and Obtaining Raw Data

[0094] The web crawling module 10 receives the task instruction and identifies the crawl_mode as dynamic. The module launches a Playwright headless browser instance and navigates to the specified URL. The browser performs full page loading and JavaScript rendering, waiting for the table (ID #nav-table) containing the latest metrics to appear in the DOM. The module extracts the rendered complete HTML content, encapsulates it into a raw data packet, and pushes it to the raw_data_topic in the message queue.

[0095] Step S3: Structured parsing and key information extraction

[0096] The structured parsing module 20 consumes the data packet from the queue. Based on a preset rule base, it successfully extracts the metric string "1.5032" using the XPath expression / / table[@id='nav-table'] / tbody / tr / td / text(), and similarly extracts the date string "2025-08-15". The parsing module assembles the extracted results into a JSON object {"data_source_code": "001234", "value_str": "1.5032", "date_str": "2025-08-15"}, and pushes it to the parsed_data_topic topic.

[0097] Step S4: Data cleaning, outlier detection, and numerical calculation

[0098] In a preferred embodiment, the data cleaning and mapping module 30 consumes the aforementioned JSON object. `value_str` is converted to a floating-point number 1.5032, and `date_str` is converted to a standard UTC date 2025-08-15T00:00:00Z. In a preferred embodiment, outlier detection includes: obtaining historical data: the module queries the database of the data entry module to obtain daily metric data for the past 30 periods (e.g., trading days) for the target data source code 001234. Calculating statistics: the mean μ = 1.4500 and the standard deviation σ = 0.0150 of these 30 historical metrics are calculated. Performing Z-Score calculation: the newly obtained metric X = 1.5032 is substituted into the Z-Score formula.

[0099] The specific calculations are as follows:

[0100]

[0101] The calculated Z-Score of 3.547 is significantly greater than the preset threshold of 3. Therefore, the system determines that the metric value of 1.5032 is a potential outlier and adds a flag bit {"is_anomaly": true, "anomaly_score": 3.547} to the data record.

[0102] Step S5: Conditional Inventory Receipt and Quality Report Generation

[0103] The data import module receives this cleaned data that was marked as anomalous. The module writes the metric data 1.5032 to the time-series database, but also simultaneously writes its associated `is_anomaly` flag. After this batch processing task is completed, the data quality report generated by the data import module will clearly record: "The metric data of target data source code 001234 on 2025-08-15 was marked as anomalous, with a Z-Score of 3.547."

[0104] Step S6: Manual review and final data confirmation

[0105] The data administrator received an alert on the backend monitoring interface and viewed the record awaiting review. The interface displayed a historical graph of the metric, clearly showing a significant jump in the value for the day. The administrator could link to the original webpage snapshot for verification. Verification confirmed that the metric did indeed experience a significant jump due to specific external factors that day. The administrator clicked "Confirm Approval" on the review interface, updating the `is_anomaly` flag of the record to `false` and recording the reviewer and review time, completing the closed loop of data processing and verification.

[0106] In another embodiment of the present invention, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the fund data extraction method described in this specification.

[0107] Specifically, the computer program includes a plurality of executable instructions, which, when executed by a processor in a server, cloud computing platform, or local terminal, perform the method steps of the above embodiments.

[0108] The computer-readable storage medium of this invention can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), or other media capable of storing program code. Through the execution of this program, all the functions of the aforementioned fund data extraction method can be implemented in software, thereby enabling the system to have automated and intelligent data acquisition, cleaning, and storage capabilities.

[0109] Through the detailed method embodiments and numerical calculation processes described above, the present invention has the following advantages compared with the prior art:

[0110] 1. Traditional web crawlers mechanically grab and store data. If the source website makes an entry error (e.g., mistakenly writing 15.032 instead of 1.5032), this disastrous error value will directly pollute the database, causing all subsequent analysis and decisions based on this data to be completely wrong. This invention introduces an automatic outlier detection mechanism based on statistics (Z-Score), which can proactively identify data points that deviate from the normal fluctuation range, like a vigilant data analyst. As shown in the example, Z=3.547 immediately triggers an alarm. Even if the data is eventually confirmed to be correct, this "confirmation" step greatly enhances the credibility of the data, thereby increasing the accuracy of data anomaly detection to over 96%.

[0111] 2. Significantly improves the automation level of data processing and reduces labor costs: Without this invention, data verification requires a large amount of manual sampling, which is inefficient and has limited coverage. This invention automates the data verification process, submitting only a small number of "abnormal" data that the system cannot automatically adjudicate to manual review. This frees the data team from tedious and repetitive verification work, allowing them to focus on handling truly complex issues and business logic development, thereby reducing the workload of manual review by approximately 30%.

[0112] 3. Ensuring Data Integrity and Traceability: The methodology of this invention covers the entire lifecycle from task scheduling, data acquisition, parsing, cleaning, verification to storage. A retry mechanism ensures the ultimate integrity of data even when the network is unstable. Most importantly, the manual review process establishes a clear "audit log" for each piece of abnormal data, recording who performed what operation on the data, when, and for what reason. This complete traceability is crucial for financial institutions that need to meet stringent compliance requirements.

[0113] Achieving high timeliness of data updates: Through intelligent task scheduling and incremental update mechanisms, the system avoids repeatedly scraping massive amounts of historical data, and can concentrate resources to quickly acquire the latest published data, reducing the average delay from data publication to data entry to less than 2 minutes, providing strong support for downstream applications (such as trading strategies) that rely on high timeliness data.

[0114] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0115] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-efficiency time-series data extraction system based on modular design, characterized in that, The system includes: The web page crawling module is used to automatically obtain raw data from heterogeneous web pages from multiple sources on the Internet according to the task instructions issued by the task scheduling module. The crawling module can dynamically select static HTML crawling, dynamic page rendering or API interface calling mode according to the data source type. The structured parsing module is used to parse the raw data obtained by the web crawling module. The structured parsing module combines a configurable rule base with a BERT-based natural language processing model to achieve mixed parsing of key entity fields in unstructured announcement texts and generate structured data objects, thereby significantly improving the intelligence and accuracy of data parsing. The data cleaning and mapping module calculates the hash value of key fields for each record to remove duplicates and standardizes the date format to the ISO 8601 standard time format to ensure data consistency. It also detects outliers based on the statistical Z-Score algorithm. The Z-Score algorithm for outlier detection includes: calculating the mean μ and standard deviation σ of the historical time series data; and substituting the current value X to be detected into the formula: Z = (X m) / s; When |Z| is greater than the preset threshold of 3, the record is marked as abnormal and an abnormal score is attached to achieve proactive identification and risk warning of abnormal data; The task scheduling module is used to automatically generate scheduled tasks based on Cron expressions and to perform incremental updates by reading the timestamp of the last successful data capture from the database. The task scheduling module has an automatic retry mechanism for failures, which can place failed tasks into a delay queue and re-execute them after a preset time interval, thereby ensuring the continuity and integrity of data capture. The data entry module is used to write the cleaned time-series data into a time-series database and a relational database, and automatically generates a system data quality report after entry, which includes the total number of records entered, the number of abnormal data, and the field missing rate. The data entry module further provides a manual review interface for correcting and tracing the marked outliers and recording the review log. Each module interacts with data through an asynchronous message queue, forming a top-down, highly cohesive, loosely coupled data processing pipeline.

2. The system according to claim 1, characterized in that, The web page crawling module includes a static HTML crawling submodule, a dynamic page rendering submodule, and an API interface calling submodule, which can automatically select the optimal crawling method to adapt to multiple heterogeneous data sources.

3. The system according to claim 1, characterized in that, The structured parsing module uses a hybrid parsing approach based on XPath, regular expressions, and the BERT named entity recognition model to automatically extract data source identifiers, key personnel information, information release time, and core time-series numerical fields from unstructured text.

4. A time-series data extraction method based on modular design, characterized in that, The method includes the following steps: S1. Task Scheduling and Task Generation: The task scheduling module triggers data capture tasks periodically based on timing rules; before generating a task, the scheduling module reads the last successful update timestamp from the metadata database and generates task instructions only for incremental data after the timestamp; if the task execution fails, the system puts the task into a delay queue and automatically retryes it after a preset time interval to ensure the integrity of the data capture. S2, Data Fetching: The web crawling module executes corresponding strategies according to the crawling mode of the task instructions: static pages use requests and lxml to directly parse the HTML structure; dynamic pages use a headless browser to simulate user behavior and wait for JavaScript to finish rendering; API pages directly obtain structured JSON or XML data through HTTP requests; finally, the crawled raw data is encapsulated into data packets and pushed to the message queue. S3, Structured Parsing: The structured parsing module combines the rule base with the BERT named entity recognition model to perform field recognition and text extraction, generate structured JSON objects, and push them to downstream modules; S4. Data Cleaning and Anomaly Detection: The data cleaning and mapping module receives a structured JSON object, performs deduplication and format standardization operations, and performs statistical outlier detection through the following steps: S41. Implement data deduplication based on key field hashing; S42. Numerical and time formats shall be standardized in units and in the ISO 8601 format; S43. Detecting outliers using the Z-Score algorithm includes: calculating the mean μ and standard deviation σ of the historical time series data; substituting the current value to be detected X into the formula Z = (X... μ) / σ; When |Z| is greater than the preset threshold 3, the record is marked as abnormal and an abnormal score is attached, thereby actively identifying abnormal data and preventing erroneous values ​​from polluting the database. S5. Data import and quality audit: The cleaned time-series data is written into a time-series database and a relational database; After the data import module completes the import process, it generates a data quality report, which includes statistics on the number of data imported, the percentage of abnormal data, and the field missing rate. S6. Manual review and final data confirmation: For flagged abnormal data, the system automatically pushes it to the manual review interface for administrators to verify and correct. All manual operations are recorded to ensure data traceability.

5. The method according to claim 4, characterized in that, After a task fails to execute, the task scheduling module places the task in a delay queue and automatically retryes it after a preset time interval, up to three times.

6. The method according to claim 4, characterized in that, After generating a data quality report, the data entry module pushes the report to the monitoring system or administrator terminal and records the manual review operation log, enabling full traceability of the data process.

7. A computer-readable storage medium having a computer program stored thereon, the program, when executed by a processor, implementing the steps of the time-series data extraction method as described in any one of claims 4-6.

Images