Multi-source heterogeneous data adaptive conversion system and method driven by pluggable resolver

Abstract

The invention relates to a multi-source heterogeneous data adaptive conversion system and method driven by a pluggable resolver. The method comprises the steps that a data acquisition task is generated based on a pre-configured data acquisition template, and the data acquisition template comprises a page type recognition rule of a data source and a data extraction strategy corresponding to a page type; accessing a data source based on the data acquisition task; automatically judging the page type of the data source according to a page type identification rule; dynamically calling a corresponding data extraction strategy based on the page type; executing the data extraction strategy to obtain original data; according to the data type of the original data, calling a corresponding pluggable parser plug-in for parsing to generate parsed data; and processing the analysis data through the general cleaning rule and the cleaning rule special for the target domain to generate target domain data. According to the method, intelligent identification, extraction and cleaning of the multi-source heterogeneous data can be realized, and the target domain data can be efficiently acquired.

Description

Technical Field

[0001] This application relates to the field of computer information processing, and more specifically, to a pluggable parser-driven adaptive conversion system and method for multi-source heterogeneous data. Background Technology

[0002] High-quality data is the foundation of various intelligent analysis and decision-making systems. Real-world multi-source data typically exhibits characteristics such as diverse origins, heterogeneous structure, and high noise levels. Specifically, data sources are scattered across hundreds or thousands of websites and platforms, with varying page structures and data formats, and often contain a large amount of advertising, redundant content, and irrelevant information. Traditional manual collection methods or single web crawler tools struggle to simultaneously meet the requirements for data breadth, depth, timeliness, and cleanliness.

[0003] For example, when building enterprise profiles or supply chain analysis models, it is often necessary to collect data from multiple different types of data sources simultaneously, such as industry research reports (PDF format), news information (dynamic HTML pages), and statistical data (table format). These data sources differ significantly in content structure, file format, and loading method.

[0004] In existing technologies, data is often collected through web scraping, represented by open-source frameworks such as Scrapy and BeautifulSoup. These technologies parse static HTML pages and extract data by defining scraping rules (such as XPath and CSS selectors). However, existing solutions lack support for dynamic pages: many modern websites use JavaScript to dynamically load content, and general-purpose web crawlers rely on headless browsers to render pages, leading to a significant decrease in scraping efficiency and an increase in resource consumption. Existing technologies also include customized data collection systems developed for specific domains (such as e-commerce, social media, and government information). These systems are typically optimized for specific scenarios, and their built-in parsing templates, anti-scraping strategies, and cleaning rules are difficult to migrate to other complex domains. Moreover, commercial systems are often black-box structures, preventing users from deeply customizing or extending them for new data sources, making it difficult to meet the refined data collection needs for specific documents or files in complex scenarios.

[0005] In summary, existing technologies have significant shortcomings in terms of efficiency, scalability, and intelligent adaptability, failing to meet the demands of current intelligent analysis systems for automated, high-fidelity acquisition and cleaning of multi-source heterogeneous data. Therefore, a novel pluggable parser-driven adaptive transformation system and method for multi-source heterogeneous data is needed.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of this, this application provides a pluggable parser-driven adaptive transformation system and method for multi-source heterogeneous data, which can realize intelligent identification, adaptive extraction and hierarchical cleaning of multi-source heterogeneous data, thereby efficiently obtaining high-quality, structured target domain data.

[0008] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0009] According to one aspect of this application, a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data is proposed. The method includes: generating a data acquisition task based on a pre-configured data acquisition template, wherein the data acquisition template includes: page type identification rules for the data source and a data extraction strategy corresponding to the page type; accessing the data source based on the data acquisition task; automatically determining the page type of the data source according to the page type identification rules; dynamically invoking the corresponding data extraction strategy based on the page type; executing the data extraction strategy to obtain raw data, wherein the raw data is multi-source heterogeneous data; parsing the raw data according to its data type by calling a corresponding pluggable parser plugin to generate parsed data; and processing the parsed data using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

[0010] Optionally, it further includes: dual storage of the target domain data; and updating the task status of the corresponding data acquisition task when the dual storage is successful.

[0011] Optionally, the data acquisition template further includes: a data acquisition strategy; generating a data acquisition task based on the pre-configured data acquisition template, including: listening to the data acquisition template; determining the task time according to the data acquisition strategy in the data acquisition template; generating the data acquisition task at the task time; and serializing the data acquisition task and pushing it into a message queue.

[0012] Optionally, automatically determining the page type of the data source according to the page type identification rules includes: analyzing one or more of the URL characteristics, HTTP response header information, and page content structure of the page of the data source according to the page type identification rules to automatically determine the page type of the data source.

[0013] Optionally, the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data, including: when the page type is a static page, calling an HTTP client to directly request and obtain the raw data; when the page type is a dynamically rendered page, calling an instance in the headless browser pool to render and then obtaining the raw data; when the page type is a document file, calling a file download module to download and obtain the raw data.

[0014] Optionally, depending on the data type of the original data, a corresponding pluggable parser plugin is invoked to parse and generate parsed data, including: when the original data is an HTMLParser object, using CSS selectors or XPath expressions to extract content from the page structure to obtain the parsed data; when the original data is a DocumentParser object, invoking a document parser to parse its content into plain text or structured data objects to obtain the parsed data; when the original data is an APIParser object, using JSONPath or XPath to extract target fields to obtain the parsed data.

[0015] According to one aspect of this application, a pluggable parser-driven adaptive transformation system for multi-source heterogeneous data is proposed. The system includes: a task generation module for generating a data acquisition task based on a pre-configured data acquisition template, wherein the data acquisition template includes: page type identification rules for the data source and a data extraction strategy corresponding to the page type; an access module for accessing the data source based on the data acquisition task; a judgment module for automatically determining the page type of the data source according to the page type identification rules; a calling module for dynamically calling the corresponding data extraction strategy based on the page type; an extraction module for executing the data extraction strategy to obtain raw data, wherein the raw data is multi-source heterogeneous data; a parsing module for parsing the raw data by calling a corresponding pluggable parser plugin according to the data type of the raw data to generate parsed data; and a cleaning module for processing the parsed data using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

[0016] Optionally, it further includes: a template generation module, used to determine the data source; generate basic information for the data source; determine a data acquisition strategy, a data extraction strategy, and a page type identification rule for the data source; and generate a data acquisition template for each data source through a configuration file based on the basic information, the data acquisition strategy, the data extraction strategy, and the page type identification rule.

[0017] Optionally, it also includes: a data storage module, used to dual-store the target domain data; and to update the task status of the corresponding data acquisition task when the dual storage is successful.

[0018] Optionally, the data storage module includes: a persistent storage unit for persistently storing the target domain data in its original file format to a distributed object storage system or database; and an indexing unit for extracting key metadata from the target domain data, storing the key metadata in an index database, and creating an index.

[0019] Optionally, the data acquisition template further includes: a data acquisition strategy; the task generation module includes: a listening unit for listening to the data acquisition template; a time unit for determining the task time according to the data acquisition strategy in the data acquisition template; a task unit for generating the data acquisition task at the task time; and a queue unit for serializing the data acquisition task and pushing it into a message queue.

[0020] Optionally, the access module is further configured to extract a data acquisition task from the message queue; read a data source from the data acquisition task; and access the data source.

[0021] Optionally, the judgment module is further configured to analyze one or more of the URL characteristics, HTTP response header information, and page content structure of the page of the data source according to the page type identification rules, so as to automatically determine the page type of the data source.

[0022] Optionally, the extraction module includes: a static unit, used to call an HTTP client to directly request and obtain the original data when the page type is a static page; a dynamic unit, used to call an instance in the headless browser pool to render and obtain the original data when the page type is a dynamically rendered page; and a document unit, used to call a file download module to download and obtain the original data when the page type is a document file.

[0023] Optionally, the parsing module includes: a webpage unit, used to extract content from the page structure using CSS selectors or XPath expressions when the original data is an HTMLParser object, to obtain the parsed data; a text unit, used to call a document parser to parse its content into plain text or structured data objects when the original data is a DocumentParser object, to obtain the parsed data; and a code unit, used to extract target fields using JSONPath or XPath when the original data is an APIParser object, to obtain the parsed data.

[0024] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0025] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0026] According to one aspect of this application, a computer program product is provided, comprising: a computer program / instructions that, when executed by a processor, implement the method as described above.

[0027] According to the pluggable parser-driven adaptive transformation system and method for multi-source heterogeneous data of this application, a data acquisition task is generated based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type. Based on the data acquisition task, the data source is accessed; the page type of the data source is automatically determined according to the page type identification rules; the corresponding data extraction strategy is dynamically invoked based on the page type; the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data; according to the data type of the raw data, the corresponding pluggable parser plugin is invoked to parse it and generate parsed data; the parsed data is processed through general cleaning rules and target domain-specific cleaning rules to generate target domain data. This method can realize intelligent identification, adaptive extraction, and hierarchical cleaning of multi-source heterogeneous data, thereby efficiently obtaining high-quality, structured target domain data.

[0028] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0029] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0030] Figure 1 This is a flowchart illustrating a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data, according to an exemplary embodiment.

[0031] Figure 2 This is a flowchart illustrating a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data according to another exemplary embodiment.

[0032] Figure 3 This is a schematic diagram illustrating a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data according to another exemplary embodiment.

[0033] Figure 4 This is a block diagram illustrating a pluggable parser-driven adaptive transformation system for multi-source heterogeneous data, according to an exemplary embodiment.

[0034] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] To address the shortcomings of existing technologies, this application aims to solve the problems of low identification efficiency, poor scalability, and insufficient data cleaning adaptability in the process of multi-source heterogeneous data acquisition and cleaning, and provides an automated data acquisition and processing method that can achieve intelligent page type identification, adaptive processing, and hierarchical cleaning.

[0037] Specifically, this application mainly addresses the following two technical issues: First, existing web crawling technologies lack intelligent identification and adaptive processing mechanisms for different types of pages, resulting in low crawling efficiency, poor scalability, and difficulty in unified management when facing multi-source heterogeneous data.

[0038] This application constructs a "page type recognition-automatic strategy matching" mechanism to achieve automatic classification and optimal strategy allocation for different types of pages. The system can automatically determine page types (including static HTML pages, dynamically rendered pages, PD documents, etc.) based on multi-dimensional features such as URL characteristics, response header information, and page structure. According to preset rules, it automatically assigns different types of pages to corresponding processing flows (such as HTTP client modules, headless browser pools, or document parsing engines), thereby achieving efficient and scalable adaptive crawling.

[0039] Second, existing data cleaning methods lack deep adaptation to domain-specific characteristics when dealing with complex multi-source data, resulting in low quality of raw data that is difficult to directly support upper-level analysis applications.

[0040] This application designs a pluggable parser and cleaning rule dynamic scheduling pipeline, realizing hierarchical scheduling of general cleaning rules and domain-specific cleaning rules. The system has built-in multiple parser plugins, which can efficiently parse different types of data such as HTML pages, PDF documents, and API return data; at the same time, it supports custom domain-specific cleaning rules, so that the raw data can meet the requirements of structured, standardized, and high-quality output after cleaning, thus providing a reliable data foundation for subsequent intelligent analysis.

[0041] The target domain data obtained in this application can be widely applied to various intelligent analysis and decision-making scenarios, providing high-quality, structured data support for upper-level data modeling, knowledge graph construction, and business risk control analysis. Typical applications include, but are not limited to, the following: Enterprise Knowledge Network Construction: This application provides a unified and standardized entry point for raw data collection and preprocessing, which can effectively solve the problems of multi-source data silos and inconsistent data quality, laying a solid foundation for building a dynamically updated enterprise-level knowledge network and entity relationship graph.

[0042] Macroeconomic and industry research and analysis: The system can automatically collect macroeconomic indicators (such as GDP, CPI, PMI, etc.) and industry dynamic information from various public data sources on a regular basis to form a continuously updated dataset, providing reliable data support for quantitative research, trend forecasting and industry analysis.

[0043] Intelligent Analysis and Risk Warning: By monitoring news, policies and abnormal signals related to specific enterprises, industries or events in real time, the system can provide timely data input for scenarios such as investment analysis, supply chain monitoring, hot information and risk warning, and assist business departments in making intelligent decisions.

[0044] Compliance and Audit Support: The system can automatically collect and archive various policies, regulations, notices, announcements, and regulatory information to form a structured and traceable policy dataset, providing data support for business compliance verification, internal auditing, and risk control for enterprises or institutions.

[0045] The content of this application will be described in detail below with the aid of specific embodiments.

[0046] Figure 1 This is a flowchart illustrating a pluggable parser-driven adaptive conversion method for multi-source heterogeneous data according to an exemplary embodiment. The pluggable parser-driven adaptive conversion method 10 for multi-source heterogeneous data includes at least steps S101 to S107.

[0047] like Figure 1As shown, in S101, a data acquisition task is generated based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type.

[0048] In one embodiment, the data acquisition template further includes a data acquisition strategy; more specifically, based on the pre-configured data acquisition template, a data acquisition task is generated, including: listening to the data acquisition template; determining the task time according to the data acquisition strategy in the data acquisition template; generating the data acquisition task at the task time; and serializing the data acquisition task and pushing it into a message queue.

[0049] More specifically, the system continuously monitors the data acquisition template. When the template is detected to be active, it determines the task generation time based on the preset data acquisition strategy in the template, automatically generates the data acquisition task at the specified time, and serializes the task before pushing it into the message queue. The data acquisition task may include parameters such as the data source URL, request header information, login credentials, page recognition rules, extraction strategy identifier, and proxy configuration, so that downstream modules can schedule it uniformly.

[0050] In S102, the data source is accessed based on the data acquisition task. For example, the data acquisition task can be extracted from a message queue; the data source can be read from the data acquisition task; and the data source can be accessed.

[0051] The system retrieves data acquisition tasks from the message queue and accesses the target website or API based on the data source address specified in the task. Access can be via HTTP / HTTPS requests, WebSocket, API calls, or file download requests. During the access process, the system automatically applies a proxy IP pool, a User-Agent pool, and random request latency to enhance anti-blocking and anti-scraping capabilities.

[0052] In S103, the page type of the data source is automatically determined according to the page type identification rules.

[0053] For example, based on the page type identification rules, one or more of the URL characteristics, HTTP response header information, and page content structure of the data source page can be analyzed to automatically determine the page type of the data source.

[0054] In practical English translation, the identification rules can be based on a combination of the following features: URL characteristics, such as the presence of keywords like ".pdf", " / api / ", or " / news / " in the URL; HTTP response header information, such as the Content-Type field; Page content structure features, such as the number of DOM node tags, the number of JavaScript scripts, and the proportion of table elements.

[0055] Based on the comprehensive analysis of the above features, the system can automatically identify whether a page is a static HTML page, a dynamically rendered page, an API interface page, or a document file.

[0056] In S104, the corresponding data extraction strategy is dynamically invoked based on the page type.

[0057] Based on the identified page type, the system dynamically calls the corresponding data extraction strategy: for static HTML pages, it calls the lightweight HTTP client module to directly request the content; for dynamically rendered pages, it calls a browser instance in the headless browser pool to render the content and waits for the key elements to load before extracting the final HTML content; for document pages (such as PDF or Word files), it calls the file download module to download the file to the local cache; and for API interfaces, it directly parses the returned JSON or XML structure data.

[0058] This dynamic strategy matching mechanism automatically binds page types to the optimal data collection method, significantly improving data acquisition efficiency and system scalability.

[0059] In step S105, the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data. The raw data may include various types such as HTML strings, PDF binary streams, JSON objects, and table files, and thus belongs to multi-source heterogeneous data. During this stage, the system automatically records the data source identifier, access time, and extraction status for subsequent tracking and task status updates.

[0060] In one embodiment, for example, when the page type is a static page, an HTTP client can be invoked to directly request and obtain the raw data; In one embodiment, for example, when the page type is a dynamically rendered page, the original data can be obtained by calling an instance in the headless browser pool to render the page. In one embodiment, for example, when the page type is a document file, a file download module can be invoked to download and obtain the original data.

[0061] In S106, based on the data type of the original data, the corresponding pluggable parser plugin is called to parse the data and generate parsed data.

[0062] In one embodiment, for example, when the original data is an HTMLParser object, CSS selectors or XPath expressions can be used to extract content from the page structure to obtain the parsed data; In one embodiment, for example, when the original data is a DocumentParser object, a document parser can be invoked to parse its content into plain text or a structured data object to obtain the parsed data; In one embodiment, for example, when the original data is an APIParser object, the target field can be extracted using JSONPath or XPath to obtain the parsed data.

[0063] In S107, the parsed data is processed using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

[0064] General cleaning rules include: removing HTML tags, removing advertising information and navigation bar content, standardizing time and character encoding formats, and removing whitespace and garbled characters.

[0065] Domain cleaning rules can be customized according to application scenarios, such as terminology standardization, table structure extraction, field mapping, and entity recognition.

[0066] The system adopts a hierarchical scheduling mechanism, first performing general cleaning, and then calling the corresponding domain cleaning rules according to the data source type and target application to ensure the structured and high-fidelity of the output data.

[0067] In one embodiment, the method further includes: dual storage of the target domain data; and updating the task status of the corresponding data acquisition task when the dual storage is successful.

[0068] More specifically, the target domain data can be persistently stored in the original file format to a distributed object storage system or database; the cleaned data can be persistently stored in JSON or the original file format to a distributed object storage system or database (such as HDFS, S3).

[0069] More specifically, for example, key metadata can be extracted from the target domain data, and this key metadata can be stored in an index database and an index can be created. Key metadata (such as title, source URL, publication time, keywords, etc.) can be extracted and written into an index database (such as Elasticsearch) to create a multidimensional index to support retrieval and statistics.

[0070] Furthermore, once data storage is successful, the data storage module returns an acknowledgment signal to the task scheduling module and updates the task status to "success". If an error occurs or a retry fails, it is marked as "failure" and a detailed log is recorded.

[0071] According to the pluggable parser-driven adaptive transformation method for multi-source heterogeneous data of this application, a data acquisition task is generated based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type. Based on the data acquisition task, the data source is accessed; the page type of the data source is automatically determined according to the page type identification rules; the corresponding data extraction strategy is dynamically invoked based on the page type; the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data; according to the data type of the raw data, the corresponding pluggable parser plugin is invoked to parse it and generate parsed data; the parsed data is processed through general cleaning rules and target domain-specific cleaning rules to generate target domain data. This method can achieve intelligent identification, adaptive extraction, and hierarchical cleaning of multi-source heterogeneous data, thereby efficiently obtaining high-quality, structured target domain data.

[0072] It should be clearly understood that this application describes how specific examples are formed and used, but the principles of this application are not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0073] Figure 2 This is a flowchart illustrating a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data according to another exemplary embodiment. Figure 2 The process shown in step 20 is... Figure 1 Supplementary description of the process shown.

[0074] like Figure 2 As shown, in S201, the data source is determined.

[0075] The system first determines the data source to be collected. Data sources can include various websites, online databases, open APIs, document download links, etc.

[0076] In one embodiment, the system can select target data sources from the data catalog according to project requirements and register and identify them, with each data source corresponding to a unique source identifier.

[0077] In another embodiment, the system can dynamically discover new data sources based on keyword search, seed URL expansion, or manual configuration.

[0078] In S202, basic information is generated for the data source. This basic information includes the data source's base URL, access protocol type (HTTP, HTTPS, FTP, etc.), request headers, authentication credentials (such as cookies, tokens, or API keys), pagination parameter rules, and other related information.

[0079] The system can automatically detect the reachability and response type of the data source, and record information such as connection timeout, status code and content type, so as to perform parameter verification during subsequent policy configuration.

[0080] In S203, a data acquisition strategy, a data extraction strategy, and page type identification rules are determined for the data source.

[0081] For the data source, determine the data acquisition strategy, data extraction strategy, and page type identification rules. The data acquisition strategy is used to define the task scheduling method and execution frequency, such as timed acquisition, periodic refresh, or event-triggered execution. This strategy may include parameters such as crawling frequency, crawling depth, number of retries, and proxy strategy; Data extraction strategies define how to extract target data from a page or interface, such as XPath rules for HTML structure, JSONPath path expressions for JSON data, or text extraction patterns for document files; Page type identification rules are used to guide the system in determining the logic of different page types, such as automatically identifying static pages, dynamic pages, API interfaces, or document pages through URL features, HTTP response header fields, or page DOM structure features.

[0082] In S204, based on the basic information, data acquisition strategy, data extraction strategy, and page type identification rules, a data acquisition template is generated for each data source through a configuration file.

[0083] The templates can be stored using an extensible markup structure (such as JSON or YAML format) for system parsing and dynamic loading. Each template contains a unique template ID, version number, and timestamp, which are used by the task scheduling module for template monitoring and updates.

[0084] In one embodiment, the system can store the generated template in the configuration center database, and the template monitoring service can continuously monitor the template status. When the template is detected to be in an "active" state, the task scheduling module will automatically generate a data acquisition task according to the acquisition strategy defined in the template, and push it to the message queue to enter the main data collection process.

[0085] Through the above steps, the system has achieved standardized definition and automated generation of data acquisition templates, ensuring that different types of data sources can be configured and managed in a unified manner, thereby providing a reliable configuration foundation for subsequent task scheduling, adaptive crawling and cleaning processing.

[0086] Figure 3 This is a schematic diagram illustrating a pluggable parser-driven adaptive transformation method for multi-source heterogeneous data according to another exemplary embodiment. For example... Figure 3As shown, this system adopts a modular and distributed architecture design, logically comprising: a configuration management component, a task scheduling component, a data acquisition component, a parsing and cleaning component, and a data storage component. These components are decoupled and communicate with each other through message queues and a central database, thus forming a highly cohesive, loosely coupled, and horizontally scalable distributed system structure.

[0087] The configuration management component is used to define and maintain crawling templates for each data source and generate configuration files that the system can execute. This component allows defining corresponding template information for each data source through a visual interface or configuration files. Templates may include: Source information: base URL, necessary request headers, login credentials (such as cookies, tokens), etc. Data acquisition strategy: including macro-level control rules such as data acquisition frequency, data acquisition depth, and time range; Page type identification rules: Logical rules used to automatically determine the page type, such as identifying list pages, detail pages, API interfaces, document pages, etc. through URL keywords, response header fields, or page structure features; Data extraction rules: Extraction logic is defined for different types of pages, such as using CSS selectors or XPath to extract HTML pages, using JSONPath to parse API interfaces, and extracting documents by page number range or full text.

[0088] The data acquisition template generated by this component serves as the input for subsequent task scheduling components.

[0089] The task scheduling component and data acquisition component are the core execution units of the system, used to realize task generation, task distribution and data acquisition.

[0090] The system has a built-in task scheduler background service that continuously monitors active templates and automatically generates data acquisition tasks at preset times based on the crawling strategy in the template. Each task contains complete information such as the source URL, parsing rules, and proxy configuration, and is serialized and pushed into a message queue to await processing by the data acquisition component.

[0091] After retrieving a task from the message queue, the optimal data retrieval method is automatically matched based on the page type: for static HTML pages, an efficient HTTP client component is called to directly request the page content; for dynamically rendered pages, a headless browser pool instance (such as Puppeteer) is called to load the page, and the final rendering result is extracted after the key elements are loaded; for document files (PDF, Word, etc.), a file download component is called to directly download and save them.

[0092] After acquiring the page content, the system extracts target fields (such as title, body text, publication time, etc.) according to the data extraction rules defined in the task, generating semi-structured raw data objects. If the current page is a list page, the system can also automatically parse the details page links and generate a new task to be pushed back to the message queue, realizing recursive crawling.

[0093] Furthermore, the system incorporates multiple anti-scraping mechanisms, such as random switching of the User-Agent pool, rotation of the proxy IP pool, and random delay strategies, to simulate human access behavior and improve the success rate of crawling and the robustness of the system.

[0094] The content parsing and cleaning component is used to parse, clean, and structure the raw data, and is a key component for the system to achieve high-quality data output.

[0095] The system automatically calls the corresponding parser plugin based on the data type (HTML, PDF, JSON, etc.): HTMLParser: accurately extracts page content using CSS selectors or XPath; DocumentParser: converts documents into structured text using text extraction engines (such as Apache Tika, PDFMiner); APIParser: parses the data structure returned by the API using JSONPath or XPath.

[0096] More specifically, the parser adopts a pluggable design, which can be dynamically loaded and expanded according to different data sources.

[0097] After the parsed results enter the cleaning rule engine, the system performs general cleaning and domain cleaning in sequence: general cleaning removes HTML tags, advertisements, style code, copyright notices, and standardizes time and character formats; domain cleaning loads corresponding cleaning rules for specific industries or application scenarios, such as terminology standardization, structured table extraction, and policy clause structuring. The final cleaned data is stored in a unified JSON object format, providing high-fidelity data for upper-level analysis.

[0098] The data storage component is used to persist and index the cleaned data and provide feedback on the task status.

[0099] More specifically, the cleaned JSON object can be saved as a file to a database or object storage system (such as HDFS or S3); core metadata such as title, URL, publication time, and keywords can also be extracted and stored in a relational database or Elasticsearch index to support multidimensional retrieval.

[0100] After data storage is complete, the component returns a status signal to the task scheduling component, updating the task status to "success" or "failure" and recording log information. This mechanism enables full-process traceability and error backtracking for the task.

[0101] In summary, this application achieves efficient acquisition, high-fidelity parsing, and structured output of multi-source heterogeneous data by constructing an adaptive acquisition framework of "page type recognition - automatic strategy matching," employing a pluggable parsing and layered cleaning pipeline, and combining it with a distributed message scheduling architecture. The system achieves significant improvements in efficiency, scalability, stability, and data quality. Specific technical effects are as follows: Unified architecture and improved data acquisition efficiency: This application constructs a unified data collection framework for page type identification and automatic strategy matching. It comprehensively determines page type (e.g., static HTML pages, dynamically rendered pages, PDF documents) based on multi-dimensional features such as URL characteristics, response header information, and page structure, and dynamically calls the optimal processing method (HTTP client, headless browser pool, document parsing engine, etc.). This framework is the first to implement a unified management model of "one framework, multiple strategies," which differs from the traditional customized development solution of "one source, one strategy," significantly improving the system's scalability and maintenance efficiency.

[0102] Thanks to the adaptive crawling mechanism, static pages are processed directly by the HTTP client, dynamic pages are rendered quickly through browser pool reuse technology, and document pages are collected by a dedicated parsing engine, which improves the overall crawling efficiency by several times and significantly reduces the average processing latency per page.

[0103] Enhanced system robustness and scalability: The system adopts a message queue-based distributed scheduling and execution architecture, enabling asynchronous task distribution and parallel processing, and supporting multi-node collaborative work and horizontal scaling. A single cluster can process thousands of tasks simultaneously, and critical tasks are scheduled with priority ensuring timeliness.

[0104] The system combines proxy IP rotation, User-Agent switching, and automatic retry mechanisms to effectively enhance its anti-blocking capability and fault tolerance against target sources, improving data collection success rate by approximately 15% and making overall operation more stable and reliable.

[0105] Data analysis and cleaning quality improvement: This application designs a parser plugin that supports dynamic loading and a hierarchical cleaning rule pipeline, realizing the coordinated scheduling of general cleaning rules and domain-specific cleaning rules. The parser adopts a plug-in design, supporting unified parsing of multiple types of data sources (HTML, PDF, API data, etc.), and users can flexibly extend it as needed.

[0106] The cleaning rules are divided into a general layer and a domain layer: the general layer performs standardized operations such as noise reduction, text normalization, and format unification; the domain layer performs specific structured tasks, such as terminology standardization, table content extraction, and clause structured parsing.

[0107] This hierarchical scheduling mechanism effectively decouples general data processing from domain adaptation logic, significantly improving the system's cleaning accuracy and data quality in high-noise and complex data scenarios.

[0108] High-fidelity output and upper-layer application support: The cleaned data generated by the system is output in a unified JSON structure, possessing complete metadata and traceability. Verification has shown that the structured accuracy of complex documents and tabular data exceeds 90%, and the completeness of key field extraction reaches over 95%.

[0109] High-fidelity data can directly support upper-level applications such as knowledge network construction, semantic analysis, predictive modeling, and compliance auditing, reducing manual processing steps and improving data utilization efficiency and intelligent analysis quality.

[0110] Resource utilization and system optimization: By using browser instance pool reuse and dynamic resource scheduling mechanisms, system resource consumption is reduced by about 70%, and overall throughput is increased by about two times under the same hardware conditions.

[0111] The configuration-driven unified architecture design reduces the time to add new data sources from several hours to tens of minutes, significantly reducing system operation and maintenance and expansion costs.

[0112] It is worth mentioning that: Playwright can also be used for dynamic page crawling. Playwright supports multiple browser engines (Chromium, Firefox, WebKit) and has more efficient parallel processing capabilities, but requires additional configuration of page waiting strategies to ensure data loading is complete. All of the above alternatives require the integration of a "page type recognition" mechanism, and are essentially still specific implementations of the "adaptive crawling" concept, without changing the core architecture design of this application.

[0113] In addition to PDFMiner, Tabula can also be used, which focuses on extracting tabular data and is suitable for the rapid parsing of structured tables in research reports. However, it has weak support for non-table content (such as text paragraphs). It is necessary to ensure that the parsing results meet the structured requirements of downstream enterprise profiles and knowledge networks, which is a technical extension of the "data parsing and cleaning" in this application.

[0114] Implementing complex business logic using a declarative rule language is suitable for scenarios requiring frequent changes to the cleansing strategy, but it has a high learning curve. It needs to be compatible with the main solution's "pluggable parser" architecture and represents an extension of the rule implementation method.

[0115] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, it performs the functions defined by the method provided in this application. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0116] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0117] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0118] Figure 4 This is a block diagram illustrating a pluggable parser-driven adaptive transformation system for multi-source heterogeneous data, according to an exemplary embodiment. Figure 4 As shown, the pluggable parser-driven multi-source heterogeneous data adaptive transformation system 40 includes: a task generation module 401, an access module 402, a judgment module 403, a calling module 404, an extraction module 405, a parsing module 406, and a cleaning module 407. The pluggable parser-driven multi-source heterogeneous data adaptive transformation system 40 may further include: a template generation module 408 and a data storage module 409.

[0119] The task generation module 401 is used to generate a data acquisition task based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type. The task generation module 401 may include: a listening unit for listening to the data acquisition template; a time unit for determining the task time according to the data acquisition strategy in the data acquisition template; a task unit for generating the data acquisition task at the task time; and a queue unit for serializing the data acquisition task and pushing it into a message queue.

[0120] The access module 402 is used to access the data source based on the data acquisition task; the access module 402 is also used to extract the data acquisition task from the message queue; read the data source from the data acquisition task; and access the data source.

[0121] The judgment module 403 is used to automatically determine the page type of the data source according to the page type identification rules; The judgment module 403 is further configured to analyze one or more of the URL characteristics, HTTP response header information, and page content structure of the data source page according to the page type identification rules, so as to automatically determine the page type of the data source.

[0122] The calling module 404 is used to dynamically call the corresponding data extraction strategy based on the page type; Extraction module 405 is used to execute the data extraction strategy to obtain raw data, wherein the raw data is multi-source heterogeneous data; The extraction module 405 may include: a static unit, used to call an HTTP client to directly request and obtain the original data when the page type is a static page; a dynamic unit, used to call an instance in the headless browser pool to render and obtain the original data when the page type is a dynamically rendered page; and a document unit, used to call a file download module to download and obtain the original data when the page type is a document file.

[0123] The parsing module 406 is used to call the corresponding pluggable parser plugin to parse the original data according to the data type of the original data, and generate parsed data; The parsing module 406 may include: a webpage unit, used to extract content from the page structure using CSS selectors or XPath expressions when the original data is an HTMLParser object, to obtain the parsed data; a text unit, used to call a document parser to parse its content into plain text or structured data objects when the original data is a DocumentParser object, to obtain the parsed data; and a code unit, used to extract target fields using JSONPath or XPath when the original data is an APIParser object, to obtain the parsed data.

[0124] The cleaning module 407 is used to process the parsed data using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

[0125] The template generation module 408 is used to determine the data source; generate basic information for the data source; determine the data acquisition strategy, data extraction strategy, and page type identification rules for the data source; and generate a data acquisition template for each data source through a configuration file based on the basic information, data acquisition strategy, data extraction strategy, and page type identification rules.

[0126] The data storage module 409 is used to dual-store the target domain data; when the dual storage is successful, the task status of the corresponding data acquisition task is updated.

[0127] The data storage module 409 may include: a persistent storage unit for persistently storing the target domain data in the original file format to a distributed object storage system or database; and an indexing unit for extracting key metadata from the target domain data, storing the key metadata in an index database, and creating an index.

[0128] According to the pluggable parser-driven multi-source heterogeneous data adaptive transformation system of this application, a data acquisition task is generated based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type. Based on the data acquisition task, the data source is accessed; the page type of the data source is automatically determined according to the page type identification rules; the corresponding data extraction strategy is dynamically invoked based on the page type; the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data; according to the data type of the raw data, the corresponding pluggable parser plugin is invoked to parse it and generate parsed data; the parsed data is processed through general cleaning rules and target domain-specific cleaning rules to generate target domain data. This method enables intelligent identification, adaptive extraction, and hierarchical cleaning of multi-source heterogeneous data, thereby efficiently obtaining high-quality, structured target domain data.

[0129] like Figure 5 As shown, this application provides an electronic device 500, including a processor 510, a communication interface 520, a memory 530, and a bus 540, wherein the processor 510 and the memory 530 communicate with each other through the bus 540. Memory 530 is used to store computer programs; When the processor 510 executes the program stored in the memory 530, it implements the pluggable parser-driven multi-source heterogeneous data adaptive conversion method of any of the above embodiments.

[0130] Communication interface 520 is used for communication between the above-mentioned electronic device and other devices.

[0131] The memory 530 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 530 may also be at least one storage device located remotely from the aforementioned processor 510.

[0132] If the methods described in this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0133] This application provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the pluggable parser-driven adaptive transformation method for multi-source heterogeneous data in any of the above embodiments. For example, based on a pre-configured data acquisition template, a data acquisition task is generated. The data acquisition template includes: page type identification rules for the data source and a data extraction strategy corresponding to the page type. Based on the data acquisition task, the data source is accessed; the page type of the data source is automatically determined according to the page type identification rules; the corresponding data extraction strategy is dynamically invoked based on the page type; the data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data; according to the data type of the raw data, a corresponding pluggable parser plugin is invoked for parsing to generate parsed data; the parsed data is processed through general cleaning rules and target domain-specific cleaning rules to generate target domain data.

[0134] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A pluggable parser-driven adaptive transformation system for multi-source heterogeneous data, characterized in that, include: The task generation module is used to generate data acquisition tasks based on a pre-configured data acquisition template. The data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type. An access module is used to access the data source based on the data acquisition task. The judgment module is used to automatically determine the page type of the data source according to the page type identification rules; The calling module is used to dynamically invoke the corresponding data extraction strategy based on the page type. The extraction module is used to execute the data extraction strategy to obtain raw data, wherein the raw data is multi-source heterogeneous data; The parsing module is used to call the corresponding pluggable parser plugin to parse the original data according to its data type, and generate parsed data. The cleaning module is used to process the parsed data using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

2. The adaptive conversion system as described in claim 1, characterized in that, Also includes: The data storage module is used to dual-store the target domain data; Upon successful dual storage, update the task status of the corresponding data acquisition task.

3. The adaptive conversion system as described in claim 2, characterized in that, The data storage module includes: A persistent storage unit is used to persistently store the target domain data in its original file format to a distributed object storage system or database. An indexing unit is established to extract key metadata from the target domain data, store the key metadata in the index database, and create an index.

4. The adaptive conversion system as described in claim 1, characterized in that, The data acquisition template further includes: a data acquisition strategy; the task generation module includes: A monitoring unit is used to monitor the data acquisition template; The time unit is used to determine the task time based on the data acquisition strategy in the data acquisition template; A task unit is configured to generate the data acquisition task during the task time. A queue unit is used to serialize the data acquisition task and push it into a message queue.

5. The adaptive conversion system as described in claim 1, characterized in that, The judgment module is also used for Based on the page type identification rules, the URL characteristics, HTTP response header information, and page content structure of the data source page are analyzed to automatically determine the page type of the data source.

6. The adaptive conversion system as described in claim 1, characterized in that, The extraction module includes: A static unit is used to call an HTTP client to directly request and obtain the original data when the page type is a static page; The dynamic unit is used to obtain the original data after rendering an instance in the headless browser pool when the page type is a dynamically rendered page. The document unit is used to call the file download module to download and obtain the original data when the page type is a document file.

7. The adaptive conversion system as described in claim 1, characterized in that, The parsing module includes: A web page unit is used to extract content from the page structure using CSS selectors or XPath expressions when the original data is an HTMLParser object, to obtain the parsed data; The text unit is used to call the document parser when the original data is a DocumentParser object, to parse its content into plain text or structured data objects, and obtain the parsed data; A code unit is used to extract target fields using JSONPath or XPath to obtain the parsed data when the original data is an APIParser object.

8. A pluggable parser-driven adaptive transformation method for multi-source heterogeneous data, characterized in that, include: A data acquisition task is generated based on a pre-configured data acquisition template, wherein the data acquisition template includes: page type identification rules for the data source and data extraction strategies corresponding to the page type; Based on the data acquisition task, access the data source; The page type of the data source is determined according to the page type identification rules; Dynamically invoke the corresponding data extraction strategy based on the page type; The data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data. Based on the data type of the original data, the corresponding pluggable parser plugin is invoked to parse it and generate parsed data; The parsed data is processed using general cleaning rules and target domain-specific cleaning rules to generate target domain data.

9. The adaptive transformation method as described in claim 8, characterized in that, Also includes: The target domain data is stored in two ways; Upon successful dual storage, update the task status of the corresponding data acquisition task.

10. The adaptive transformation method as described in claim 8, characterized in that, The data acquisition template also includes: a data acquisition strategy; Based on a pre-configured data acquisition template, a data acquisition task is generated, including: Listen to the data acquisition template; The task time is determined based on the data acquisition strategy in the data acquisition template. The data acquisition task is generated at the specified task time. The data acquisition task is serialized and pushed into a message queue.

11. The adaptive transformation method as described in claim 8, characterized in that, The page type of the data source is automatically determined according to the page type identification rules, including: Based on the page type identification rules, the URL characteristics, HTTP response header information, and page content structure of the data source page are analyzed to automatically determine the page type of the data source.

12. The adaptive transformation method as described in claim 8, characterized in that, The data extraction strategy is executed to obtain raw data, which is multi-source heterogeneous data, including: When the page type is a static page, the HTTP client is invoked to directly request and obtain the original data. When the page type is a dynamically rendered page, the original data is obtained after rendering by calling an instance in the headless browser pool. When the page type is a document file, the file download module is invoked to download and obtain the original data.

13. The adaptive transformation method as described in claim 8, characterized in that, Based on the data type of the original data, the corresponding pluggable parser plugin is invoked to parse it and generate parsed data, including: When the original data is an HTMLParser object, CSS selectors or XPath expressions are used to extract content from the page structure to obtain the parsed data; When the original data is a DocumentParser object, the document parser is called to parse its content into plain text or structured data objects to obtain the parsed data; When the original data is an APIParser object, the target fields are extracted using JSONPath or XPath to obtain the parsed data.

14. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 8-13.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 8 to 13.

Images