Data general survey method and device for multi-source heterogeneous data source

Abstract

The invention discloses a data general survey method and device for a multi-source heterogeneous data source, and the method comprises the steps: S1, data source initialization and metadata maintenance: accessing the multi-source heterogeneous data source, and building a data source information table containing basic information, metadata and state identification; s2, a dynamic scheduling strategy is generated, wherein an initial scheduling plan is generated according to the data source priority, the updating frequency and the data size; s3, increment identification and statistics: performing timestamp comparison or transaction log analysis on the structured data source, performing file hash or block verification on the unstructured data source, and outputting an increment statistics log; and S4, multi-source data integration and output, visual display and interface output are carried out. The method has the technical effects that through three innovations of the dynamic scheduling engine, the increment recognition rule engine and the unified adaptation framework, the resource utilization rate is improved, the processing efficiency is leap, and the expansibility is remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field related to data processing, and in particular to a data census method and device for multi-source heterogeneous data sources. Background Art

[0002] Existing data census programs generally have the following defects:

[0003] Traditional solutions use fixed-frequency full-data scanning (such as daily scheduled tasks), which cannot dynamically adjust the census rhythm according to the real-time status of the data source (such as network fluctuations, data update frequency), resulting in resource waste or statistical lag, and lack of scheduling flexibility; repeated scanning of all data, especially indiscriminate processing of massive static data, consumes a lot of computing resources and has a slow response speed (for example, a single scan of TB-level unstructured files (such as log files) takes more than 4 hours, and resource consumption is more than 5 times higher than incremental calculation), resulting in low computing efficiency; the incremental recognition logic of different data sources varies greatly, and traditional solutions need to be customized and developed, with insufficient scalability, resulting in (3) poor adaptability to heterogeneous data sources and high adaptation costs. Summary of the Invention

[0004] In order to overcome the existing technical defects, the purpose of the present invention is to provide a data census method and device for multi-source heterogeneous data sources to solve the above technical problems. This method and device are suitable for scenarios such as finance, government affairs, and enterprise data middle stations that require real-time monitoring of changes in multiple types of data sources.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] According to one aspect of the present invention, a data census method for multi-source heterogeneous data sources is designed, comprising the following steps:

[0007] Step S1, data source initialization and metadata maintenance: access multiple heterogeneous data sources and establish a data source information table containing basic information, metadata and status identification;

[0008] Step S2, dynamic scheduling strategy generation: Generate an initial scheduling plan based on data source priority, update frequency, and data volume, and dynamically adjust the scheduling interval and concurrency based on real-time status;

[0009] Step S3, incremental identification and statistics: Use timestamp comparison or transaction log analysis for structured data sources, and use file hash or block verification for unstructured data sources, and output incremental statistical logs;

[0010] Step S4: Multi-source data integration and output: convert the statistical results into a standardized format, visualize and output the results through an interface.

[0011] In some embodiments, in step S2:

[0012] Priority grading includes: setting core business libraries to high priority, shortening scheduling intervals for high-frequency data sources, and enabling block scheduling for massive data sources;

[0013] Dynamic adjustments include skipping offline data sources through heartbeat monitoring and adjusting the number of concurrent tasks based on historical time consumption to avoid resource preemption.

[0014] In some embodiments, the real-time status dynamic adjustment in step S2 is triggered by:

[0015] Automatically extend the scheduling interval when detecting that the data source network delay exceeds the threshold;

[0016] When the task execution time exceeds 150% of the historical average, the number of concurrent tasks for the data source is automatically reduced.

[0017] In some implementations, the incremental identification of structured data sources in step S3 includes:

[0018] Timestamp-based comparison: extract new / modified records by comparing current data with the timestamp of the last census;

[0019] Transaction log analysis: Capture database INSERT / UPDATE / DELETE operation logs in real time.

[0020] In some implementations, the incremental identification of unstructured data sources in step S3 includes:

[0021] File-level increment: compares file modification time with hash value to identify new / changed files;

[0022] Content-level increment: Hash values ​​are calculated for large files in blocks, and only the changed data blocks are counted.

[0023] In some implementations, the incremental identification of unstructured data sources further includes:

[0024] Incremental aggregation: Summarizes the number of records, files, or bytes by data source type and generates a statistical log with timestamp, increment type, and change amount.

[0025] In some embodiments, the standardized format in step S4 is a JSON format, and the visual dashboard supports displaying data volume trends and data source health by day / week / month dimensions.

[0026] In some embodiments, further comprising:

[0027] Exception handling mechanism: When the data source connection is abnormal, scheduling is automatically suspended and an alarm is triggered; if the task execution fails, it is retried according to the exponential backoff strategy.

[0028] According to another aspect of the present invention, a data census device is designed, comprising:

[0029] Data source access module: provides standardized interfaces and supports JDBC, HTTP and file system protocols;

[0030] Dynamic scheduling engine: includes a policy generation unit and a task execution unit, used for scheduling plan generation and task triggering;

[0031] Incremental recognition component: includes a rule engine and a difference calculation unit, used to load incremental rules and perform difference comparison;

[0032] Statistical calculation module: performs aggregate calculations on incremental data, generates statistical reports and stores them in the historical database;

[0033] Status monitoring module: collects task indicators in real time and triggers alarms;

[0034] Metadata management module: maintains the data source metadata dictionary and supports field-level change monitoring.

[0035] In some embodiments, the metadata management module automatically updates the census logic in the incremental recognition rules when detecting the addition / deletion of a field;

[0036] The dynamic scheduling engine also includes:

[0037] Resource allocation unit: dynamically allocates CPU and memory resources to task execution nodes based on server load;

[0038] Priority preemption unit: high-priority tasks can interrupt the execution of low-priority tasks;

[0039] The rule engine of the incremental recognition component is preset:

[0040] SQL log parsing rules for structured data sources;

[0041] Block hashing algorithm for unstructured data sources;

[0042] When adding a new data source type, the rule base is expanded through a plug-in architecture.

[0043] The technical effects of the present invention are as follows: through three major innovations, namely, a dynamic scheduling engine, an incremental recognition rule engine, and a unified adaptation framework, the present invention achieves: 1. Improved resource utilization: By dynamically adjusting the scheduling interval and the number of concurrencies, the resource idle rate is reduced by 40%, and the delay of high-priority tasks is shortened by 50%; by scheduling massive data in blocks, the peak memory usage is reduced by 60%; 2. A leap in processing efficiency: incremental statistics only process changed data (accounting for <10% of the full amount), and the processing speed is increased by 5-10 times (actually measured MySQL incremental scan is 1 minute vs. 30 minutes for the full amount); the file system adopts block hash verification, and the processing time of large files is reduced by 80%; 3. Significantly enhanced scalability: the adaptation period of new data sources is compressed from 2 weeks to 1 day; and minute-level data change capture is supported (such as the API interface triggering incremental statistics every 5 minutes). BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a data census method for multi-source heterogeneous data sources according to an embodiment of the present invention;

[0045] Figure 2 This is a structural diagram of the data census device. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0047] Example 1

[0048] refer to Figure 1 As shown, the present invention provides a data census method for multi-source heterogeneous data sources, comprising the following steps:

[0049] Data source initialization and metadata maintenance: Connect to multiple heterogeneous data sources, such as MySQL, MongoDB, CSV files, and Restful APIs, and establish a data source information table containing basic information, metadata, and status identifiers. Basic information includes: data source ID, type (structured / unstructured), and connection parameters; metadata includes: table name / file name, field / attribute definition, and data update frequency label (high frequency / low frequency); status identifiers include: online / offline, last census time, and historical increment threshold.

[0050] Dynamic scheduling strategy generation: Generates an initial scheduling plan based on data source priority, update frequency, and data volume, and dynamically adjusts the scheduling interval and concurrency based on real-time status. Priority grading includes: setting core business libraries to high priority based on data source importance, shortening scheduling intervals for high-frequency data sources, and enabling block scheduling for massive data sources to generate an initial scheduling plan. Dynamic adjustments include skipping offline data sources through heartbeat monitoring and adjusting the number of concurrent tasks based on historical time consumption to avoid resource preemption. Real-time status dynamic adjustments are triggered by the following methods:

[0051] Automatically extend the scheduling interval when detecting that the data source network delay exceeds the threshold;

[0052] When the task execution time exceeds 150% of the historical average, the number of concurrent tasks for the data source is automatically reduced.

[0053] Incremental identification and statistics: Use timestamp comparison or transaction log parsing for structured data sources, and file hashing or block verification for unstructured data sources, and output incremental statistical logs. Incremental identification of structured data sources includes:

[0054] Timestamp-based comparison: extract new / modified records by comparing current data with the timestamp of the last census;

[0055] Transaction log analysis: parse MySQL Binlog and PostgreSQL WAL logs to capture database INSERT / UPDATE / DELETE operation logs in real time.

[0056] Incremental identification of unstructured data sources includes:

[0057] File-level increment: compares file modification time with hash value to identify new / changed files;

[0058] Content-level increment: Hash values ​​are calculated for large files in blocks, and only the changed data blocks are counted.

[0059] Incremental aggregation: Summarizes the number of records, files, or bytes by data source type and generates a statistical log with timestamp, increment type, and change amount.

[0060] Multi-source data integration and output: Convert statistical results (such as the number of incremental records in the database and the size of new files in the file system) into a standardized format and store them in JSON format. The standardized format is visualized through a visual dashboard and output through an interface. The visual dashboard supports displaying data volume trends and data source health by daily / weekly / monthly dimensions. The API interface provides a RESTful API for external system calls to achieve integration with the data governance platform and monitoring system.

[0061] In some embodiments, an exception handling mechanism is also included: when the data source connection is abnormal, scheduling is automatically suspended and an alarm is triggered; after the task execution fails, it is retried according to the exponential backoff strategy.

[0062] Example 2

[0063] refer to Figure 2 As shown, the data census device provided by the present invention based on the data census method described in Example 1 includes:

[0064] Data source access module: provides standardized interfaces, supports JDBC, HTTP, file system and other protocols, and implements data source registration, authentication and connection management;

[0065] Dynamic Scheduling Engine: This includes a policy generation unit and a task execution unit, which are used for scheduling plan generation and task triggering. The policy generation unit generates an initial scheduling plan (such as a Cron expression) based on data source tags (priority, frequency), and the task execution unit triggers the census task as planned, supporting task pause / retry / priority preemption.

[0066] Incremental identification component: includes a rule engine and a difference calculation unit, which are used to load incremental rules and perform difference comparison. The rule engine automatically loads the corresponding incremental identification rules (such as SQL log parsing rules and file hash algorithms) according to the data source type. The difference calculation unit performs operations such as timestamp comparison and hash value comparison, and outputs the incremental data range.

[0067] Statistical calculation module: performs aggregation calculations (such as summation and deduplication) on incremental data, generates statistical reports and stores them in the historical database; the statistical calculation module is connected to the visual dashboard through the data interface for visual display.

[0068] Status monitoring module: collects data source connection status, task execution time, resource usage and other indicators in real time, and triggers abnormal alarms (such as emails and text messages);

[0069] Metadata management module: maintains the data source metadata dictionary and supports field-level change monitoring (such as automatically updating the census logic when a field is added / deleted); when the metadata management module detects the addition / deletion of a field, it automatically updates the census logic in the incremental recognition rule.

[0070] In some embodiments, the dynamic scheduling engine further includes:

[0071] Resource allocation unit: dynamically allocates CPU and memory resources to task execution nodes based on server load;

[0072] Priority preemption unit: A high-priority task can interrupt the execution of a low-priority task.

[0073] In some embodiments, the rule engine of the incremental recognition component presets:

[0074] SQL log parsing rules for structured data sources;

[0075] Block hashing algorithm for unstructured data sources;

[0076] When adding a new data source type, the rule base is expanded through a plug-in architecture.

[0077] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A data census method for multi-source heterogeneous data sources, characterized in that: The following steps are involved: Step S1, data source initialization and metadata maintenance: access multiple heterogeneous data sources and establish a data source information table containing basic information, metadata and status identification; Step S2, dynamic scheduling strategy generation: Generate an initial scheduling plan based on data source priority, update frequency, and data volume, and dynamically adjust the scheduling interval and concurrency based on real-time status; Step S3, incremental identification and statistics: Use timestamp comparison or transaction log analysis for structured data sources, and use file hash or block verification for unstructured data sources, and output incremental statistical logs; Step S4: Multi-source data integration and output: convert the statistical results into a standardized format, visualize and output the results through an interface.

2. A data census method for multi-source heterogeneous data sources according to claim 1, characterized in that: In the step S2: Priority grading includes: setting core business libraries to high priority, shortening scheduling intervals for high-frequency data sources, and enabling block scheduling for massive data sources; Dynamic adjustments include skipping offline data sources through heartbeat monitoring and adjusting the number of concurrent tasks based on historical time consumption to avoid resource preemption.

3. A data census method for multi-source heterogeneous data sources according to claim 1 or 2, characterized in that: The real-time status dynamic adjustment of step S2 is triggered by the following methods: Automatically extend the scheduling interval when detecting that the data source network delay exceeds the threshold; When the task execution time exceeds 150% of the historical average, the number of concurrent tasks for the data source is automatically reduced.

4. The data census method for multi-source heterogeneous data sources according to claim 1 is characterized in that: The incremental identification of structured data sources in step S3 includes: Timestamp-based comparison: extract new / modified records by comparing current data with the timestamp of the last census; Transaction log analysis: Capture database INSERT / UPDATE / DELETE operation logs in real time.

5. The data census method for multi-source heterogeneous data sources according to claim 1 is characterized in that: The incremental identification of unstructured data sources in step S3 includes: File-level increment: compares file modification time with hash value to identify new / changed files; Content-level increment: Hash values ​​are calculated for large files in blocks, and only the changed data blocks are counted.

6. A data census method for multi-source heterogeneous data sources according to claim 5, characterized in that: The incremental identification of unstructured data sources further includes: Incremental aggregation: Summarizes the number of records, files, or bytes by data source type and generates a statistical log with timestamp, increment type, and change amount.

7. The data census method for multi-source heterogeneous data sources according to claim 1 is characterized in that: The standardized format in step S4 is JSON format, and the visual dashboard supports displaying data volume trends and data source health by day / week / month.

8. The data census method for multi-source heterogeneous data sources according to claim 1 is characterized in that: Also includes: Exception handling mechanism: When the data source connection is abnormal, scheduling is automatically suspended and an alarm is triggered; If a task fails, it will be retried using an exponential backoff strategy.

9. A data census device based on the data census method according to any one of claims 1 to 8, characterized in that: include: Data source access module: provides standardized interfaces and supports JDBC, HTTP and file system protocols; Dynamic scheduling engine: includes a policy generation unit and a task execution unit, used for scheduling plan generation and task triggering; Incremental recognition component: includes a rule engine and a difference calculation unit, used to load incremental rules and perform difference comparison; Statistical calculation module: performs aggregate calculations on incremental data, generates statistical reports and stores them in the historical database; Status monitoring module: collects task indicators in real time and triggers alarms; Metadata management module: maintains the data source metadata dictionary and supports field-level change monitoring.

10. The data census device according to claim 9, characterized in that: The metadata management module automatically updates the census logic in the incremental recognition rules when detecting the addition / deletion of a field; The dynamic scheduling engine also includes: Resource allocation unit: dynamically allocates CPU and memory resources to task execution nodes based on server load; Priority preemption unit: high-priority tasks can interrupt the execution of low-priority tasks; The rule engine of the incremental recognition component is preset: SQL log parsing rules for structured data sources; Block hashing algorithm for unstructured data sources; When adding a new data source type, the rule base is expanded through a plug-in architecture.

Images