Data acquisition and integration method, system, equipment and medium

Abstract

The invention provides a data acquisition and integration method, system and device and a medium, and belongs to the technical field of information. The method comprises the following steps: packaging multi-source heterogeneous data into an original data packet in a standard format through a unified packaging protocol; cleaning, converting and standardizing the original data packet by using a rule engine to generate a standardized data object; based on the metadata description information and the field mapping relation, intelligent fusion of the multi-source data is achieved, and a fusion data record with complete business semantics is generated; a full-link quality monitoring system is established, a quality report is generated through multi-dimensional quality index evaluation, and an alarm is triggered; displaying the operation state through a visual interface and receiving configuration parameters; and realizing dynamic scheduling and resource optimization of the distributed tasks based on the task dependency graph. According to the invention, standardization, automation and intellectualization of the whole data acquisition and integration process are realized, and the data processing efficiency and the quality reliability are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and more specifically relates to a data acquisition and integration method, system, device and medium. Background Technology

[0002] With the deepening of information technology development, data has become a crucial production resource for enterprises, social organizations, and related institutions. Various business systems, IoT devices, online platforms, and third-party services generate massive amounts of data daily. This data is scattered across different databases, interfaces, file systems, and sensor networks, exhibiting significant differences in data format, structure, and standards, resulting in marked heterogeneity and complexity. Existing information systems were often built independently, lacking unified standards and interface specifications, leading to difficulties in data interconnection and the formation of "information silos," severely impacting the efficiency of data sharing and utilization.

[0003] Currently, data acquisition and integration technologies still face several challenges in practical applications. First, data sources are complex and diverse, acquisition methods are cumbersome, and inconsistent protocols between different systems lead to low data acquisition efficiency. Second, the acquired data suffers from problems such as inconsistent formats, missing fields, redundancy, and semantic inconsistencies, making it difficult to guarantee data quality. Third, existing data integration solutions generally rely on manual configuration and static mapping, which cannot dynamically adjust mapping relationships according to business changes, resulting in poor scalability. In addition, for real-time processing and monitoring of large-scale data, traditional systems often adopt a centralized architecture, which has obvious performance bottlenecks and cannot meet the business needs of high concurrency and high timeliness.

[0004] In today's world, where data governance, intelligent analysis, and decision support are increasingly important, how to efficiently and accurately collect, clean, transform, and integrate data from multi-source, heterogeneous environments has become a core issue in big data technology systems. While existing technologies have proposed some ETL tools and data platform solutions, most lack automated intelligent probing capabilities, cannot perform dynamic task scheduling and data quality monitoring, and have limited data granularity, making it difficult to support the fusion and real-time sharing of multi-source data in complex business scenarios. Summary of the Invention

[0005] To address the above issues, the present invention aims to provide a data acquisition and integration method, system, device, and medium. It achieves standardized acquisition of multi-source heterogeneous data through a unified encapsulation protocol, completes data cleaning and transformation based on a rule engine, realizes intelligent data fusion driven by metadata, establishes a full-link quality monitoring system to ensure data reliability, and optimizes processing efficiency through dynamic task scheduling, thereby constructing a complete, efficient, and reliable data acquisition and integration solution.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a data acquisition and integration method, including: Raw data is obtained from multiple heterogeneous data sources, and the raw data is encapsulated to form a raw data packet with a unified encapsulation format. Perform data cleaning, format conversion, and semantic standardization on the raw data packets to generate standardized data objects with standard data structures; Based on predefined metadata description information and field mapping relationships, data fusion processing is performed on standardized data objects to generate fused data records with complete business semantics; The system collects the original data packets, standardized data objects, and fused data records, generates a data quality report based on preset quality assessment rules, and triggers alarm notifications according to anomaly detection rules. Get data quality reports and alarm notifications, display the information through a visual interface, and provide a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters; The system reads data quality reports and user configuration parameters, parses the execution order of each processing task based on the task dependency graph, decomposes the data processing task into subtasks through the scheduling engine and distributes them to distributed computing nodes for execution, and dynamically adjusts the scheduling strategy according to the task execution status in the data quality report.

[0007] In an optional implementation, the step of obtaining raw data from multiple heterogeneous data sources and encapsulating the raw data to form a raw data packet with a unified encapsulation format includes: Establish data connection channels with relational databases, NoSQL databases, and API interfaces; Based on a timed triggering mechanism or an event-driven mechanism, data extraction operations are performed through a data connection channel to obtain raw data; Add source system identifier, data format description, collection timestamp and data quality marker to the raw data; The original data is encapsulated using a unified serialization protocol to form an original data packet with standard header information and payload structure; The integrity and validity of the original data packet are verified using a data verification algorithm.

[0008] In an optional implementation, the step of performing data cleaning, format conversion, and semantic standardization on the original data packet to generate a standardized data object with a standard data structure includes: Perform null value processing, format normalization, abnormal data filtering, and duplicate data elimination operations on the raw data packets; Through field mapping and semantic transformation processing, the heterogeneous data in the original data packet is converted into a unified target data model; Standardized data objects are generated based on the target data model.

[0009] In an optional implementation, the step of performing data fusion processing on standardized data objects based on predefined metadata description information and field mapping relationships to generate fused data records with complete business semantics includes: Establish semantic relationships between different standardized data objects based on predefined metadata description information; Based on the semantic association, field matching rules are determined, and standardized data objects from different data sources are matched according to predefined field mapping relationships. Based on the matching results, standardized data objects are merged to generate fused data records.

[0010] In one optional implementation, the original data packets, standardized data objects, and fused data records are collected; a data quality report is generated based on preset quality assessment rules; and an alarm notification is triggered according to anomaly detection rules, including: The original data packets are collected, and the timeliness index of the collection is obtained by calculating the difference between the current time and the collection timestamp. Collect the standardized data objects and obtain format standardization indicators by verifying the matching degree between the data structure of the standardized data objects and the target data model; Collect the fused data records and use a rule engine to verify the logical relationship between the related data in the fused data records to obtain data consistency indicators; Based on preset quality assessment rules, a weighted statistical algorithm is used to comprehensively analyze the data collection timeliness index, format standardization index, and data consistency index to calculate the data quality score and generate a data quality report. When the data quality score is lower than a preset threshold, a corresponding alarm notification is generated through an alarm trigger.

[0011] In an optional implementation, the acquisition of data quality reports and alarm notifications is displayed through a visual interface, while a configuration interface is provided to receive user-input processing rules and parameter configurations, generating user-configured parameters, including: Obtain the data quality report and analyze its data quality score, data collection timeliness index, format standardization index, and data consistency index; The quality score is displayed in the form of a dashboard using visualization components, and the indicators of collection timeliness, format standardization, and data consistency are displayed in the form of trend charts. Receive alarm notifications, display alarm notifications in real time through the alarm panel, and provide historical alarm query function; The system receives user-inputted data processing rules and system parameter configurations through a configuration interface, performs format and logic validation, and stores the validated data processing rules and system parameter configurations as structured user configuration parameters.

[0012] In an optional implementation, the steps of reading the data quality report and user configuration parameters, parsing the execution order of each processing task based on the task dependency graph, decomposing the data processing task into subtasks and distributing them to distributed computing nodes for execution through the scheduling engine, and dynamically adjusting the scheduling strategy according to the task execution status in the data quality report include: Read the data quality report to obtain the data quality score and related alarm notifications; Read the task priority and resource allocation rules configured in the user configuration parameters; Based on a predefined task dependency graph, the execution logic relationships between data acquisition, data cleaning, and data fusion tasks are analyzed. Based on the execution logic relationship and task priority, the scheduling engine decomposes each data processing task into sub-tasks that can be executed in parallel. Based on the resource allocation rules, the subtasks are distributed to distributed computing nodes for execution; Monitor the execution progress and resource consumption of the subtasks; Based on the data quality score and related anomaly alarm notifications, and in conjunction with the subtask execution status, the task scheduling strategy is dynamically adjusted; when the quality score is lower than the threshold, the execution priority of the relevant data processing tasks is adjusted; based on the anomaly alarm notifications, the abnormal subtasks are rescheduled or resources are reallocated; and based on the subtask execution progress, the load of each distributed computing node is dynamically balanced.

[0013] Secondly, embodiments of this application also provide a data acquisition and integration system, including: The data acquisition module is used to acquire raw data from multiple heterogeneous data sources, encapsulate the raw data, and form raw data packets with a unified encapsulation format. The data processing module is used to perform data cleaning, format conversion and semantic standardization on the raw data packets to generate standardized data objects with standard data structures. The data fusion module is used to perform data fusion processing on standardized data objects based on predefined metadata description information and field mapping relationships, and generate fused data records with complete business semantics. The data management module is used to collect the raw data packets, standardized data objects, and fused data records, generate data quality reports based on preset quality assessment rules, and trigger alarm notifications according to anomaly detection rules. The visualization monitoring module is used to obtain data quality reports and alarm notifications, display information through a visual interface, and also provides a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters. The task scheduling module is used to read data quality reports and user configuration parameters, parse the execution order of each processing task based on the task dependency graph, decompose the data processing task into subtasks through the scheduling engine and distribute them to distributed computing nodes for execution, and dynamically adjust the scheduling strategy according to the task execution status in the data quality report.

[0014] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data acquisition and integration method described in any of the above descriptions.

[0015] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data acquisition and integration method as described in any of the above.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: The data acquisition and integration method provided in this application firstly employs a unified encapsulation protocol to standardize and encapsulate multi-source heterogeneous raw data, ensuring the uniformity of data entry points. Secondly, based on a rule engine-based data cleaning and format conversion mechanism, data structure standardization is achieved through null value handling, anomaly filtering, and semantic transformation. Thirdly, a metadata-driven fusion model is utilized to achieve intelligent fusion of multi-source data through semantic association and field mapping. Fourthly, a full-link quality monitoring system is established, ensuring data quality through multi-dimensional indicator quantitative evaluation and threshold alarm mechanisms. Finally, a dynamic scheduling strategy based on task dependency graphs maximizes processing efficiency through distributed computing and load balancing. This method effectively solves the consistency, quality, and efficiency problems in multi-source heterogeneous data integration, providing a reliable foundation for data applications.

[0017] This application establishes a standardized data encapsulation format and employs a unified serialization protocol to encapsulate data from heterogeneous data sources such as relational databases, NoSQL databases, and API interfaces into raw data packets with standard header information and payload structures. Each data packet is equipped with a source system identifier, data format description, acquisition timestamp, and data quality marker, and its integrity and validity are verified through a data validation algorithm. This unified encapsulation method effectively solves the compatibility problem of multi-source heterogeneous data during the acquisition phase, ensuring that various data sources can be uniformly acquired and processed.

[0018] This application utilizes a configurable rule engine, enabling automated data cleaning, format conversion, and semantic standardization. Specifically, it includes: handling null values, format normalization, filtering outlier data, and eliminating duplicate data using predefined rules; converting heterogeneous data into a unified target data model using field mapping and semantic transformation mechanisms; and supporting custom conversion scripts to implement personalized processing logic. This rule-driven approach significantly improves the automation and consistency of data processing while reducing manual intervention.

[0019] This application can automatically establish semantic relationships between different standardized data objects. It identifies fields with the same business meaning through semantic matching algorithms, establishes relationships between data objects, and achieves intelligent fusion of multi-source data according to a predefined merging strategy. This method effectively breaks down information silos and achieves deep data integration.

[0020] This application establishes a comprehensive quality assessment system covering the entire process from data acquisition to fusion. It collects key quality indicators at each stage in real time and performs weighted calculations and comprehensive analysis based on pre-defined assessment rules. The system can automatically generate data quality reports containing quality scores and promptly trigger alarm notifications when anomalies are detected. This comprehensive quality monitoring ensures the reliability and traceability of data quality.

[0021] This application analyzes the execution order of processing tasks based on a task dependency graph and decomposes complex tasks into parallel-executable sub-task units through a scheduling engine. Based on real-time monitoring of task execution status, resource consumption, and data quality feedback, the task scheduling strategy is dynamically adjusted, including automatically reallocating abnormal tasks, dynamically adjusting the load on computing nodes, and adaptively adjusting task retry strategies. This intelligent scheduling mechanism significantly improves system processing efficiency and scalability.

[0022] This application can be widely used in fields such as big data platforms, financial risk control systems, industrial IoT, smart cities, healthcare, and energy and transportation, and has significant technical and application value in promoting the integration and sharing of data resources. Attached Figure Description

[0023] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the data acquisition and integration method provided in this application.

[0025] Figure 2 A schematic diagram of the data acquisition and integration system provided in this application.

[0026] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] The various embodiments of this disclosure will be described more fully in the detailed steps of the data acquisition and integration method described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0028] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1 The diagram shows a flowchart of a data acquisition and integration method in a specific embodiment. The method includes: S1: Obtain raw data from multiple heterogeneous data sources, encapsulate the raw data, and form raw data packets with a unified encapsulation format.

[0031] First, establish data connection channels with diverse and heterogeneous data sources. Specifically, for relational databases such as MySQL, configure the connection string and authentication information using a JDBC driver; for NoSQL databases such as MongoDB, set the host address and port using its native client library; for API interfaces such as RESTful APIs, configure the base URL and authentication token using an HTTP client library. These connection channels support long-lived connection pooling management to reduce the overhead of frequent connection establishment. For example, in a Java environment, Apache Commons DBCP can be used to manage the database connection pool, or the OkHttp library can be used to handle API requests.

[0032] Then, data extraction is performed based on either a timed triggering mechanism or an event-driven mechanism. Timed triggering uses a scheduling framework like Quartz to set cron expressions to periodically pull data from the data source; event-driven mechanisms, on the other hand, trigger extraction in real time by listening for changes in the data source, such as database binlogs or API webhooks. For example, when extracting data from a MySQL database, the Debezium tool can be configured to capture binlog events; when extracting from an API interface, a webhook listener can be set to respond to data update events. The extracted raw data includes structured data such as database table records and unstructured data such as JSON documents.

[0033] Meta-information is added to the extracted raw data, including source system identifiers such as data source name or ID, data format descriptions such as JSON or CSV, acquisition timestamps in ISO 8601 format, and data quality markers such as complete or missing. This metadata is appended to the data header as key-value pairs. For example, for JSON data obtained from a sensor API, the source identifier `sensor_api`, the format description `application / json`, the timestamp `2023-10-01T12:00:00Z`, and the quality marker `valid` are added.

[0034] At this point, a unified serialization protocol such as Apache Avro or Protocol Buffers is used to encapsulate the raw data. The serialization process converts the data into binary format and adds standard header information, including version number, data length, checksum, and payload structure to store the original data content. For example, when using Avro, JSON data is serialized into Avro objects according to a predefined schema, with the header containing magic bytes and schema ID.

[0035] Finally, the integrity and validity of the original data packet are verified using a data verification algorithm. For example, the CRC32 algorithm is used to calculate the checksum of the payload and compare it with the checksum in the header; at the same time, it is verified whether the data packet conforms to the schema definition. If the verification fails, the error is recorded and a retry mechanism is triggered.

[0036] S2: Perform data cleaning, format conversion, and semantic standardization on the original data packets to generate standardized data objects with standard data structures.

[0037] In a specific implementation, data cleaning is first performed on the original data packets. Null value handling includes filling numeric fields with the mean or median; for text fields, using default values ​​or deleting records. Format normalization includes standardizing date formats, such as converting to YYYY-MM-DD, and numeric formats, such as retaining two decimal places. Abnormal data filtering is implemented using a rule engine such as Drools, for example, filtering out records where the age field is greater than 150. Duplicate data removal is based on primary keys or hash values, using Bloom filters or distributed collections such as Redis Sets to detect and remove duplicates.

[0038] Then, through field mapping and semantic transformation, the heterogeneous data in the original data packet is transformed into a unified target data model. Field mapping maps source field names to target field names based on configuration tables such as CSV files, for example, mapping first_name to firstName. Semantic transformation uses conversion functions such as UDFs to handle unit conversions or encoding conversions, such as converting temperature units from Fahrenheit to Celsius, or mapping status code A to Active. The target data model adopts a common structure, such as Apache Avro Schema or JSON Schema, defining field types and constraints.

[0039] Finally, standardized data objects are generated based on the target data model. The cleaned and transformed data is encapsulated into objects, such as POJO classes in Java, containing getter / setter methods for fields. The objects are serialized to a target format such as Parquet or ORC, and metadata such as the model version is appended. For example, the generated standard object might contain a string field `id`, a numeric field `value`, and a timestamp, and its conformity to the model definition is verified.

[0040] S3: Based on predefined metadata description information and field mapping relationships, perform data fusion processing on standardized data objects to generate fused data records with complete business semantics.

[0041] In a specific implementation, firstly, semantic relationships between different standardized data objects are established based on predefined metadata description information. This metadata description information is stored in a metadata repository such as Apache Atlas and includes the business meaning, origin, and relationship definitions of the data objects. For example, standardized objects from user databases and log files can be associated through primary and foreign key relationships such as user IDs or business rules such as time windows.

[0042] Then, field matching rules are determined based on semantic relationships. Field matching is performed on standardized data objects from different data sources according to predefined field mapping relationships, such as in a YAML configuration file. For example, a matching rule might specify that an association should be performed when the userId of a user object is equal to the customerId of an order object. The matching process uses join algorithms such as hash join or sort-merge join, and is executed in a distributed computing framework such as Apache Spark.

[0043] Finally, the standardized data objects are merged based on the matching results. During the merge, conflicting data is handled; for example, multiple values ​​for the same field are addressed using priority strategies such as prioritizing the latest timestamp or using aggregation functions such as summation. Merged data records are generated, containing all related fields and adding merge metadata such as the merge time. For example, after merging user and order data, the generated record includes the user's name, order amount, and order time.

[0044] S4: Collect the original data packets, standardized data objects, and fused data records, generate a data quality report based on preset quality assessment rules, and trigger alarm notifications according to anomaly detection rules.

[0045] In a specific implementation, raw data packets are collected, and the timeliness index is obtained by calculating the difference between the current time and the collection timestamp. For example, the delay in seconds is obtained by subtracting the timestamp in the data packet header from the current system time, and a threshold is set such that a delay exceeding 60 seconds is considered a delay. The index is stored in numerical form for subsequent analysis.

[0046] Collect standardized data objects and obtain format conformity metrics by verifying the match between their data structure and the target data model. Validation includes field type checks, such as whether string lengths meet requirements, and constraint checks, such as whether non-empty fields are missing. Use a pattern validation library such as Apache Avro Validator to calculate the percentage of match; for example, 95% indicates that a small number of field types do not match.

[0047] Collect and integrate data records, and use a rule engine to verify the logical relationships between related data to obtain data consistency metrics. Rule engines such as Drools execute predefined business rules, such as checking that order amounts cannot be negative or that user ages must be greater than 18. Consistency metrics are calculated based on rule pass rates, such as the percentage of pass rules out of the total number of rules.

[0048] At this point, based on preset quality assessment rules, a weighted statistical algorithm is used to comprehensively analyze the timeliness, format standardization, and data consistency indicators. The weights are configured, such as timeliness accounting for 30%. A data quality score is calculated using the formula: Data Quality Score = Timeliness Score × First Weight + Standardization Score × Second Weight + Consistency Score × Third Weight. A data quality report is generated, including score details and indicator trends.

[0049] Finally, when the data quality score falls below a preset threshold, such as 80 points, an alarm notification is generated via an alarm trigger. The alarm trigger integrates with notification tools such as email SMTP or Slack webhook to send a message containing details of the anomaly, such as the metric value. Simultaneously, alarm records are stored in a database for later retrieval.

[0050] S5: Obtain data quality reports and alarm notifications, display the information through a visual interface, and provide a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters.

[0051] In a specific implementation, the data quality report is first obtained, and its data quality score, collection timeliness indicators, format standardization indicators, and data consistency indicators are parsed. The parsing uses a JSON parsing library such as Jackson to convert the report into an internal object for easy use by the visualization component.

[0052] Then, the quality score is displayed as a dashboard using visualization components, and the indicators are shown as trend charts. Front-end frameworks such as ECharts render the dashboard, displaying the current score and level (e.g., green indicates good); the trend charts show historical indicator changes and support time range filtering. For example, a line chart shows the fluctuations in timeliness indicators over the past 7 days.

[0053] Simultaneously, alarm notifications are received and displayed in real-time through an alarm panel, which also provides a historical alarm query function. The alarm panel uses WebSocket for real-time updates, displaying a list of alarm times, sources, and levels; historical queries support filtering by time and keyword searching, with data loaded paginated from the database.

[0054] Finally, the system receives user-inputted data processing rules and system parameter configurations via a configuration interface. This interface is a REST API that accepts JSON-formatted input, such as data cleaning rules or scheduling parameters. It performs format validation (e.g., field type checks) and logical validation (e.g., rule conflict detection). Once validation is successful, the configuration is stored as structured user configuration parameters, for example, written to a MySQL configuration table or a ZooKeeper node.

[0055] S6: Reads data quality reports and user configuration parameters, parses the execution order of each processing task based on the task dependency graph, decomposes the data processing task into sub-tasks through the scheduling engine and distributes them to distributed computing nodes for execution, and dynamically adjusts the scheduling strategy according to the task execution status in the data quality report. In a specific implementation, the data quality report is first read to obtain the data quality score and related alarm notifications. Simultaneously, the task priorities (high, medium, low) and resource allocation rules (CPU and memory quotas) configured in the user configuration parameters are read. This data is loaded into the scheduler's memory as a basis for decision-making.

[0056] At this point, based on a predefined task dependency graph, the execution logic relationships between data acquisition, data cleaning, and data fusion tasks are analyzed. The dependency graph is represented using a Directed Acyclic Graph (DAG), where vertices represent tasks and edges represent dependencies. For example, the data cleaning task depends on the completion of the data acquisition task. The execution order is determined through graph traversal algorithms such as topological sorting.

[0057] Then, based on the execution logic and task priorities, the scheduling engine decomposes each data processing task into subtasks that can be executed in parallel. Scheduling engines like Apache Airflow break down tasks into fine-grained units; for example, a data cleaning task is decomposed into multiple subtasks based on data shards. Subtasks are accompanied by metadata, such as the input data path and resource requirements.

[0058] Furthermore, based on resource allocation rules, subtasks are distributed to distributed computing nodes for execution. Compute node clusters, such as Kubernetes clusters, receive subtasks and allocate container resources according to resource rules. The execution progress and resource consumption of subtasks are monitored, and metrics such as CPU utilization and task duration are collected using monitoring tools like Prometheus.

[0059] Finally, based on data quality scores and related anomaly alerts, and considering the execution status of subtasks, the task scheduling strategy is dynamically adjusted. When the quality score falls below a threshold, the execution priority of related data processing tasks is adjusted; for example, the priority of data cleaning tasks is increased to improve quality. Based on anomaly alerts, abnormal subtasks are rescheduled or resources are reallocated; for example, failed tasks are moved to idle nodes. Based on the subtask execution progress, the load on each distributed computing node is dynamically balanced, and load balancing algorithms such as least connections are used to reallocate tasks.

[0060] In this embodiment, by constructing a complete technical system based on a unified encapsulation protocol for multi-source data acquisition, rule engine-driven intelligent processing, metadata-driven data fusion, end-to-end quality monitoring, and dynamic task scheduling, the system achieves standardized and automated processing of data from acquisition to integration. This effectively solves the compatibility, consistency, and quality issues in the integration of multi-source heterogeneous data, and significantly improves the efficiency, reliability, and scalability of data processing.

[0061] like Figure 2 As shown, the following are embodiments of the data acquisition and integration system provided in this disclosure. This system and the data acquisition and integration methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the data acquisition and integration system, please refer to the embodiments of the above data acquisition and integration methods.

[0062] A data acquisition and integration system, comprising: The data acquisition module is used to obtain raw data from multiple heterogeneous data sources, encapsulate the raw data, and form raw data packets with a unified encapsulation format.

[0063] The data processing module is used to perform data cleaning, format conversion, and semantic standardization on the raw data packets to generate standardized data objects with standard data structures.

[0064] The data fusion module is used to perform data fusion processing on standardized data objects based on predefined metadata description information and field mapping relationships, and generate fused data records with complete business semantics.

[0065] The data management module is used to collect the raw data packets, standardized data objects, and fused data records, generate data quality reports based on preset quality assessment rules, and trigger alarm notifications according to anomaly detection rules.

[0066] The visualization monitoring module is used to obtain data quality reports and alarm notifications, display the information through a visual interface, and provide a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters.

[0067] The task scheduling module is used to read data quality reports and user configuration parameters, parse the execution order of each processing task based on the task dependency graph, decompose the data processing task into subtasks through the scheduling engine and distribute them to distributed computing nodes for execution, and dynamically adjust the scheduling strategy according to the task execution status in the data quality report.

[0068] The data acquisition and integration system provided in this embodiment establishes a standardized data processing link, transforms raw data packets into standardized data objects, and finally generates fused data records with business semantics. Through a multi-layer quality monitoring system, key indicators of each stage of data processing are quantitatively evaluated to ensure controllable data quality. Based on real-time quality feedback and user configuration, the task scheduling strategy is dynamically adjusted to achieve intelligent allocation of data processing resources.

[0069] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0070] The data acquisition and integration method provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0071] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0072] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0073] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0074] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0075] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0076] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0077] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0078] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0079] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0080] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0081] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0082] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0083] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0084] The aforementioned electronic device realizes the data acquisition and integration method of this application by constructing a complete technical system based on a unified encapsulation protocol for multi-source data acquisition, rule engine-driven intelligent processing, metadata-driven data fusion, end-to-end quality monitoring, and dynamic task scheduling. This achieves the beneficial effect of comprehensively improving the standardization, automation, and intelligence level of the data acquisition and integration process, as well as the system processing efficiency.

[0085] The storage medium provided in this application stores a program product capable of implementing data acquisition and integration methods.

[0086] Data acquisition and integration methods include: Raw data is obtained from multiple heterogeneous data sources, and the raw data is encapsulated to form a raw data packet with a unified encapsulation format. Perform data cleaning, format conversion, and semantic standardization on the raw data packets to generate standardized data objects with standard data structures; Based on predefined metadata description information and field mapping relationships, data fusion processing is performed on standardized data objects to generate fused data records with complete business semantics; The system collects the original data packets, standardized data objects, and fused data records, generates a data quality report based on preset quality assessment rules, and triggers alarm notifications according to anomaly detection rules. Get data quality reports and alarm notifications, display the information through a visual interface, and provide a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters; The system reads data quality reports and user configuration parameters, parses the execution order of each processing task based on the task dependency graph, decomposes the data processing task into subtasks through the scheduling engine and distributes them to distributed computing nodes for execution, and dynamically adjusts the scheduling strategy according to the task execution status in the data quality report.

[0087] In some possible implementations, the data acquisition and integration method of this disclosure can be implemented as a program product including program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0088] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0089] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data acquisition and integration method, characterized in that, include: Raw data is obtained from multiple heterogeneous data sources, and the raw data is encapsulated to form a raw data packet with a unified encapsulation format. Perform data cleaning, format conversion, and semantic standardization on the raw data packets to generate standardized data objects with standard data structures; Based on predefined metadata description information and field mapping relationships, data fusion processing is performed on standardized data objects to generate fused data records with complete business semantics; The system collects the original data packets, standardized data objects, and fused data records, generates a data quality report based on preset quality assessment rules, and triggers alarm notifications according to anomaly detection rules. Get data quality reports and alarm notifications, display the information through a visual interface, and provide a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters; The system reads data quality reports and user configuration parameters, parses the execution order of each processing task based on the task dependency graph, decomposes the data processing task into subtasks through the scheduling engine and distributes them to distributed computing nodes for execution, and dynamically adjusts the scheduling strategy according to the task execution status in the data quality report.

2. The data acquisition and integration method according to claim 1, characterized in that, The process of obtaining raw data from multiple heterogeneous data sources and encapsulating the raw data to form a raw data packet with a unified encapsulation format includes: Establish data connection channels with relational databases, NoSQL databases, and API interfaces; Based on a timed triggering mechanism or an event-driven mechanism, data extraction operations are performed through a data connection channel to obtain raw data; Add source system identifier, data format description, collection timestamp and data quality marker to the raw data; The original data is encapsulated using a unified serialization protocol to form an original data packet with standard header information and payload structure; The integrity and validity of the original data packet are verified using a data verification algorithm.

3. The data acquisition and integration method according to claim 2, characterized in that, The process of performing data cleaning, format conversion, and semantic standardization on the original data packets to generate standardized data objects with standard data structures includes: Perform null value processing, format normalization, abnormal data filtering, and duplicate data elimination operations on the raw data packets; Through field mapping and semantic transformation processing, the heterogeneous data in the original data packet is converted into a unified target data model; Standardized data objects are generated based on the target data model.

4. The data acquisition and integration method according to claim 3, characterized in that, The process of performing data fusion processing on standardized data objects based on predefined metadata description information and field mapping relationships to generate fused data records with complete business semantics includes: Establish semantic relationships between different standardized data objects based on predefined metadata description information; Based on the semantic association, field matching rules are determined, and standardized data objects from different data sources are matched according to predefined field mapping relationships. Based on the matching results, standardized data objects are merged to generate fused data records.

5. The data acquisition and integration method according to claim 4, characterized in that, The system collects the original data packets, standardized data objects, and fused data records; generates a data quality report based on preset quality assessment rules; and triggers alarm notifications according to anomaly detection rules, including: The original data packets are collected, and the timeliness index of the collection is obtained by calculating the difference between the current time and the collection timestamp. Collect the standardized data objects and obtain format standardization indicators by verifying the matching degree between the data structure of the standardized data objects and the target data model; Collect the fused data records and use a rule engine to verify the logical relationship between the related data in the fused data records to obtain data consistency indicators; Based on preset quality assessment rules, a weighted statistical algorithm is used to comprehensively analyze the data collection timeliness index, format standardization index, and data consistency index to calculate the data quality score and generate a data quality report. When the data quality score is lower than a preset threshold, a corresponding alarm notification is generated through an alarm trigger.

6. The data acquisition and integration method according to claim 5, characterized in that, The process of obtaining data quality reports and alarm notifications is displayed through a visual interface. It also provides a configuration interface to receive user-input processing rules and parameter configurations, generating user-configured parameters, including: Obtain the data quality report and analyze its data quality score, data collection timeliness index, format standardization index, and data consistency index; The quality score is displayed in the form of a dashboard using visualization components, and the indicators of collection timeliness, format standardization, and data consistency are displayed in the form of trend charts. Receive alarm notifications, display alarm notifications in real time through the alarm panel, and provide historical alarm query function; The system receives user-inputted data processing rules and system parameter configurations through a configuration interface, performs format and logic validation, and stores the validated data processing rules and system parameter configurations as structured user configuration parameters.

7. The data acquisition and integration method according to claim 6, characterized in that, The process involves reading data quality reports and user configuration parameters, parsing the execution order of each processing task based on a task dependency graph, decomposing data processing tasks into subtasks through a scheduling engine and distributing them to distributed computing nodes for execution, and dynamically adjusting the scheduling strategy based on the task execution status in the data quality report. This includes: Read the data quality report to obtain the data quality score and related alarm notifications; Read the task priority and resource allocation rules configured in the user configuration parameters; Based on a predefined task dependency graph, the execution logic relationships between data acquisition, data cleaning, and data fusion tasks are analyzed. Based on the execution logic relationship and task priority, the scheduling engine decomposes each data processing task into sub-tasks that can be executed in parallel. Based on the resource allocation rules, the subtasks are distributed to distributed computing nodes for execution; Monitor the execution progress and resource consumption of the subtasks; Based on the data quality score and related anomaly alarm notifications, and in conjunction with the subtask execution status, the task scheduling strategy is dynamically adjusted; when the quality score is lower than the threshold, the execution priority of the relevant data processing tasks is adjusted; based on the anomaly alarm notifications, the abnormal subtasks are rescheduled or resources are reallocated; and based on the subtask execution progress, the load of each distributed computing node is dynamically balanced.

8. A data acquisition and integration system, characterized in that, The system employs the data acquisition and integration method as described in any one of claims 1 to 7; The system includes: The data acquisition module is used to acquire raw data from multiple heterogeneous data sources, encapsulate the raw data, and form raw data packets with a unified encapsulation format. The data processing module is used to perform data cleaning, format conversion and semantic standardization on the raw data packets to generate standardized data objects with standard data structures. The data fusion module is used to perform data fusion processing on standardized data objects based on predefined metadata description information and field mapping relationships, and generate fused data records with complete business semantics. The data management module is used to collect the raw data packets, standardized data objects, and fused data records, generate data quality reports based on preset quality assessment rules, and trigger alarm notifications according to anomaly detection rules. The visualization monitoring module is used to obtain data quality reports and alarm notifications, display information through a visual interface, and also provides a configuration interface to receive user input of processing rules and parameter configurations, and generate user configuration parameters. The task scheduling module is used to read data quality reports and user configuration parameters, parse the execution order of each processing task based on the task dependency graph, decompose the data processing task into subtasks through the scheduling engine and distribute them to distributed computing nodes for execution, and dynamically adjust the scheduling strategy according to the task execution status in the data quality report.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data acquisition and integration method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data acquisition and integration method as described in any one of claims 1 to 7.

Images