Data integration method, device and system

Abstract

The invention provides a data integration method, device and system, and relates to the technical field of data processing, and the method comprises the steps: obtaining a data collection scheme and a data processing scheme configured by a center end; according to a task type of a data acquisition task and the data acquisition scheme, acquiring source data corresponding to the data acquisition task in a multi-source data acquisition end; according to the data processing scheme, performing data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain target data; and synchronizing the target data to the central end. According to the method, comprehensive, diversified, adaptive and real-time data integration processing is realized, so that the consistency, the accuracy and the availability of the data are ensured, and the efficiency and the accuracy of data integration are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data integration method, device and system. Background Art

[0002] With the rapid development of medical informatization, data collection platforms play an increasingly critical role in the medical industry. At present, a major challenge facing hospital information systems is the problem of data interconnection and interoperability. The lack of data connectivity between different systems has led to a large waste of resources. Data integration can integrate data from multiple different data sources so that data can be interconnected between different systems. Therefore, how to effectively achieve data integration is an important topic that the industry urgently needs to study.

[0003] In the related art, open source data integration tools are usually used to integrate data from multiple different data sources. However, these open source data integration tools mainly provide simple data cleaning and conversion functions, and their data quality assurance capabilities are relatively poor. Therefore, in scenarios with strict data quality requirements, these open source data integration tools often find it difficult to achieve efficient and accurate data integration, resulting in poor data integration performance. Summary of the invention

[0004] The present invention provides a data integration method, device and system to solve the problem in the prior art that in scenarios with strict data quality requirements, open source data integration tools are often difficult to achieve efficient and accurate data integration, resulting in poor data integration performance, thereby improving the efficiency and accuracy of data integration.

[0005] The present invention provides a data integration method, comprising: Obtain the data collection and processing solutions configured on the central end; According to the task type of the data collection task and the data collection scheme, source data corresponding to the data collection task is collected in the multi-source data collection terminal; According to the data processing scheme, the source data is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain target data; The target data is synchronized to the central end.

[0006] According to a data integration method provided by the present invention, the data processing scheme is used to perform data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain target data, including: According to the data standardization sub-scheme in the data processing scheme, the source data is subjected to data standardization processing to obtain standardized data; According to the data quality control sub-scheme in the data processing scheme, performing data quality detection processing on the standardized data; In the case where the standardized data passes the data quality detection, the standardized data is converted into a data format according to the data conversion sub-scheme in the data processing scheme to obtain the target data.

[0007] According to a data integration method provided by the present invention, the data standardization sub-scheme in the data processing scheme is used to perform data standardization processing on the source data to obtain standardized data, including: Based on the abstract syntax tree, extract fields from the source data, and generate standardized processing statements according to the mapping relationship between the extracted fields and the standardized fields in the data standardization sub-scheme; According to the standardization processing statement, the source data is subjected to data standardization processing to obtain the standardized data.

[0008] According to a data integration method provided by the present invention, the data quality detection process is performed on the standardized data according to the data quality control sub-scheme in the data processing scheme, including: Analyze and obtain at least one-dimensional quality control information from the data quality control sub-scheme; According to each quality detection condition corresponding to the quality control information of each dimension, the standardized data is subjected to data quality detection to obtain a detection score of the standardized data under each quality detection condition; The detection scores under all quality detection conditions corresponding to the quality control information of each dimension are respectively integrated; According to the fusion result, it is determined whether the standardized data passes the data quality detection.

[0009] According to a data integration method provided by the present invention, the at least one-dimensional quality control information includes at least one item of accuracy quality control information, integrity quality control information, logic quality control information, timeliness quality control information and stability quality control information.

[0010] According to a data integration method provided by the present invention, the data processing scheme is used to perform data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain target data, including: Determine a target data mapping mode corresponding to the source data according to the type of the data acquisition terminal to which the source data belongs; According to the target data mapping mode, performing a data mapping operation on the source data to obtain data to be processed; According to the data processing scheme, data standardization processing, data quality detection processing and data format conversion processing are performed on the data to be processed to obtain the target data.

[0011] According to a data integration method provided by the present invention, the source data corresponding to the data collection task is collected in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme, including: Determining a target data acquisition terminal corresponding to the data acquisition task in the multi-source data acquisition terminal according to the task type and the acquisition terminal configuration information in the data acquisition scheme; According to the collection frequency and the content of the data to be collected in the data collection scheme, the source data is collected in the target data collection terminal.

[0012] According to a data integration method provided by the present invention, synchronizing the target data to the central end includes: For the current data synchronization, according to the current upload flag, the current data block to be uploaded is loaded into the target data; According to the data synchronization batch size, the current data block to be uploaded is transmitted to the central end in the form of a log; Receiving an identifier of the successfully saved log returned by the central end; According to the identification, determining the next upload flag; The next data synchronization is performed according to the next upload flag, until all data blocks in the target data are synchronized to the central end.

[0013] According to a data integration method provided by the present invention, the method further includes: When an alarm is detected in any of the execution processes of the data acquisition task, including the acquisition process, the processing process, and the synchronization process, the alarm is compensated for according to the abnormal information type corresponding to the alarm; When it is determined that the alarm is eliminated, the operation corresponding to any execution process is re-executed on the data collection task.

[0014] According to a data integration method provided by the present invention, the source data corresponding to the data collection task is collected in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme, including: If a collection interruption occurs during the collection process of any batch of data blocks corresponding to the data collection task, data repair is performed according to the position of the data blocks of any batch; When it is determined that the data repair is successful, the data blocks of any batch and the data blocks after the data blocks of any batch are collected again in the multi-source data collection end according to the interruption position of the data blocks of any batch, the task type and the data collection plan, until all the data blocks corresponding to the data collection task are collected; The source data is generated according to all data blocks corresponding to the data acquisition task.

[0015] The present invention also provides a data integration device, comprising: An acquisition unit, used to acquire a data collection scheme and a data processing scheme configured by the central end; A collection unit, configured to collect source data corresponding to the data collection task in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme; A processing unit, configured to perform data standardization processing, data quality detection processing and data format conversion processing on the source data according to the data processing scheme to obtain target data; A synchronization unit is used to synchronize the target data to the central end.

[0016] The present invention also provides a data integration system, comprising a data collection end, an edge end and a central end; The data collection end is connected to the edge end in communication, and the edge end is connected to the central end in communication via a reverse proxy server and an application programming interface gateway; Each edge terminal is used to execute any one of the data integration methods described above.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the data integration methods described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data integration method as described above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the data integration methods described above.

[0020] The data integration method, device and system provided by the present invention adaptively and accurately collect the source data required for data integration in the multi-source data acquisition terminal according to the data acquisition scheme configured by the central end and the task type of the data acquisition task, and use the comprehensive and rich data quality control logic in the data processing scheme configured by the central end, including but not limited to data standardization processing, data quality detection processing and data format conversion processing to perform real-time processing of the source data to ensure that the data is transmitted between the source and the target with almost no delay, effectively improve the real-time performance of data integration, and realize comprehensive and diversified data standardization governance to improve data quality, ensure the consistency, accuracy and availability of data, thereby significantly improving the efficiency and accuracy of data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the data integration method provided by the present invention.

[0023] Figure 2 It is a structural schematic diagram of the data integration system provided by the present invention.

[0024] Figure 3 It is a flow chart of the steps of obtaining the data collection scheme and the data processing scheme provided by the present invention.

[0025] Figure 4 It is a schematic diagram of the data synchronization process provided by the present invention.

[0026] Figure 5 It is a structural schematic diagram of the data integration device provided by the present invention.

[0027] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The data integration method, device and system provided in the present application can be applied to various data processing scenarios, including but not limited to medical data processing scenarios, Internet of Things data processing scenarios, etc. This embodiment does not specifically limit this. The following takes the medical data processing scenario as an example to describe the data integration method, device and system provided in the present application.

[0030] With the rapid development of medical informatization, data collection platforms play an increasingly critical role in the medical industry. At present, a major challenge facing hospital information systems is the problem of data interconnection and interoperability. The lack of data connectivity between different systems has led to a large waste of resources. In addition, with the advancement of big data and artificial intelligence technologies, higher standards have been put forward for the efficiency and quality control of data integration, especially in the sensitive and complex medical and health fields. Therefore, how to effectively achieve data integration is an important topic that the industry urgently needs to study.

[0031] In related technologies, open source data integration tools, such as Kettle (Pentaho Data Integration, PDI) tools, are usually used to integrate data from multiple different data sources. Among them, Kettle is an open source data integration tool that provides powerful extraction, transformation, and loading (ETL) functions and is widely used in data warehouse construction, data migration, and daily data processing tasks. Kettle allows users to design and execute data transformation and ETL processes in a visual way through its graphical user interface, Spoon. This approach simplifies the design and management of complex data integration tasks and makes it easy for non-technical users to operate. Kettle can connect to a variety of data sources, including various SQL databases (such as MySQL, Oracle, PostgreSQL) and various file formats (such as CSV, Excel). This makes Kettle very suitable for handling data integration requirements from heterogeneous environments. Kettle provides a wide range of data transformation options, including data filtering, sorting, aggregation, and complex data logic processing, supporting the implementation of complex data integration strategies. Kettle supports integration into job scheduling systems to realize the automated execution of ETL tasks. At the same time, it provides detailed logging and error handling mechanisms to help monitor the execution status of data integration processes.

[0032] Although Kettle is a mature data integration tool that performs well in data conversion and batch processing, it still has some limitations when facing some specific needs in the current medical information field. These limitations mainly include: First, lack of real-time data processing capabilities: Kettle is mainly designed for batch processing operations and has poor adaptability to scenarios that require real-time data synchronization. This may cause delays in medical environments that require instant data feedback, affecting the timeliness of decision-making and operations.

[0033] Second, insufficient quality control: Although Kettle provides simple data cleaning and conversion functions, it lacks a comprehensive data quality control mechanism, resulting in poor data quality assurance capabilities. In medical data processing, data accuracy and integrity are crucial, and any quality issues may have a direct impact on patient safety.

[0034] In summary, in scenarios with strict data quality requirements, existing open source data integration tools often find it difficult to achieve efficient and accurate data integration, resulting in poor data integration performance.

[0035] In this regard, the present embodiment provides a more robust, flexible and efficient data integration method, which performs real-time data processing and synchronization through rich data quality control logic, including but not limited to data standardization processing, data quality detection processing and data format conversion processing, to ensure that data is transmitted between the source and the target with almost no delay, adapting to the high real-time requirements of medical information systems, and improving data quality, realizing data standardization governance, ensuring the accuracy and consistency of data during the synchronization process, thereby improving the efficiency, accuracy and docking cost of data integration, so that this data integration method can be better applied to medical environments with extremely high requirements for data real-time, quality and stability, improving the efficiency of medical services, and providing stronger protection for patient safety.

[0036] Figure 1 The figure is a flow chart of the data integration method provided by the present invention. The method can be applied to a data integration system.

[0037] Figure 2 is a schematic diagram of the structure of the data integration system provided by the present invention; Figure 2 As shown in the figure, the system includes a data collection end, an edge end, and a center end. Among them, the intranet of the data collection end is deployed with at least a hospital information system (HIS), a picture archiving and communication system (PACS), a laboratory information system (LIS), and an electronic medical record system (EMR); the data collection end can provide various types of medical data through the systems deployed in the intranet.

[0038] The edge end is at least configured with a collection platform and a basic service component; the collection platform is used to execute the data integration method provided in this application, and specifically can process the source data collected from the data collection end into target data and then integrate it into the central end for data sharing management, thereby realizing data integration. The collection platform at least includes an acquisition unit, a collection unit, a processing unit and a synchronization unit; the processing unit includes but is not limited to a master data management (Master Data Management, MDM) system for providing data conversion functions, a quality control system for performing data quality detection and processing functions, and a standardized data management system for performing data standardization processing functions. The basic service component is used to provide the necessary computing, storage and network resources for the edge end, and support various basic needs of the collection platform. The basic service component includes at least a database, a registration or configuration center, and a user center. Among them, each user, such as hospital workers, data implementation personnel, IT workers, etc., can interact with the edge end through a reverse proxy.

[0039] The central end is at least configured with a collection platform, a message middleware, a master data management system, a quality control system, a standardized data management system, and a basic service component. Among them, the collection platform is used to collect data transmitted by the edge end; the message middleware can interact with the collection platform of the central end to realize asynchronous data storage, and can also interact with the master data management system, the quality control system, and the standardized data management system to realize asynchronous data transmission. The master data management system, the quality control system, and the standardized data management system are respectively used to provide data conversion sub-schemes, data quality control sub-schemes, and data standardization sub-schemes, and are respectively used to provide data conversion functions, data quality control functions, and data standardization processing functions. The basic service component is used to provide the necessary computing, storage, and network resources for the central end to support the basic needs of the collection platform. The basic service component includes at least a database, a registration or configuration center, and a user center. Among them, the edge end and each user, such as the central developer, data processing manager, etc., can interact with the central end for data through the reverse proxy server and the application programming interface gateway (also known as the API gateway) in turn.

[0040] It should be noted that the execution subject of this method may be an edge terminal equipped with a collection platform. Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0041] Step 110, obtaining the data collection scheme and data processing scheme configured by the central end.

[0042] Figure 3 It is a flow chart of the steps of obtaining the data collection scheme and the data processing scheme provided by the present invention; Figure 3As shown, here, the data processing scheme is a scheme configured by the central end for data standardization processing, data quality detection processing and data format conversion processing. Specifically, when the central developer triggers the operation of issuing the standard data set version, the central end acquisition platform (hereinafter referred to as the central acquisition platform) sends the standard data set to the central end standardized data management system (hereinafter referred to as the central standard management system), so that the central standard management system generates a data standardization sub-scheme and returns it to the central acquisition platform; in addition, the central acquisition platform can also send the standard data set to the central end quality control system (hereinafter referred to as the central quality control system), so that the central quality control system generates a data quality control sub-scheme and returns it to the central acquisition platform; the central acquisition platform can integrate the acquired data standardization sub-scheme, data quality control sub-scheme and data conversion sub-scheme to obtain a data processing scheme, so as to coordinate the global data acquisition and synchronization operation. The acquisition here can be downloading or receiving, etc., and this implementation does not specifically limit this.

[0043] Among them, the data standardization sub-scheme is generated by the standardized data management system at the center end according to the preset data format, data type, and data length to configure the standardized information. It is used to ensure that all collected and synchronized data meet unified standards to maintain data consistency and availability in the entire system. The data quality control sub-scheme is configured by the quality control system at the center end according to the data set standardization file, including specific data quality control logic and procedures. It is used to ensure the quality of data during the collection and synchronization process and prevent erroneous or incomplete data from affecting data analysis and decision-making. The data conversion sub-scheme is formed by the data management system at the center end according to the data set standardization file, establishing the data mapping relationship between different systems, and configuring the corresponding conversion logic information and dictionary mapping information according to the data mapping relationship. It is used to ensure the consistency and correctness of data when converting data between different systems or modules.

[0044] like Figure 3 As shown, the data collection plan here is generated by the center according to the template based on actual business needs to guide the edge end to collect data. It is used to ensure that the collection activities are carried out in accordance with established standards and rules.

[0045] Optionally, the edge end can send configuration acquisition status to the central end to synchronously obtain the latest configured data collection plan and data processing plan from the central collection platform in real time to ensure that the collection tasks performed by the edge end are consistent with the requirements of the central end, thereby providing data integration processing performance.

[0046] Step 120 , collecting source data corresponding to the data collection task in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme.

[0047] Optionally, after obtaining the data collection plan and data processing plan configured by the central end, the edge end may first load the data collection task that needs to be processed currently; the data collection task may be configured and generated by the user through the specified task input by the user in the front-end interface, or it may be triggered and generated by other controllers, which is not specifically limited in this embodiment.

[0048] After acquiring the data collection task, the edge end may be the task type and data collection scheme of the joint data collection task, and collect the source data corresponding to the data collection task in the multi-source data collection end. For example, it may be based on the task type and the data collection scheme to perform an overall match to determine the target data collection end storing the active data, and then collect and obtain the source data corresponding to the data collection task from the target data collection end, or it may be based on the task type matching and the data collection scheme to match in sequence to determine the target data collection end storing the active data, and then collect and obtain the source data corresponding to the data collection task from the target data collection end, and this embodiment does not specifically limit this.

[0049] The multi-source data acquisition terminal here includes but is not limited to various types of medical equipment terminals, databases, etc., and this embodiment does not make specific limitations on this.

[0050] Step 130: According to the data processing scheme, the source data is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain target data.

[0051] Optionally, after acquiring the source data, the edge end may directly perform data standardization, data quality detection and data format conversion on the source data according to the data processing plan to obtain standardized, high-quality target data, or it may perform unified data mapping on the source data and then perform data standardization, data quality detection and data format conversion on the source data according to the data processing plan to obtain standardized, high-quality target data. This embodiment does not specifically limit this.

[0052] If data standardization fails, data quality inspection fails, or data format conversion fails during processing, the edge needs to display the corresponding quality control report to prompt the delivery implementation personnel to adjust the data quality so that users can easily process and optimize the data flow, thereby achieving real-time adjustment of data quality and improving data flow management and orchestration capabilities.

[0053] If during the processing, the data standardization process is successful, the data quality test is passed, and the data format conversion process is successful, the edge end can store the standardized and high-quality target data obtained in this way to the internal database, and initiate the data synchronization process of the central end to synchronize the target data to the central end.

[0054] The data standardization processing, data quality detection processing and data format conversion processing here may be performed sequentially according to a set order, or may be performed in parallel, and this embodiment does not specifically limit this.

[0055] The data standardization processing here is used to connect the data according to the standardized information in the data processing plan, so that the mapped data is consistent with the requirements of the standardized information. For example, the standardized information contains at least sub-segments such as name, date of birth and medical record number, and the medical record number is unique. After the data is standardized through this standardized information, the processed data must also have sub-segments such as name, date of birth and medical record number, and the medical record number is unique.

[0056] The data quality detection process here is used to check whether the quality of the data meets the data quality requirements, including but not limited to one or more of accuracy, completeness, logic, timeliness and stability, which is not specifically limited in this embodiment.

[0057] The data format conversion processing here is used to convert the data format to a format consistent with the storage format of the central end. For example, if the date format in the source system (that is, the data collection system corresponding to the source data) is "MM / DD / YYYY", and the date format required by the target system (that is, the central end) is "YYYY-MM-DD", then it is necessary to convert the date format in the data collected from the source system into the date format required by the target system according to the data conversion information in the data processing plan.

[0058] Step 140: synchronizing the target data to the central end.

[0059] After obtaining the target data, the edge end can synchronize the target data to the central end in real time to achieve cloud-edge synchronization.

[0060] The data synchronization here can be in the form of logs or message queues, etc., to directly synchronize the target data to the central end in real time; it can also be based on specific business needs, after the target data is filtered, sorted and aggregated to generate a specific business view, and then synchronized to the central end in real time in the form of logs or message queues, etc. This embodiment does not make specific limitations on this.

[0061] It should be noted that during the data synchronization process, a data reconciliation operation can also be set to reconcile the target data to ensure the synchronization of the target data at the edge and the center. Specifically, at a set time, the data difference between the target data stored in the database at the edge and the target data stored in the database at the center is calculated. If there is a data difference between the two, the edge triggers the corresponding data acquisition task to retransmit the data to achieve data repair, thereby completing the reconciliation compensation. After completing the reconciliation compensation, the processing unit at the edge can also display the details of the reconciliation compensation, thereby achieving real-time synchronization verification and difference compensation of the target data at the edge and the center, that is, synchronization of heterogeneous databases to ensure data consistency.

[0062] In addition, after the center synchronizes to the target data, it can also use the local collection platform to collect data and use the local quality control system to verify the data quality. If the data quality verification is not met, the corresponding quality control report will be displayed to prompt the delivery implementation personnel to adjust the data quality. If the data quality verification is met, the target data will be stored in the internal database to achieve the purpose of data aggregation and sharing of different edge terminals.

[0063] The method provided in this embodiment adaptively and accurately collects the source data required for data integration in the multi-source data collection terminal according to the data collection scheme configured at the central end and the task type of the data collection task, and uses the comprehensive and rich data quality control logic in the data processing scheme configured at the central end, including but not limited to data standardization processing, data quality detection processing and data format conversion processing to perform real-time processing of the source data to ensure that the data is transmitted between the source and the target with almost no delay, effectively improve the real-time performance of data integration, and realize comprehensive and diversified data standardization governance to improve data quality, ensure the consistency, accuracy and availability of the data, thereby significantly improving the efficiency and accuracy of data integration.

[0064] In some embodiments, step 130 specifically includes: Step 131, performing data standardization processing on the source data according to the data standardization sub-scheme in the data processing scheme to obtain standardized data; Step 132, performing data quality detection processing on the standardized data according to the data quality control sub-scheme in the data processing scheme; Step 133, when the standardized data passes the data quality detection, the standardized data is converted into a data format according to the data conversion sub-scheme in the data processing scheme to obtain the target data.

[0065] The data processing scheme herein at least includes a data standardization sub-scheme, a data quality control sub-scheme, and a data conversion sub-scheme, wherein each sub-scheme includes processing logic and / or information corresponding to the respective processing operations.

[0066] Optionally, when executing step 130, the data processing scheme may be parsed to obtain a data standardization sub-scheme, a data quality control sub-scheme, and a data conversion sub-scheme. Then, the source data may be firstly subjected to data standardization processing according to the processing logic and / or information in the data standardization sub-scheme in the data processing scheme to obtain standardized data.

[0067] After obtaining the standardized data, the standardized data can be subjected to data quality inspection and processing according to the processing logic and / or information in the data quality control sub-scheme in the data processing scheme to determine whether the source data meets the data quality requirements, thereby generating a corresponding quality control report.

[0068] The data quality detection processing here can be real-time quality control processing or offline quality control processing, which is not specifically limited in this embodiment. The real-time quality control processing is to convert the source data collected in real time into standardized data, and then perform data quality detection processing in real time according to the processing logic and / or information in the data quality control sub-scheme to generate a corresponding quality control report; the offline quality control processing is to convert the collected source data into standardized data, and then perform data quality detection processing according to the processing logic and / or information in the data quality control sub-scheme to generate a corresponding quality control report.

[0069] When it is determined that the standardized data has passed the data quality test based on the quality control report, the standardized data is converted in real time according to the processing logic and / or information in the data conversion sub-scheme in the data processing scheme to obtain target data consistent with the storage format of the central end, and a corresponding conversion report is generated.

[0070] It should be noted that during the data processing process, the data before and after each processing step can be synchronously stored in the edge database to ensure the traceability and integrity of the data.

[0071] The method provided in this embodiment performs real-time processing of source data based on the rich and comprehensive data quality control logic in the data processing scheme, including but not limited to data standardization processing, data quality detection processing and data format conversion processing, so as to ensure that the collected data is transmitted between the source and the target with almost no delay, so as to improve the real-time performance of data integration, improve data quality, ensure the accuracy and consistency of data during the synchronization process, thereby improving the efficiency and accuracy of data integration. In some embodiments, step 131 specifically includes: Based on the abstract syntax tree, extract fields from the source data, and generate standardized processing statements according to the mapping relationship between the extracted fields and the standardized fields in the data standardization sub-scheme; According to the standardization processing statement, the source data is subjected to data standardization processing to obtain the standardized data.

[0072] Alternatively, since non-standard data collection will result in the subsequent business systems (such as the central end or other edge ends) being unable to use the data, it is difficult to implement data storage and sharing, and the docking cost is high. Therefore, it is necessary to standardize the source data so that it meets the standardization requirements of the field to which the source data belongs. The specific implementation steps are as follows: First, use the Abstract Syntax Tree (AST) to extract fields from the SQL statements corresponding to the source data, and automatically generate standardized processing statements based on the mapping relationship between the extracted fields and the standardized fields in the data standardization sub-scheme, so as to automatically convert the source data according to the standardized fields and complete the standardized data mapping.

[0073] After obtaining the standardization processing statement, the source data may be subjected to data standardization processing according to the standardization processing statement through at least one of business view mapping (SQL statement), freemarker (script conversion), groovy (code) or dictionary conversion to obtain standardized data.

[0074] For example, the patient name in the source data is name, and the patient name in the standard data set is patient_name. You can write the SQL statement "select name as patient_name from table" to implement the business view mapping of the source data and obtain standardized data.

[0075] The standardized fields in the data standardization sub-scheme here can be formed by collecting the information data meta-directory specification files in the medical field, such as the health information data meta-directory specification files, by the standardized data management system at the central end, and selecting and determining the corresponding standardized data set metadata from the information data meta-directory specifications in the medical field according to actual business needs, so as to configure the fields according to the standardized data set metadata.

[0076] The method provided in this embodiment realizes the automatic conversion of source data into standardized data based on field extraction, mapping, and generation of standardized processing statements based on the abstract syntax tree, reduces the connection cost caused by non-standard data collection, and improves data integration efficiency and data sharing capabilities.

[0077] In some embodiments, step 132 specifically includes: Analyze and obtain at least one-dimensional quality control information from the data quality control sub-scheme; According to each quality detection condition corresponding to the quality control information of each dimension, the standardized data is subjected to data quality detection to obtain a detection score of the standardized data under each quality detection condition; The detection scores under all quality detection conditions corresponding to the quality control information of each dimension are respectively integrated; According to the fusion result, it is determined whether the standardized data passes the data quality detection.

[0078] Optionally, when executing the data quality detection step of step 132, at least one dimension of quality control information may be first parsed from the data quality control sub-scheme. The quality control information of different dimensions here represents different quality control processing logics, including a plurality of quality detection conditions corresponding to each quality control processing logic. The data quality control sub-scheme here is generated by the quality control system at the central end according to the project requirements and the quality control rule information formulated according to the standard management data set.

[0079] After obtaining the quality control information of each dimension, data quality detection may be performed on the standardized data one by one according to each quality detection condition corresponding to the quality control information of each dimension, so as to obtain the detection score of the standardized data under each quality detection condition.

[0080] Then, the detection scores under all quality detection conditions corresponding to each dimension of quality control information are fused to obtain the detection scores corresponding to each dimension of quality control information. The fusion here can be direct addition or weighted addition, etc., which is not specifically limited in this embodiment.

[0081] After obtaining the detection scores corresponding to the quality control information of each dimension, the detection scores corresponding to the quality control information of each dimension can be compared with the score thresholds corresponding to the quality control information of each dimension to determine whether the standardized data has passed the data quality detection corresponding to the quality control information of each dimension. If the standardized data passes the data quality detection corresponding to all the quality control information of each dimension, it indicates that the standardized data has passed the final data quality detection; if the standardized data fails to pass the data quality detection corresponding to any one of the quality control information of each dimension, it indicates that the standardized data has failed the final data quality detection. At this time, a corresponding quality control report can be generated based on the data quality detection results to prompt the three parties to rectify or adjust the docking data quality.

[0082] It should be noted that the data quality detection corresponding to each dimension of quality control information can be implemented through statements such as SQL statements and Groovy scripts.

[0083] In some embodiments, the at least one-dimensional quality control information includes at least one of accuracy quality control information, integrity quality control information, logic quality control information, timeliness quality control information, and stability quality control information.

[0084] Here, accuracy quality control information is used to verify the accuracy of standardized data, such as testing whether the field format and dictionary of standardized data are correct. The field format includes but is not limited to numbers, letters, and ranges, and the dictionary includes but is not limited to project fields and standardized dictionaries; for example, checking whether the field corresponding to the medical record number meets the specific format and uniqueness requirements, thereby determining whether the field is correct.

[0085] Here, the integrity quality control information is used to verify the integrity of the standardized data, that is, to verify whether the required items in the standardized data are complete. For example, for a patient registration form, it must include a consultation number. If not, it indicates that the form is incomplete.

[0086] Here, logical property control information is used to jointly verify the logical relationship between multiple fields in the standardized data, such as verifying the uniqueness of the primary key and the logic between the master and slave tables.

[0087] Here, timely quality control information is used to verify the timeliness of standardized data, that is, to verify whether the data is uploaded in time, such as whether the difference between the time of business occurrence and the time of upload is within the preset time. For example, the patient registration form data shows that the patient visited the doctor on December 26, and the data was uploaded on December 31, which is one day later, and does not meet the timeliness requirements.

[0088] The stability quality control information here is used to verify the stability of the standardized data, that is, to verify whether the data is uploaded stably, such as whether there is data update upload within the set verification cycle. For example, if the verification cycle is set to one day, if there is a gap in the standardized data within one day, it does not meet the stability requirements.

[0089] The method provided in this embodiment realizes data-rich and comprehensive data quality control logic through data quality detection of multi-dimensional quality control information, thereby effectively improving the data quality and reliability after data integration.

[0090] In some embodiments, step 130 further includes: Determine a target data mapping mode corresponding to the source data according to the type of the data acquisition terminal to which the source data belongs; According to the target data mapping mode, performing a data mapping operation on the source data to obtain data to be processed; According to the data processing scheme, data standardization processing, data quality detection processing and data format conversion processing are performed on the data to be processed to obtain the target data.

[0091] Optionally, since the edge end provided in this embodiment supports various types of data acquisition end docking methods, including but not limited to PostgreSQL, MySQL, Oracle, CacheDB, IRIS, WebService, Hypertext Transfer Protocol (HTTP), MQ consumption acquisition, API interface acquisition (push and pull modes), elasticSearch acquisition. Therefore, in order to achieve high efficiency and effectiveness of data integration, it is possible to first associate and obtain the target data mapping mode corresponding to the source data according to the type of data acquisition end to which the source data belongs. Then, according to the target data mapping mode, a data mapping operation is performed on the source data to convert the source data into a data format that can be recognized and processed within the edge end to ensure that data from different sources can be subsequently processed in a unified or compatible manner. Then, according to the data processing scheme, the data to be processed is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain the target data. The specific processing steps can be referred to steps 131-133, which will not be repeated here.

[0092] The target data mapping mode here includes freemarker mode or jstl mode.

[0093] Exemplarily, when the type of data collection end corresponding to the source data is API interface collection, the data is generally pushed or provided by a third party (ie, the data collection end). The data of the third-party system is quite different from the data of the standard management data set. The freemarker script and interface agent can be executed on the edge to complete the data conversion, and then the converted data can be processed and stored.

[0094] The method provided in this embodiment ensures seamless data exchange between data from heterogeneous data sources by supporting multiple data acquisition terminal docking modes and adopting target data mapping modes such as freemarker or jstl, without the need to modify the existing system, reducing the complexity of technical integration, and ensuring that data from heterogeneous data sources can be integrated and processed in a unified, compatible and efficient manner, thereby achieving high efficiency, effectiveness, high configurability and zero intrusion of data integration.

[0095] In some embodiments, step 120 specifically includes: Determining a target data acquisition terminal corresponding to the data acquisition task in the multi-source data acquisition terminal according to the task type and the acquisition terminal configuration information in the data acquisition scheme; According to the collection frequency and the content of the data to be collected in the data collection scheme, the source data is collected in the target data collection terminal.

[0096] The data collection scheme here at least includes the configuration information of the collection end that can perform data collection, as well as the collection frequency during the data collection process and the content of the data to be collected (ie, the content of the data to be collected).

[0097] Optionally, when executing the data collection step in step 120, the collection terminal configuration information, collection frequency and the content of the data to be collected can be first parsed from the data collection scheme. Then, based on the association mapping relationship between the task type and the data collection terminal, and the association mapping relationship between the collection terminal configuration information and the data collection terminal, the type of processable tasks is linked in the multi-source data collection terminal to be associated and matched with the task type of the data collection task, and the configuration information is associated and matched with the collection terminal configuration information in the data collection scheme. The target data collection terminal here can be one or more, which is specifically determined according to the actual association matching results, and this embodiment does not specifically limit this. The content of the data to be collected here can be the data corresponding to the table to be collected, the field to be collected, etc., which can be flexibly configured according to actual needs to achieve high configurability of data integration.

[0098] After obtaining the target data collection end, a connection can be established between the edge end and the target data collection end, and a cursor can be opened according to the collection frequency in the data collection plan to collect data matching the content of the data to be collected in the target data collection end, thereby obtaining the source data.

[0099] The method provided in this embodiment achieves efficient and accurate data integration and improves data processing efficiency and quality by intelligently matching and collecting source data according to task types and collection plans.

[0100] In some embodiments, step 140 specifically includes: For the current data synchronization, according to the current upload flag, the current data block to be uploaded is loaded into the target data; According to the data synchronization batch size, the current data block to be uploaded is transmitted to the central end in the form of a log; Receiving an identifier of the successfully saved log returned by the central end; According to the identification, determining the next upload flag; The next data synchronization is performed according to the next upload flag, until all data blocks in the target data are synchronized to the central end.

[0101] Figure 4 is a schematic diagram of the data synchronization process provided by the present invention; Figure 4As shown, in the data synchronization process in step 140, data synchronization can be achieved through bidirectional synchronization under a unidirectional network, so that the central end can control all data required for the data collection tasks of each edge end. The specific implementation steps are as follows: For the current data synchronization, when the edge end requests the central end to synchronize the target data, it can load the current data block to be uploaded in the target data according to the current upload flag and limit the number according to the data synchronization batch size; initiate a request to the central end to transmit the current data block to be uploaded to the central end in the form of a log according to the data synchronization batch size, so that after receiving the log, the central end can save the log and update the local log, and when the saving is successful, generate an identifier for the log and return it to the edge end.

[0102] The edge receives the identifier of the successfully saved log returned by the center to determine whether the next data synchronization task has been modified based on the identifier of the returned log. If modified, the current upload flag is modified based on the identifier of the returned log to obtain the next upload flag.

[0103] According to the next upload flag, the next data synchronization is iterated according to the above steps until all data blocks in the target data are synchronized to the center end, and the edge end upload is completed.

[0104] Similarly, for each data synchronization, when the edge end requests the central end to synchronize the download of target data, the edge end can send a request to the central end to request the log currently required to be downloaded, so that the central end can update the download flag according to the request, obtain the current download flag, and load the current data block to be downloaded in the memory database according to the current download flag, and return a response message to the edge end according to the data synchronization batch size limit, so as to transmit the current data block to be downloaded to the edge end in the form of a log according to the data synchronization batch size.

[0105] After receiving the log, the edge saves the log and updates the local log. When the log is saved successfully, the confirmation character (ack) interface of the center is called to upload the identifier of the successfully saved log to the center so that the center can update the next download flag and iterate the next data synchronization step until all the data that the edge needs to download is synchronized.

[0106] As can be seen above, during the entire data synchronization process, the center and the edge each maintain their own flags to avoid the problem of two-way overwriting caused by simultaneous modification of both ends. When two-way modification occurs, the task information set by the edge delivery personnel is the main focus, and the center will merge the edge data to ensure the accuracy and consistency of the data, avoid data errors caused by two-way overwriting, and thus ensure the real-time and effectiveness of data synchronization.

[0107] The method provided in this embodiment ensures the accuracy, consistency and real-time performance of data synchronization through the flag bit maintenance of the center end and the edge end and the merging strategy of the edge end task information priority.

[0108] In some embodiments, the method further comprises: When an alarm is detected in any of the execution processes of the data acquisition task, including the acquisition process, the processing process, and the synchronization process, the alarm is compensated for according to the abnormal information type corresponding to the alarm; When it is determined that the alarm is eliminated, the operation corresponding to any execution process is re-executed on the data collection task.

[0109] It should be noted that existing integration tools, such as Kettle, have limited exception handling capabilities and mainly rely on logs and manual intervention to solve problems. They lack automated error recovery and retry mechanisms, which are not robust enough in a continuously running production environment.

[0110] In order to solve this problem, the method provided in this embodiment automatically performs exception compensation according to the exception compensation strategy corresponding to the exception information type when an exception occurs at any node in the data integration process, so as to ensure the continuity and integrity of data synchronization, thereby effectively improving fault tolerance performance and improving data integration quality.

[0111] Optionally, when an alarm is detected during any execution process of the data collection task, including the collection process, processing process and synchronization process, the alarm may be sent to the user terminal in the form of text message, email, etc. to prompt the user terminal that an abnormality has occurred in the data collection task.

[0112] And according to the type of abnormal information corresponding to the alarm, locate the abnormal source information, that is, the cause of the alarm, such as network failure, quality control system call error, MDM system call error, standardized data management system error, etc., so as to automatically retry according to the retry strategy corresponding to different abnormal source information to compensate for the alarm abnormality. The retry here can be a timed trigger or a real-time trigger when an alarm is detected, etc., and this embodiment does not make specific limitations on this.

[0113] When it is determined that the alarm is eliminated, the operation corresponding to the execution process is re-executed on the data collection task to achieve retry execution of the data collection task.

[0114] The data collection task here executes the operation corresponding to the execution process, which may be manually or automatically started according to the log page, etc. This embodiment does not make any specific limitation to this.

[0115] The method provided in this embodiment ensures the continuity and integrity of data synchronization through automatic exception compensation and retry mechanism, thereby improving data integration quality and fault tolerance performance.

[0116] In some embodiments, step 120 further includes: If a collection interruption occurs during the collection process of any batch of data blocks corresponding to the data collection task, data repair is performed according to the position of the data blocks of any batch; When it is determined that the data repair is successful, the data blocks of any batch and the data blocks after the data blocks of any batch are collected again in the multi-source data collection end according to the interruption position of the data blocks of any batch, the task type and the data collection plan, until all the data blocks corresponding to the data collection task are collected; The source data is generated according to all data blocks corresponding to the data acquisition task.

[0117] It should be noted that during the data synchronization process, existing integration tools such as Kettle do not have a built-in breakpoint resume function. This means that after network instability or system failure, the data synchronization task may need to start from scratch, which is unacceptable in large-scale data processing and will result in data loss or inefficient synchronization.

[0118] In order to solve this problem, the method provided in this embodiment configures the breakpoint resume function, allowing the data synchronization process to be continued from the last successful synchronization point after being interrupted, thereby reducing the risk of data loss. At the same time, the automatic retry and task retransmission functions are added to improve the fault tolerance of the system.

[0119] Optionally, during the data collection process, data collection may be performed according to a preset batch size to ensure stable data transmission. If an error occurs during the collection process of any batch of data blocks corresponding to the data collection task, the collection of the data blocks of the batch will be interrupted. The position of the data blocks of the batch can be recorded here, and the resume function can be started to repair the data based on the position of the data blocks of the batch. Data repair is performed specifically according to the position of the data blocks of the batch, so that when it is determined that the data repair is successful, the data blocks of the batch and the data blocks after the data blocks of the batch are collected again according to the interruption position of any batch of data blocks, the task type and the data collection plan, until the data blocks corresponding to the data collection task are collected. This ensures that when data transmission is interrupted, the current state is automatically saved, and synchronization is restored from the interruption point when conditions permit, effectively preventing data loss in an unstable network environment.

[0120] The method provided in this embodiment effectively reduces the risk of data loss and improves data synchronization efficiency and fault tolerance by configuring breakpoint resume, automatic retry and task retransmission functions.

[0121] The data integration device provided by the present invention is described below. The data integration device described below and the data integration method described above can be referenced to each other.

[0122] Figure 5 is a schematic diagram of the structure of the data integration device provided by the present invention; Figure 5 As shown, the device comprises: The acquisition unit 510 is used to acquire the data collection scheme and data processing scheme configured by the central end; The collection unit 520 is used to collect source data corresponding to the data collection task in the multi-source data collection terminal according to the task type of the data collection task and the data collection scheme; The processing unit 530 is used to perform data standardization processing, data quality detection processing and data format conversion processing on the source data according to the data processing scheme to obtain target data; The synchronization unit 540 is used to synchronize the target data to the central end.

[0123] The device provided in this embodiment adaptively and accurately collects the source data required for data integration in the multi-source data acquisition terminal according to the data acquisition scheme configured at the central end and the task type of the data acquisition task, and uses the comprehensive and rich data quality control logic in the data processing scheme configured at the central end, including but not limited to data standardization processing, data quality detection processing and data format conversion processing to perform real-time processing of the source data to ensure that the data is transmitted between the source and the target with almost no delay, effectively improve the real-time performance of data integration, and realize comprehensive and diversified data standardization governance to improve data quality, ensure the consistency, accuracy and availability of the data, thereby significantly improving the efficiency and accuracy of data integration.

[0124] In some embodiments, the processing unit is specifically configured to: According to the data standardization sub-scheme in the data processing scheme, the source data is subjected to data standardization processing to obtain standardized data; According to the data quality control sub-scheme in the data processing scheme, performing data quality detection processing on the standardized data; In the case where the standardized data passes the data quality detection, the standardized data is converted into a data format according to the data conversion sub-scheme in the data processing scheme to obtain the target data.

[0125] In some embodiments, the processing unit is further configured to: Based on the abstract syntax tree, extract fields from the source data, and generate standardized processing statements according to the mapping relationship between the extracted fields and the standardized fields in the data standardization sub-scheme; According to the standardization processing statement, the source data is subjected to data standardization processing to obtain the standardized data.

[0126] In some embodiments, the processing unit is further configured to: Analyze and obtain at least one-dimensional quality control information from the data quality control sub-scheme; According to each quality detection condition corresponding to the quality control information of each dimension, the standardized data is subjected to data quality detection to obtain a detection score of the standardized data under each quality detection condition; The detection scores under all quality detection conditions corresponding to the quality control information of each dimension are respectively integrated; According to the fusion result, it is determined whether the standardized data passes the data quality detection.

[0127] In some embodiments, the at least one-dimensional quality control information includes at least one of accuracy quality control information, integrity quality control information, logic quality control information, timeliness quality control information, and stability quality control information.

[0128] In some embodiments, the processing unit is further configured to: Determine a target data mapping mode corresponding to the source data according to the type of the data acquisition terminal to which the source data belongs; According to the target data mapping mode, performing a data mapping operation on the source data to obtain data to be processed; According to the data processing scheme, data standardization processing, data quality detection processing and data format conversion processing are performed on the data to be processed to obtain the target data.

[0129] In some embodiments, the acquisition unit is specifically configured to: Determining a target data acquisition terminal corresponding to the data acquisition task in the multi-source data acquisition terminal according to the task type and the acquisition terminal configuration information in the data acquisition scheme; According to the collection frequency and the content of the data to be collected in the data collection scheme, the source data is collected in the target data collection terminal.

[0130] In some embodiments, the synchronization unit is specifically configured to: For the current data synchronization, according to the current upload flag, the current data block to be uploaded is loaded into the target data; According to the data synchronization batch size, the current data block to be uploaded is transmitted to the central end in the form of a log; Receiving an identifier of the successfully saved log returned by the central end; According to the identification, determining the next upload flag; The next data synchronization is performed according to the next upload flag, until all data blocks in the target data are synchronized to the central end.

[0131] In some embodiments, the device further comprises a compensation unit, specifically configured to: When an alarm is detected in any of the execution processes of the data acquisition task, including the acquisition process, the processing process, and the synchronization process, the alarm is compensated for according to the abnormal information type corresponding to the alarm; When it is determined that the alarm is eliminated, the operation corresponding to any execution process is re-executed on the data collection task.

[0132] In some embodiments, the acquisition unit is further used for: If a collection interruption occurs during the collection process of any batch of data blocks corresponding to the data collection task, data repair is performed according to the position of the data blocks of any batch; When it is determined that the data repair is successful, the data blocks of any batch and the data blocks after the data blocks of any batch are collected again in the multi-source data collection end according to the interruption position of the data blocks of any batch, the task type and the data collection plan, until all the data blocks corresponding to the data collection task are collected; The source data is generated according to all data blocks corresponding to the data acquisition task.

[0133] In some embodiments, this embodiment also provides a data integration system. Figure 2 As shown, the system includes a data collection end, an edge end and a center end; The data collection end is connected to the edge end in communication, and the edge end is connected to the central end in communication via a reverse proxy server and an application programming interface gateway; Each of the edge terminals is used to execute the data integration method provided in the above embodiments.

[0134] In the system provided by this embodiment, the edge end adaptively and accurately collects the source data required for data integration in the multi-source data acquisition end according to the data acquisition scheme configured by the central end and the task type of the data acquisition task, and uses the comprehensive and rich data quality control logic in the data processing scheme configured by the central end, including but not limited to data standardization processing, data quality detection processing and data format conversion processing to perform real-time processing of the source data to ensure that the data is transmitted between the source and the target with almost no delay, effectively improve the real-time performance of data integration, and realize comprehensive and diversified data standardization governance to improve data quality, ensure the consistency, accuracy and availability of the data, thereby significantly improving the efficiency and accuracy of data integration.

[0135] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0136] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the data integration method, which includes: obtaining the data acquisition scheme and the data processing scheme configured by the central end; according to the task type of the data acquisition task and the data acquisition scheme, collecting the source data corresponding to the data acquisition task in the multi-source data acquisition end; according to the data processing scheme, performing data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain the target data; synchronizing the target data to the central end.

[0137] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data integration method provided by the above-mentioned methods, which includes: obtaining a data acquisition plan and a data processing plan configured by a central end; according to the task type of the data acquisition task and the data acquisition plan, collecting source data corresponding to the data acquisition task in a multi-source data acquisition end; according to the data processing plan, performing data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain target data; and synchronizing the target data to the central end.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data integration method provided by the above-mentioned methods, the method comprising: obtaining a data acquisition scheme and a data processing scheme configured by a central end; according to the task type of the data acquisition task and the data acquisition scheme, collecting source data corresponding to the data acquisition task in a multi-source data acquisition end; according to the data processing scheme, performing data standardization processing, data quality detection processing and data format conversion processing on the source data to obtain target data; and synchronizing the target data to the central end.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data integration method, characterized in that: include: Obtain the data collection and processing solutions configured on the central end; According to the task type of the data collection task and the data collection scheme, source data corresponding to the data collection task is collected in the multi-source data collection terminal; According to the data processing scheme, the source data is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain target data; The target data is synchronized to the central end.

2. The data integration method according to claim 1, characterized in that: According to the data processing scheme, the source data is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain target data, including: According to the data standardization sub-scheme in the data processing scheme, the source data is subjected to data standardization processing to obtain standardized data; According to the data quality control sub-scheme in the data processing scheme, performing data quality detection processing on the standardized data; In the case where the standardized data passes the data quality detection, the standardized data is converted into a data format according to the data conversion sub-scheme in the data processing scheme to obtain the target data.

3. The data integration method according to claim 2, characterized in that: The step of performing data standardization processing on the source data according to the data standardization sub-scheme in the data processing scheme to obtain standardized data includes: Based on the abstract syntax tree, extract fields from the source data, and generate standardized processing statements according to the mapping relationship between the extracted fields and the standardized fields in the data standardization sub-scheme; According to the standardization processing statement, the source data is subjected to data standardization processing to obtain the standardized data.

4. The data integration method according to claim 2, characterized in that: The step of performing data quality detection processing on the standardized data according to the data quality control sub-scheme in the data processing scheme includes: Analyze and obtain at least one-dimensional quality control information from the data quality control sub-scheme; According to each quality detection condition corresponding to the quality control information of each dimension, the standardized data is subjected to data quality detection to obtain a detection score of the standardized data under each quality detection condition; The detection scores under all quality detection conditions corresponding to the quality control information of each dimension are respectively integrated; According to the fusion result, it is determined whether the standardized data passes the data quality detection.

5. The data integration method according to claim 4, characterized in that: The at least one-dimensional quality control information includes at least one of accuracy quality control information, integrity quality control information, logic quality control information, timeliness quality control information and stability quality control information.

6. The data integration method according to any one of claims 1 to 5, characterized in that: According to the data processing scheme, the source data is subjected to data standardization processing, data quality detection processing and data format conversion processing to obtain target data, including: Determine a target data mapping mode corresponding to the source data according to the type of the data acquisition terminal to which the source data belongs; According to the target data mapping mode, performing a data mapping operation on the source data to obtain data to be processed; According to the data processing scheme, data standardization processing, data quality detection processing and data format conversion processing are performed on the data to be processed to obtain the target data.

7. The data integration method according to any one of claims 1 to 5, characterized in that: The step of collecting source data corresponding to the data collection task in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme includes: Determining a target data acquisition terminal corresponding to the data acquisition task in the multi-source data acquisition terminal according to the task type and the acquisition terminal configuration information in the data acquisition scheme; According to the collection frequency and the content of the data to be collected in the data collection scheme, the source data is collected in the target data collection terminal.

8. The data integration method according to any one of claims 1 to 5, characterized in that: The step of synchronizing the target data to the central end includes: For the current data synchronization, according to the current upload flag, the current data block to be uploaded is loaded into the target data; According to the data synchronization batch size, the current data block to be uploaded is transmitted to the central end in the form of a log; Receiving an identifier of the successfully saved log returned by the central end; Determine the next upload flag according to the identifier; The next data synchronization is performed according to the next upload flag, until all data blocks in the target data are synchronized to the central end.

9. The data integration method according to any one of claims 1 to 5, characterized in that: The method further comprises: When an alarm is detected in any of the execution processes of the data acquisition task, including the acquisition process, the processing process, and the synchronization process, the alarm is compensated for according to the abnormal information type corresponding to the alarm; When it is determined that the alarm is eliminated, the operation corresponding to any execution process is re-executed on the data collection task.

10. The data integration method according to any one of claims 1 to 5, characterized in that: The step of collecting source data corresponding to the data collection task in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme includes: If a collection interruption occurs during the collection process of any batch of data blocks corresponding to the data collection task, data repair is performed according to the position of the data blocks of any batch; When it is determined that the data repair is successful, the data blocks of any batch and the data blocks after the data blocks of any batch are collected again in the multi-source data collection end according to the interruption position of the data blocks of any batch, the task type and the data collection plan, until all the data blocks corresponding to the data collection task are collected; The source data is generated according to all data blocks corresponding to the data acquisition task.

11. A data integration device, characterized in that: include: An acquisition unit, used to acquire a data collection scheme and a data processing scheme configured by the central end; A collection unit, configured to collect source data corresponding to the data collection task in a multi-source data collection terminal according to the task type of the data collection task and the data collection scheme; A processing unit, configured to perform data standardization processing, data quality detection processing and data format conversion processing on the source data according to the data processing scheme to obtain target data; A synchronization unit is used to synchronize the target data to the central end.

12. A data integration system, characterized in that: Including data collection end, edge end and center end; The data collection end is connected to the edge end in communication, and the edge end is connected to the central end in communication via a reverse proxy server and an application programming interface gateway; Each of the edge terminals is used to execute the data integration method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data integration method according to any one of claims 1 to 10 is implemented.

Images