Medical data ETL task synchronization method and device

Abstract

The invention provides a medical data ETL task synchronization method and device, and the method comprises the steps: determining a to-be-written target data table corresponding to a to-be-synchronized data source based on an incidence relation between the to-be-synchronized data source and a target data table in a target database; determining an intermediate file corresponding to the to-be-written target data table based on the data relevance between the to-be-written target data table, wherein the to-be-written target data tables with data association correspond to the same intermediate file; based on the to-be-written target data table, obtaining to-be-synchronized data of which the attribute name is matched with the field name of the to-be-written target data table in the to-be-synchronized data source; writing the to-be-synchronized data into an intermediate file corresponding to the to-be-written target data table; and writing the data in the intermediate file into the corresponding to-be-written target data table in batches. According to the method, the storage space occupancy rate of the target database and the write-in load of the database are reduced, the data synchronization efficiency is improved, and meanwhile, excessive useless data is prevented from being imported.

Description

technical field [0001] The invention relates to the technical field of databases, in particular to a medical data ETL task synchronization method and device. Background technique [0002] With the steady development of medical information construction, the medical data center can provide high-quality data services for clinical, scientific research, drug research and development and other fields. Among them, the medical data center is inseparable from the construction of the ETL (Extract-Transform-Load) data processing system. When building an ETL data processing system, due to the large number of heterogeneous data sources, how to efficiently obtain data from a large number of heterogeneous data sources and write them into the data center is the core issue. [0003] In the medical field, the update frequency of various heterogeneous data sources (such as medical record data, registration data, etc.) is relatively fast, and the amount of individual data is small, so a large ...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a medical data ETL task synchronization method and device to solve the defects of low micro-batch data writing efficiency and large storage space occupation in the prior art

Images