A medical data etl task synchronization method and device

Abstract

The present invention provides a medical data ETL task synchronization method and device, wherein the method includes: based on the association relationship between the data source to be synchronized and the target data table in the target database, determine the target data to be written corresponding to the data source to be synchronized table; based on the data association between the target data tables to be written, determine the intermediate file corresponding to the target data table to be written; wherein, the target data table to be written with data association corresponds to the same intermediate file; based on the target to be written Data table, obtain the data to be synchronized whose attribute name in the data source to be synchronized matches the field name to be written into the target data table; write the data to be synchronized into the intermediate file corresponding to the target data table to be written into; write the intermediate The data in the file is written in batches to the corresponding target data table to be written. The invention reduces the storage space occupancy rate of the target database and the load of database writing, improves the efficiency of data synchronization, and avoids importing too much useless data at the same time.

Description

technical field [0001] The invention relates to the technical field of databases, in particular to a medical data ETL task synchronization method and device. Background technique [0002] With the steady development of medical information construction, the medical data center can provide high-quality data services for clinical, scientific research, drug research and development and other fields. Among them, the medical data center is inseparable from the construction of the ETL (Extract-Transform-Load) data processing system. When building an ETL data processing system, due to the large number of heterogeneous data sources, how to efficiently obtain data from a large number of heterogeneous data sources and write them into the data center is the core issue. [0003] In the medical field, the update frequency of various heterogeneous data sources (such as medical record data, registration data, etc.) is relatively fast, and the amount of individual data is small, so a large ...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a medical data ETL task synchronization method and device to solve the defects of low micro-batch data writing efficiency and large storage space occupation in the prior art

Images