Data synchronization method and device and server
A data synchronization and data technology, applied in the field of data processing, can solve the problems of poor data synchronization flexibility and adaptability, and manual intervention, etc., to achieve the effect of improving flexibility and adaptability and reducing manual intervention
Pending Publication Date: 2021-06-08
BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD +1
View PDF0 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In related technologies, components such as Sqoop, Flume, or Hive can be used to complete the above data synchronization process; specifically, according to the requirements of the business system, engineers can manually write Sqoop, Flume, or HQL statements to achieve data synchronization. If the business system changes, engineers are required to Modify the relevant statements used for data synchronization, so this method requires more manual intervention, resulting in poor flexibility and adaptability of data synchronization
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0027] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0028] At present, in the construction of a data warehouse, the operation of data synchronization mainly depends on the three major components of Sqoop, Flume or Hive. According to the requirements of the business system, engineers manually write Sqoop, Flume or HQL statements to realize the data synchronization process of the business system database.
[0029] When using Sqoop or Flume components for data synchronization, during the data synchronization process, there are intermediate results saved from memory to disk, a...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a data synchronization method and device and a server. The method comprises the following steps: loading pre-configured configuration information; wherein the configuration information comprises a data source and metadata of data needing to be synchronized in the data source; obtaining and caching target data corresponding to the metadata from the data source; and synchronizing the target data to a data table matched with the metadata in a preset data warehouse. Configuration information such as a data source and metadata of data needing to be synchronized in the data source is written into configuration information, then data synchronization is carried out based on the configuration information, and when the data source or the metadata needs to be updated, only the corresponding configuration information needs to be updated; an engineer does not need to manually modify related code statements for data synchronization; manual intervention is reduced in the mode; therefore, the flexibility and adaptability of data synchronization are improved.
Description
technical field [0001] The present invention relates to the technical field of data processing, in particular to a data synchronization method, device and server. Background technique [0002] In data warehouse construction, data synchronization is an important step. In the process of data synchronization, it is necessary to collect and summarize multi-source heterogeneous data from business systems, online logs, etc., and then store them in the partition table of the ODS layer of the data warehouse; the data stored in the data warehouse is used to support the daily operation of the enterprise. The data requirements of the global application. In related technologies, components such as Sqoop, Flume, or Hive can be used to complete the above data synchronization process; specifically, according to the requirements of the business system, engineers can manually write Sqoop, Flume, or HQL statements to achieve data synchronization. If the business system changes, engineers are...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/27
CPCG06F16/275G06F16/254
Inventor 王森
Owner BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD