Full-amount data synchronization method and device, equipment and medium

A technology of full data and incremental data, applied in the field of devices, equipment and media, and full data synchronization methods, can solve problems affecting data table reading and writing, and achieve the effect of improving efficiency and frequency

Pending Publication Date: 2021-07-09
GUANGZHOU HUYA TECH CO LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a full data synchronization method, device, equipment and medium to solve the problem in the prior art that the data in the database is fully synchronized to the data warehouse and affect the reading and writing of the data table, and at the same time improve the efficiency of full data synchronization

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Full-amount data synchronization method and device, equipment and medium
  • Full-amount data synchronization method and device, equipment and medium
  • Full-amount data synchronization method and device, equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] figure 1 It is a flow chart of a full data synchronization method provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation where the data in the database is fully synchronized to the data warehouse, for example, the data in the MySQL database can be synchronized to the HIVE data warehouse. In the case of full-volume synchronization, the method can be executed by a full-volume data synchronization device provided in any embodiment of the present invention, which can be composed of hardware and / or software, and can generally be integrated into computer equipment.

[0027] Such as figure 1 As shown, the full data synchronization method provided in this embodiment includes the following steps:

[0028] S110. Obtain the incremental log data of the source database imported into the data warehouse within the current synchronization time period.

[0029] The source database refers to a database that requires full data synchronization, ...

Embodiment 2

[0064] figure 2 It is a flow chart of a full data synchronization method provided in Embodiment 2 of the present invention. This embodiment is embodied on the basis of the above embodiments, wherein, the incremental data row record is determined according to the incremental log data, which can be specifically for:

[0065] For the first type of log data whose operation type is insertion or deletion in the incremental log data, the data row record corresponding to the first type of log data is used as an incremental data row record;

[0066] For the second type of log data in the incremental log data whose operation type is updated and whose primary key value has not been changed, the updated data row record corresponding to the second type of log data is used as the incremental data row record;

[0067] For the third type of log data in the incremental log data whose operation type is updating and changing the primary key value, the operation type corresponding to the third ...

Embodiment 3

[0124] image 3 It is a schematic diagram of the module structure of a full data synchronization device provided by Embodiment 3 of the present invention. This embodiment is applicable to the situation where the data in the database is fully synchronized to the data warehouse, for example, the data in the MySQL database can be sent to the HIVE data warehouse In the case of full synchronization, the device can be realized by software and / or hardware, and can generally be integrated into computer equipment.

[0125] Such as image 3 As shown, the full data synchronization device provided in this embodiment specifically includes: a log acquisition module 310 , a data merging module 320 and a data filtering and storage module 330 . in,

[0126] The log acquisition module 310 is used to acquire the incremental log data of the source database imported into the data warehouse within the current synchronization time period;

[0127] The data merging module 320 is configured to gene...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a full-amount data synchronization method and device, equipment and a medium. The method comprises the following steps: acquiring incremental log data of a source database imported into a data warehouse in a current synchronization time period; in the data warehouse, generating undetermined total data corresponding to the current synchronization time period according to the incremental log data and total data corresponding to an adjacent previous synchronization time period; and screening the undetermined total data, and determining and storing target total data corresponding to the current synchronization time period. According to the technical scheme, the synchronization of the total data of the source database is completed in the data warehouse based on the log data of the source database, the read-write operation of the data table in the source database is not influenced, the data total synchronization efficiency is improved, and the total data synchronization frequency can also be improved.

Description

technical field [0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a method, device, device, and medium for synchronizing full data. Background technique [0002] With the promotion and popularization of the Internet and big data technology, data warehouses have gradually become mainstream. Since the data warehouse has a large storage capacity and can handle structured data, the data in the database can be synchronized to the data warehouse for processing. [0003] In the application scenario of synchronizing the data in the database to the data warehouse, it is usually to synchronize all the data in the data table to the data warehouse. If the data volume of the data table is large, the time-consuming of full data synchronization will be very long, and it will also affect the reading and writing of the data table, and the cycle of full data synchronization should not be too frequent. Contents of the invention ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/27G06F16/23G06F16/28
CPCG06F16/27G06F16/2358G06F16/283
Inventor 陶胜
Owner GUANGZHOU HUYA TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap