Data synchronization method, data synchronization device and computer readable storage medium

By obtaining the configuration table between the data warehouse and the in-memory database, directly converting and writing data formats, the problem of untimely data synchronization between the data warehouse and the in-memory database is solved, efficient and real-time data synchronization is achieved, and the system is reduced risk.

CN119066124APending Publication Date: 2024-12-03中国邮政储蓄银行股份有限公司
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411094300.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, data synchronization between data warehouses and in-memory databases is not timely, and high concurrency and real-time data updates cannot be supported. The existing methods rely on transit equipment or file landing, resulting in waste of resources and system risks.

Method used

By obtaining the configuration table between the data warehouse and the in-memory database, obtaining the key-value pairs corresponding to the data, and performing format conversion processing, the data is directly written to the in-memory database, avoiding the landing of the relay device and files, and realizing data synchronization.

Benefits of technology

It realizes timely data synchronization between the data warehouse and the in-memory database, improves data synchronization efficiency, reduces system risks, and supports high concurrency and real-time data updates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066124A_ABST
    Figure CN119066124A_ABST
Patent Text Reader

Abstract

The invention provides a data synchronization method, a data synchronization device and a computer readable storage medium, the method is applied to a data synchronization system, the data synchronization system comprises a data warehouse and a memory database, and the method comprises the following steps: obtaining a configuration table, the configuration table is at least used for representing a corresponding relation between data in the data warehouse and data in the memory database; according to the configuration table, key-value pairs corresponding to data in the data warehouse are obtained, original data are obtained, and the key-value pairs comprise keys and key values corresponding to the keys; format conversion processing is carried out on the original data to obtain to-be-synchronized data, and the data format of the to-be-synchronized data is the same as the data format of the memory database; and writing the to-be-synchronized data into the memory database. The problem that data synchronization between the data warehouse and the memory database is not timely is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular, to a data synchronization method, a data synchronization device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In-memory databases can be used to process large amounts of data and high concurrency due to their high-speed access and high concurrency. Data warehouses have a wide range of applications in data association aggregation and data analysis due to their powerful storage and distributed computing capabilities. However, the slow access speed of data warehouses and their low high-concurrency performance have hampered their development in high-speed online businesses.

[0003] Therefore, there is an urgent need for a method to synchronize the data in the data warehouse to the in-memory database in a timely manner. Summary of the invention

[0004] The main purpose of the present application is to provide a data synchronization method, a data synchronization device, a computer-readable storage medium and a computer program product, so as to at least solve the problem of untimely data synchronization between a data warehouse and an in-memory database in the prior art.

[0005] According to one aspect of the present application, a data synchronization method is provided, and the method is applied to a data synchronization system, wherein the data synchronization system includes a data warehouse and an in-memory database, and comprises: obtaining a configuration table, wherein the configuration table is at least used to characterize the correspondence between the data in the data warehouse and the data in the in-memory database; according to the configuration table, obtaining a key-value pair corresponding to the data in the data warehouse to obtain original data, wherein the key-value pair includes the key and the key value corresponding to the key; performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the in-memory database; and writing the data to be synchronized into the in-memory database.

[0006] Optionally, the in-memory database includes multiple clusters, any one of the clusters includes multiple nodes, and the data synchronization system also includes a relational database. Obtaining the configuration table includes: obtaining the unique identity information of each cluster, the key of each cluster, the network protocol address of each node, the data operation type of the in-memory database, the unique identity information of the data warehouse, the data operation type of the data to be synchronized, and the expiration time of the data to be synchronized stored in the relational database to obtain the configuration table.

[0007] Optionally, the configuration table includes at least an SQL statement, and the SQL statement is used to process the data in the data warehouse. According to the configuration table, key-value pairs corresponding to the data in the data warehouse are obtained to obtain the original data, including: sending a request message to the data warehouse, wherein the request message is used to request synchronization of the data in the data warehouse; modifying a predetermined field in the SQL statement to obtain a select statement, wherein the predetermined field includes a key and a key value, and the key is used to uniquely identify the key value; using the select statement to obtain the key-value pairs corresponding to the data in the data warehouse to obtain the original data.

[0008] Optionally, performing format conversion processing on the original data to obtain the data to be synchronized includes: converting the data format of the key value corresponding to the original data into the data format of the in-memory database to obtain the data to be synchronized.

[0009] Optionally, the memory database includes multiple clusters, any one of the clusters includes multiple storage nodes, and writing the data to be synchronized into the memory database includes: obtaining the slot interval range corresponding to each of the storage nodes to obtain multiple target ranges, wherein the storage nodes correspond one-to-one to the slot interval ranges; determining a synchronization address based on the data to be synchronized and the target range; and synchronizing the data to be synchronized to the synchronization address of the memory database.

[0010] Optionally, determining the synchronization address based on the data to be synchronized and the target range includes: obtaining the key corresponding to the data to be synchronized, and determining whether the key is within the target range; when the key is within the target range, determining that the address of the storage node corresponding to the target range is the synchronization address.

[0011] Optionally, synchronizing the data to be synchronized to the synchronization address of the memory database includes: grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; and batch writing the multiple data groups into different storage partitions of the same synchronization address.

[0012] According to another aspect of the present application, a data synchronization device is provided, which is applied to a data synchronization system, wherein the data synchronization system includes a data warehouse and an in-memory database, and includes: a first acquisition unit, used to acquire a configuration table, wherein the configuration table is at least used to characterize the correspondence between the data in the data warehouse and the data in the in-memory database; a second acquisition unit, used to acquire a key-value pair corresponding to the data in the data warehouse according to the configuration table to obtain original data, wherein the key-value pair includes the key and the key value corresponding to the key; a processing unit, used to perform format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the in-memory database; and a writing unit, used to write the data to be synchronized into the in-memory database.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0014] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the methods described are implemented.

[0015] Applying the technical solution of the present application, a data synchronization method is provided. The method is applied to a data synchronization system. The data synchronization system includes a data warehouse and an in-memory database. First, a configuration table is obtained. The configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the in-memory database. Then, according to the configuration table, the key-value pairs corresponding to the data in the data warehouse are obtained to obtain the original data. The key-value pairs include a key and a key value corresponding to the key. Then, the original data is format-converted to obtain the data to be synchronized. The data format of the data to be synchronized is the same as that of the in-memory database. Finally, the data to be synchronized is written into the in-memory database. By obtaining the corresponding relationship between the data of the pre-configured data warehouse and the in-memory database, that is, the configuration table, and then determining the data to be synchronized according to the configuration, data synchronization does not need to be performed through a transfer device or file landing. The data update in the data warehouse can be synchronized to the in-memory database of the production system in a timely manner. The problem of untimely data synchronization between the data warehouse and the in-memory database is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1A hardware structure block diagram of a mobile terminal for executing a data synchronization method provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a data synchronization method according to an embodiment of the present application is shown;

[0019] Figure 3 A specific flow chart of a data synchronization method provided according to an embodiment of the present application is shown;

[0020] Figure 4 A structural block diagram of a data synchronization device provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0027] Big Data Data Warehouse: Big Data Data Warehouse is a Hive data warehouse built on Hadoop. It processes massive data through HiveQL language. It stores data in Hadoop's HDFS. The data features include structured data, semi-structured data, and unstructured data. Its characteristics are storing massive data and batch processing of large-scale data. The data warehouse referred to in this application can be a HIVE data warehouse.

[0028] In-memory database: An in-memory database is a database that stores data in memory rather than on disk. The mainstream in-memory database currently used in the industry is the Redis in-memory database. In the Redis in-memory database, data exists in the form of key-value pairs. In this application, the in-memory database may be a Redis in-memory database.

[0029] At present, there are basically two methods for synchronizing data to the memory database in the prior art. First, the data in the data warehouse is extracted into a text file, and then the text file content is read by the application to synchronize the data according to the memory database format. In addition, the method of extracting the data in the data warehouse into a text file and then operating it cannot fully utilize the advantages of the distributed operation of the data warehouse, and the distributed operation becomes a single-machine operation. While wasting resources, it is also very limited by the performance of the single machine.

[0030] 1. For single-node or sentinel Redis memory databases, the Pipeline method is used to merge a large amount of data into a batch operation to update or write to the memory database, avoiding a write request for each piece of data, thereby saving network IO overhead and improving write efficiency. However, single-node or sentinel memory databases are no longer the mainstream form of use because they cannot be expanded online and do not have the disadvantages of read-write separation.

[0031] 2. For clustered Redis memory databases, RedisTemplate's multiSet method is usually used for writing. However, this method requires that the data be in the same slot of the Redis cluster, otherwise batch writing cannot be achieved. In addition, this method cannot set the data expiration time, and is not suitable for scenarios where data automatic expiration needs to be set, which can easily cause garbage data backlog.

[0032] Therefore, in the process of synchronizing the data warehouse to the memory database in the prior art, there are the following problems:

[0033] 1) The existing technology cannot support configurable data synchronization in data synchronization, and cannot dynamically configure the key-value pair of the memory database Redis and the data source. Changes in data source data or modifications to keys require code changes. There is repeated development and poor reusability.

[0034] 2) The existing technology uses a memory database in sentinel mode for data synchronization. However, the high availability and concurrency of the sentinel mode are not as good as those of the cluster mode, and it cannot support online expansion and read-write separation, so it is an obsolete mode.

[0035] 3) The existing batch writing cluster mode cannot support invalid data cleaning, resulting in a large amount of invalid data, making the system prone to risks in positioning and locating historical invalid data.

[0036] 4) Existing technologies cannot synchronize big data warehouses with in-memory databases in real time. They all need to be executed through transfer equipment or file landing. The above two transfer methods have large transfer machine resource overhead and huge system risks. They cannot be used during business peak periods. After data is updated, it cannot be synchronized to the production system in time, and the latest data cannot be displayed.

[0037] As introduced in the background technology, in the prior art, data synchronization between the data warehouse and the memory database is not timely. To solve the above problem, the embodiments of the present application provide a data synchronization method, a data synchronization device, a computer-readable storage medium and a computer program product.

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a data synchronization method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1Different configurations are shown.

[0040] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data synchronization method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] In this embodiment, a data synchronization method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Figure 2 is a flow chart of a data synchronization method according to an embodiment of the present application. Figure 2 As shown, the above method is applied to a data synchronization system, the above data synchronization system includes a data warehouse and an in-memory database, and the method includes the following steps:

[0043] Step S201, obtaining a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0044] Specifically, the above configuration table can be one table or multiple tables. For example, one is a Redis cluster connection information table, including cluster ID, cluster node IP, cluster password and other information; the other is a write data job configuration table, including SQL statements for extracting data from the data warehouse, data warehouse name, target Redis cluster ID, data operation type, data TTL expiration time, etc.

[0045] Step S202, according to the configuration table, obtaining the key-value pair corresponding to the data in the data warehouse to obtain the original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0046] Specifically, first, submit a Spark job to the Hadoop cluster, which reads the Redis cluster connection information table and the data writing job configuration table in the configuration module and extracts data from the Hive warehouse. After the data extraction operation is completed and the original data is obtained, the filtered data in the Hive data warehouse is temporarily stored in the memory of the distributed cluster of the Spark job.

[0047] Step S203, performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0048] Specifically, the data format conversion process can convert multiple data formats supported by the Hive data warehouse into a data format supported by the Redis memory database to be written.

[0049] Step S204, writing the data to be synchronized into the memory database.

[0050] Specifically, writing data to an in-memory database can be achieved in several steps: First, write the data writing logic. For key-value storage, you may need to define keys and key values. For more complex data structures, you need to design the data writing logic based on the data types supported by the in-memory database. Then use the API or commands provided by the in-memory database to perform data writing operations, such as SET, LPUSH, SADD, HSET and other commands, depending on the in-memory database and data type you are using. When writing data, take into account possible errors and write corresponding error handling logic, such as retry mechanisms, error logging, etc. After the data is written, verification is performed to ensure that the data is stored correctly in the in-memory database.

[0051] Through this embodiment, a data synchronization method is provided, which is applied to a data synchronization system, and the data synchronization system includes a data warehouse and a memory database. First, a configuration table is obtained, and the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database; then, according to the configuration table, the key-value pair corresponding to the data in the data warehouse is obtained to obtain the original data, and the key-value pair includes a key and a key value corresponding to the key; then, the original data is format-converted to obtain the data to be synchronized, and the data format of the data to be synchronized is the same as the data format of the memory database; finally, the data to be synchronized is written into the memory database. By obtaining the corresponding relationship between the data of the pre-configured data warehouse and the memory database, that is, the configuration table, and then determining the data to be synchronized according to the configuration, data synchronization does not need to be performed through a transfer device or file landing. The data update in the data warehouse can be synchronized to the memory database of the production system in a timely manner. The problem of untimely data synchronization between the data warehouse and the memory database is solved.

[0052] In the specific implementation process, the above-mentioned memory database includes multiple clusters, any of the above-mentioned clusters includes multiple nodes, and the above-mentioned data synchronization system also includes a relational database. The above-mentioned step S201 can be implemented by the following steps: Step S2011, obtain the unique identity information of each of the above-mentioned clusters stored in the above-mentioned relational database, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized, and obtain the above-mentioned configuration table. This method can further quickly establish the above-mentioned configuration table.

[0053] Specifically, the Redis cluster configuration and the data extraction job configuration are stored in a relational database. Those skilled in the art only need to be familiar with the SQL language to extract the specified data information from the HIVE warehouse and write it into Redis, without the need for development experience related to Redis interaction.

[0054] In order to further improve the adaptability and extensiveness of the data synchronization method, the above configuration table at least includes SQL statements, and the above SQL statements are used to process the data in the above data warehouse. The above step S202 of the present application can be implemented by the following steps: Step S2021, sending a request message to the above data warehouse, wherein the above request message is used to request synchronization of the data in the above data warehouse; Step S2022, modifying the predetermined field in the above SQL statement to obtain a select statement, wherein the above predetermined field includes a key and a key value, and the above key is used to uniquely identify the above key value; Step S2023, using the above select statement to obtain the key-value pair corresponding to the data in the above data warehouse to obtain the original data. The SQL statement for data extraction supports variable replacement, adapts to the dynamic changes of different job parameters, and can extract incremental or full data in a simple way, effectively improving the adaptability and extensive application.

[0055] Specifically, data is extracted from the Hive warehouse according to the configured SQL statement, and some temporary variables in the SQL statement are replaced during execution. The two fields selected in the SQL statement are the target key and the key value.

[0056] The above step S203 can be implemented in other ways, for example: step S2031, converting the data format of the above key value corresponding to the above original data into the data format of the above memory database to obtain the above data to be synchronized. Consistent data format can reduce the time of data conversion, thereby speeding up the speed of data synchronization and further improving the overall data synchronization efficiency.

[0057] Specifically, when synchronizing data information from a big data warehouse to an in-memory database, the data is formatted into the memory structure of Dataframe and Dataset in memory, and the corresponding Redis data structure is identified in the structure to implement data format conversion within the memory, converting the data structure of the Hive data warehouse into the data structure of Redis.

[0058] In some embodiments, the above-mentioned memory database includes multiple clusters, and any of the above-mentioned clusters includes multiple storage nodes. The above-mentioned step S204 can be specifically implemented by the following steps: step S2041, obtaining the slot interval range corresponding to each of the above-mentioned storage nodes, and obtaining multiple target ranges, wherein the above-mentioned storage nodes correspond to the above-mentioned slot interval ranges one by one; step S2042, determining the synchronization address according to the above-mentioned data to be synchronized and the above-mentioned target range; step S2043, synchronizing the above-mentioned data to be synchronized to the above-mentioned synchronization address of the above-mentioned memory database. This method can further improve the shortcomings of the inability to automatically update historical data and abnormal information positioning during data synchronization.

[0059] Specifically, the synchronization address includes the connection information of the memory database, such as host name, port number, database instance, etc. Use the appropriate client library or API to establish a connection to the memory database, and use the written synchronization logic to write the data to the memory database. The above data synchronization can be a batch operation or a single data item. After the synchronization is completed, verify whether the data is correctly written to the memory database. Record key information during the synchronization process, such as the start time, end time, amount of synchronized data, problems encountered, etc.

[0060] The above step S2042 can also be implemented by the following steps: Step S20421, obtaining the key corresponding to the above data to be synchronized, and determining whether the above key is within the above target range; Step S20422, when the above key is within the above target range, determining the address of the above storage node corresponding to the above target range as the above synchronization address. This method can further solve the problem that synchronization from a large data warehouse to an in-memory database cannot be executed during business use, and data results are delayed.

[0061] Specifically, connect to any node in the cluster to query other nodes in the cluster and the range of slots allocated on each node, calculate in memory which slot each key belongs to, and then obtain which node in the Redis cluster should be written.

[0062] In some specific embodiments, the above step S2043 can be implemented by the following steps: step S20431, grouping the above data to be synchronized corresponding to the same storage node to obtain multiple data groups; step S20432, batch writing the multiple data groups into different storage partitions of the same synchronization address. This method can further solve the problem of low efficiency of relying on the relay machine during data synchronization.

[0063] Specifically, an elastic partitioner can be used to repartition the data, redistribute the data keys distributed on the same node in the Redis cluster to one or more new partitions, and ensure that all keys under each data partition have the same target Redis node. Then start multiple computing processes to process each partition separately. In the process of processing a single partition, first read the first piece of data in the partition, obtain the corresponding Redis cluster node and establish a direct connection with the node. Since it has been ensured that the data in the same partition all have the same target Redis node, the pipeline method can be used to write all the data in the partition to the node in batches, and the corresponding data expiration time TTL can be set during the writing process. At this point, the data synchronization process from the entire configured data warehouse to the in-memory database is completed. The specific flow chart of the above data synchronization method is shown in the following figure. Figure 3 shown.

[0064] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the data synchronization method of the present application will be described in detail below in combination with specific embodiments.

[0065] The embodiment of the present application also provides a data synchronization device. It should be noted that the data synchronization device of the embodiment of the present application can be used to execute the data synchronization method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0066] The following is an introduction to the data synchronization device provided in the embodiment of the present application.

[0067] Figure 4 is a schematic diagram of a data synchronization device according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0068] A first acquisition unit 10 is used to acquire a configuration table, wherein the configuration table is at least used to represent the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0069] Specifically, the above configuration table can be one table or multiple tables. For example, one is a Redis cluster connection information table, including cluster ID, cluster node IP, cluster password and other information; the other is a write data job configuration table, including SQL statements for extracting data from the data warehouse, data warehouse name, target Redis cluster ID, data operation type, data TTL expiration time, etc.

[0070] A second acquisition unit 20 is used to acquire a key-value pair corresponding to the data in the data warehouse according to the configuration table to obtain original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0071] Specifically, first, submit a Spark job to the Hadoop cluster, which reads the Redis cluster connection information table and the data writing job configuration table in the configuration module and extracts data from the Hive warehouse. After the data extraction operation is completed and the original data is obtained, the filtered data in the Hive data warehouse is temporarily stored in the memory of the distributed cluster of the Spark job.

[0072] The processing unit 30 is used to perform format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0073] Specifically, the data format conversion process can convert multiple data formats supported by the Hive data warehouse into a data format supported by the Redis memory database to be written.

[0074] The writing unit 40 is used to write the above-mentioned data to be synchronized into the above-mentioned memory database.

[0075] Specifically, writing data to an in-memory database can be achieved in several steps: First, write the data writing logic. For key-value storage, you may need to define keys and key values. For more complex data structures, you need to design the data writing logic based on the data types supported by the in-memory database. Then use the API or commands provided by the in-memory database to perform data writing operations, such as SET, LPUSH, SADD, HSET and other commands, depending on the in-memory database and data type you are using. When writing data, take into account possible errors and write corresponding error handling logic, such as retry mechanisms, error logging, etc. After the data is written, verification is performed to ensure that the data is stored correctly in the in-memory database.

[0076] Through this embodiment, a data synchronization device is provided, which is applied to a data synchronization system. The data synchronization system includes a data warehouse and a memory database. The first acquisition unit acquires a configuration table, which is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database; the second acquisition unit acquires the key-value pair corresponding to the data in the data warehouse according to the configuration table to obtain the original data, and the key-value pair includes a key and a key value corresponding to the key; the processing unit performs format conversion processing on the original data to obtain the data to be synchronized, and the data format of the data to be synchronized is the same as the data format of the memory database; the writing unit writes the data to be synchronized into the memory database. By acquiring the corresponding relationship between the data of the pre-configured data warehouse and the memory database, that is, the configuration table, and then determining the data to be synchronized according to the configuration, it is not necessary to perform data synchronization through a transfer device or file landing. The data update in the data warehouse can be synchronized to the memory database of the production system in a timely manner. The problem of untimely data synchronization between the data warehouse and the memory database is solved.

[0077] In the specific implementation process, the above-mentioned memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple nodes, the above-mentioned data synchronization system also includes a relational database, and the above-mentioned first acquisition unit includes a first acquisition module, which is used to obtain the unique identity information of each of the above-mentioned clusters stored in the above-mentioned relational database, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized, so as to obtain the above-mentioned configuration table. The device can further quickly establish the above-mentioned configuration table.

[0078] Specifically, the Redis cluster configuration and the data extraction job configuration are stored in a relational database. Those skilled in the art only need to be familiar with the SQL language to extract the specified data information from the HIVE warehouse and write it into Redis, without the need for development experience related to Redis interaction.

[0079] In order to further improve the adaptability and extensiveness of the data synchronization method, the above-mentioned configuration table at least includes SQL statements, and the above-mentioned SQL statements are used to process the data in the above-mentioned data warehouse. The above-mentioned second acquisition unit of the present application includes a sending module, a modification module and a second acquisition module, wherein the sending module is used to send request information to the above-mentioned data warehouse, wherein the above-mentioned request information is used to request synchronization of the data in the above-mentioned data warehouse; the modification module is used to modify the predetermined fields in the above-mentioned SQL statements to obtain a select statement, and the above-mentioned predetermined fields include a key and a key value, and the above-mentioned key is used to uniquely identify the above-mentioned key value; the second acquisition module is used to use the above-mentioned select statement to obtain the key-value pair corresponding to the data in the above-mentioned data warehouse to obtain the original data. The SQL statement for data extraction supports variable replacement, adapts to the dynamic changes of different job parameters, and can extract incremental or full data in a simple way, which effectively improves the adaptability and extensive application.

[0080] Specifically, data is extracted from the Hive warehouse according to the configured SQL statement, and some temporary variables in the SQL statement are replaced during execution. The two fields selected in the SQL statement are the target key and the key value.

[0081] The processing unit includes a conversion module, which is used in step S2031 to convert the data format of the key value corresponding to the original data into the data format of the memory database to obtain the data to be synchronized. The consistency of data format can reduce the time of data conversion, thereby speeding up the speed of data synchronization and further improving the overall data synchronization efficiency.

[0082] Specifically, when synchronizing data information from a big data warehouse to an in-memory database, the data is formatted into the memory structure of Dataframe and Dataset in memory, and the corresponding Redis data structure is identified in the structure to implement data format conversion within the memory, converting the data structure of the Hive data warehouse into the data structure of Redis.

[0083] In some embodiments, the memory database includes multiple clusters, any one of the clusters includes multiple storage nodes, and the write unit includes a third acquisition module, a determination module, and a synchronization module, wherein the third acquisition module is used to obtain the slot interval range corresponding to each of the storage nodes to obtain multiple target ranges, wherein the storage nodes correspond to the slot interval ranges one by one; the determination module is used to determine the synchronization address according to the data to be synchronized and the target range; the synchronization module is used to synchronize the data to be synchronized to the synchronization address of the memory database. This device can further improve the shortcomings of the inability to automatically update historical data and abnormal information positioning during data synchronization.

[0084] Specifically, the synchronization address includes the connection information of the memory database, such as host name, port number, database instance, etc. Use the appropriate client library or API to establish a connection to the memory database, and use the written synchronization logic to write the data to the memory database. The above data synchronization can be a batch operation or a single data item. After the synchronization is completed, verify whether the data is correctly written to the memory database. Record key information during the synchronization process, such as the start time, end time, amount of synchronized data, problems encountered, etc.

[0085] The determination module includes a first determination submodule and a second determination submodule, wherein the first determination submodule is used to obtain the key corresponding to the data to be synchronized and determine whether the key is within the target range; the second determination submodule is used to determine that the address of the storage node corresponding to the target range is the synchronization address when the key is within the target range. The device can further solve the problem that synchronization from a large data warehouse to an in-memory database cannot be performed during business use and data results are delayed.

[0086] Specifically, connect to any node in the cluster to query other nodes in the cluster and the range of slots allocated on each node, calculate in memory which slot each key belongs to, and then obtain which node in the Redis cluster should be written.

[0087] In some specific embodiments, the synchronization module includes a processing submodule and a writing submodule, wherein the processing submodule is used to group the data to be synchronized corresponding to the same storage node to obtain multiple data groups; the writing submodule is used to batch write the multiple data groups into different storage partitions of the same synchronization address. The device can further solve the problem of low efficiency of relying on a relay machine during data synchronization.

[0088] Specifically, an elastic partitioner can be used to repartition the data, redistribute the data keys distributed on the same node in the Redis cluster to one or more new partitions, and ensure that all keys under each data partition have the same target Redis node. Then start multiple computing processes to process each partition separately. In the process of processing a single partition, first read the first data of the partition, obtain the corresponding Redis cluster node and establish a direct connection with the node. Since it has been ensured that the data in the same partition is the same target Redis node, the data of the partition can be written to the node in batches using a pipeline, and the corresponding data expiration time TTL can be set during the writing process. At this point, the data synchronization process from the entire configured data warehouse to the in-memory database is completed.

[0089] The data synchronization device includes a processor and a memory, and the first acquisition unit, the second acquisition unit, the processing unit, and the writing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0090] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and data can be synchronized by adjusting kernel parameters.

[0091] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0092] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data synchronization method.

[0093] Specifically, the data synchronization method includes:

[0094] Step S201, obtaining a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0095] Specifically, the above configuration table can be one table or multiple tables. For example, one is a Redis cluster connection information table, including cluster ID, cluster node IP, cluster password and other information; the other is a write data job configuration table, including SQL statements for extracting data from the data warehouse, data warehouse name, target Redis cluster ID, data operation type, data TTL expiration time, etc.

[0096] Step S202, according to the configuration table, obtaining the key-value pair corresponding to the data in the data warehouse to obtain the original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0097] Specifically, first, submit a Spark job to the Hadoop cluster, which reads the Redis cluster connection information table and the data writing job configuration table in the configuration module and extracts data from the Hive warehouse. After the data extraction operation is completed and the original data is obtained, the filtered data in the Hive data warehouse is temporarily stored in the memory of the distributed cluster of the Spark job.

[0098] Step S203, performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0099] Specifically, the data format conversion process can convert multiple data formats supported by the Hive data warehouse into a data format supported by the Redis memory database to be written.

[0100] Step S204, writing the data to be synchronized into the memory database.

[0101] Specifically, writing data to an in-memory database can be achieved in several steps: First, write the data writing logic. For key-value storage, you may need to define keys and key values. For more complex data structures, you need to design the data writing logic based on the data types supported by the in-memory database. Then use the API or commands provided by the in-memory database to perform data writing operations, such as SET, LPUSH, SADD, HSET and other commands, depending on the in-memory database and data type you are using. When writing data, take into account possible errors and write corresponding error handling logic, such as retry mechanisms, error logging, etc. After the data is written, verification is performed to ensure that the data is stored correctly in the in-memory database.

[0102] Optionally, the above-mentioned in-memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple nodes, and the above-mentioned data synchronization system also includes a relational database. Obtaining the configuration table includes: obtaining the unique identity information of each of the above-mentioned clusters, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned in-memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized stored in the above-mentioned relational database, to obtain the above-mentioned configuration table.

[0103] Optionally, the above-mentioned configuration table includes at least an SQL statement, and the above-mentioned SQL statement is used to process the data in the above-mentioned data warehouse. According to the above-mentioned configuration table, the key-value pairs corresponding to the data in the above-mentioned data warehouse are obtained to obtain the original data, including: sending a request information to the above-mentioned data warehouse, wherein the above-mentioned request information is used to request synchronization of the data in the above-mentioned data warehouse; modifying the predetermined field in the above-mentioned SQL statement to obtain a select statement, the above-mentioned predetermined field includes a key and a key value, and the above-mentioned key is used to uniquely identify the above-mentioned key value; using the above-mentioned select statement to obtain the key-value pairs corresponding to the data in the above-mentioned data warehouse to obtain the original data.

[0104] Optionally, performing format conversion processing on the original data to obtain the data to be synchronized includes: converting the data format of the key value corresponding to the original data into the data format of the memory database to obtain the data to be synchronized.

[0105] Optionally, the above-mentioned memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple storage nodes, and the above-mentioned data to be synchronized is written into the above-mentioned memory database, including: obtaining the slot interval range corresponding to each of the above-mentioned storage nodes, and obtaining multiple target ranges, wherein the above-mentioned storage nodes correspond one-to-one to the above-mentioned slot interval ranges; determining the synchronization address according to the above-mentioned data to be synchronized and the above-mentioned target range; synchronizing the above-mentioned data to be synchronized to the above-mentioned synchronization address of the above-mentioned memory database.

[0106] Optionally, the synchronization address is determined based on the data to be synchronized and the target range, including: obtaining the key corresponding to the data to be synchronized, and determining whether the key is within the target range; if the key is within the target range, determining that the address of the storage node corresponding to the target range is the synchronization address.

[0107] Optionally, synchronizing the data to be synchronized to the synchronization address of the memory database includes: grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; and batch writing the multiple data groups into different storage partitions of the same synchronization address.

[0108] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the data synchronization method is executed when the program is running.

[0109] Specifically, the data synchronization method includes:

[0110] Step S201, obtaining a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0111] Specifically, the above configuration table can be one table or multiple tables. For example, one is a Redis cluster connection information table, including cluster ID, cluster node IP, cluster password and other information; the other is a write data job configuration table, including SQL statements for extracting data from the data warehouse, data warehouse name, target Redis cluster ID, data operation type, data TTL expiration time, etc.

[0112] Step S202, according to the configuration table, obtaining the key-value pair corresponding to the data in the data warehouse to obtain the original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0113] Specifically, first, submit a Spark job to the Hadoop cluster, which reads the Redis cluster connection information table and the data writing job configuration table in the configuration module and extracts data from the Hive warehouse. After the data extraction operation is completed and the original data is obtained, the filtered data in the Hive data warehouse is temporarily stored in the memory of the distributed cluster of the Spark job.

[0114] Step S203, performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0115] Specifically, the data format conversion process can convert multiple data formats supported by the Hive data warehouse into a data format supported by the Redis memory database to be written.

[0116] Step S204, writing the data to be synchronized into the memory database.

[0117] Specifically, writing data to an in-memory database can be achieved in several steps: First, write the data writing logic. For key-value storage, you may need to define keys and key values. For more complex data structures, you need to design the data writing logic based on the data types supported by the in-memory database. Then use the API or commands provided by the in-memory database to perform data writing operations, such as SET, LPUSH, SADD, HSET and other commands, depending on the in-memory database and data type you are using. When writing data, take into account possible errors and write corresponding error handling logic, such as retry mechanisms, error logging, etc. After the data is written, verification is performed to ensure that the data is stored correctly in the in-memory database.

[0118] Optionally, the above-mentioned in-memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple nodes, and the above-mentioned data synchronization system also includes a relational database. Obtaining the configuration table includes: obtaining the unique identity information of each of the above-mentioned clusters, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned in-memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized stored in the above-mentioned relational database, to obtain the above-mentioned configuration table.

[0119] Optionally, the above-mentioned configuration table includes at least an SQL statement, and the above-mentioned SQL statement is used to process the data in the above-mentioned data warehouse. According to the above-mentioned configuration table, the key-value pairs corresponding to the data in the above-mentioned data warehouse are obtained to obtain the original data, including: sending a request information to the above-mentioned data warehouse, wherein the above-mentioned request information is used to request synchronization of the data in the above-mentioned data warehouse; modifying the predetermined field in the above-mentioned SQL statement to obtain a select statement, the above-mentioned predetermined field includes a key and a key value, and the above-mentioned key is used to uniquely identify the above-mentioned key value; using the above-mentioned select statement to obtain the key-value pairs corresponding to the data in the above-mentioned data warehouse to obtain the original data.

[0120] Optionally, performing format conversion processing on the original data to obtain the data to be synchronized includes: converting the data format of the key value corresponding to the original data into the data format of the memory database to obtain the data to be synchronized.

[0121] Optionally, the above-mentioned memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple storage nodes, and the above-mentioned data to be synchronized is written into the above-mentioned memory database, including: obtaining the slot interval range corresponding to each of the above-mentioned storage nodes, and obtaining multiple target ranges, wherein the above-mentioned storage nodes correspond one-to-one to the above-mentioned slot interval ranges; determining the synchronization address according to the above-mentioned data to be synchronized and the above-mentioned target range; synchronizing the above-mentioned data to be synchronized to the above-mentioned synchronization address of the above-mentioned memory database.

[0122] Optionally, the synchronization address is determined based on the data to be synchronized and the target range, including: obtaining the key corresponding to the data to be synchronized, and determining whether the key is within the target range; if the key is within the target range, determining that the address of the storage node corresponding to the target range is the synchronization address.

[0123] Optionally, synchronizing the data to be synchronized to the synchronization address of the memory database includes: grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; and batch writing the multiple data groups into different storage partitions of the same synchronization address.

[0124] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0125] Step S201, obtaining a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0126] Step S202, according to the configuration table, obtaining the key-value pair corresponding to the data in the data warehouse to obtain the original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0127] Step S203, performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0128] Step S204, writing the data to be synchronized into the memory database.

[0129] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0130] Optionally, the above-mentioned in-memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple nodes, and the above-mentioned data synchronization system also includes a relational database. Obtaining the configuration table includes: obtaining the unique identity information of each of the above-mentioned clusters, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned in-memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized stored in the above-mentioned relational database, to obtain the above-mentioned configuration table.

[0131] Optionally, the above-mentioned configuration table includes at least an SQL statement, and the above-mentioned SQL statement is used to process the data in the above-mentioned data warehouse. According to the above-mentioned configuration table, the key-value pairs corresponding to the data in the above-mentioned data warehouse are obtained to obtain the original data, including: sending a request information to the above-mentioned data warehouse, wherein the above-mentioned request information is used to request synchronization of the data in the above-mentioned data warehouse; modifying the predetermined field in the above-mentioned SQL statement to obtain a select statement, the above-mentioned predetermined field includes a key and a key value, and the above-mentioned key is used to uniquely identify the above-mentioned key value; using the above-mentioned select statement to obtain the key-value pairs corresponding to the data in the above-mentioned data warehouse to obtain the original data.

[0132] Optionally, performing format conversion processing on the original data to obtain the data to be synchronized includes: converting the data format of the key value corresponding to the original data into the data format of the memory database to obtain the data to be synchronized.

[0133] Optionally, the above-mentioned memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple storage nodes, and the above-mentioned data to be synchronized is written into the above-mentioned memory database, including: obtaining the slot interval range corresponding to each of the above-mentioned storage nodes, and obtaining multiple target ranges, wherein the above-mentioned storage nodes correspond one-to-one to the above-mentioned slot interval ranges; determining the synchronization address according to the above-mentioned data to be synchronized and the above-mentioned target range; synchronizing the above-mentioned data to be synchronized to the above-mentioned synchronization address of the above-mentioned memory database.

[0134] Optionally, the synchronization address is determined based on the data to be synchronized and the target range, including: obtaining the key corresponding to the data to be synchronized, and determining whether the key is within the target range; if the key is within the target range, determining that the address of the storage node corresponding to the target range is the synchronization address.

[0135] Optionally, synchronizing the data to be synchronized to the synchronization address of the memory database includes: grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; and batch writing the multiple data groups into different storage partitions of the same synchronization address.

[0136] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0137] Step S201, obtaining a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database;

[0138] Step S202, according to the configuration table, obtaining the key-value pair corresponding to the data in the data warehouse to obtain the original data, wherein the key-value pair includes the key and the key value corresponding to the key;

[0139] Step S203, performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database;

[0140] Step S204, writing the data to be synchronized into the memory database.

[0141] Optionally, the above-mentioned in-memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple nodes, and the above-mentioned data synchronization system also includes a relational database. Obtaining the configuration table includes: obtaining the unique identity information of each of the above-mentioned clusters, the key of each of the above-mentioned clusters, the network protocol address of each of the above-mentioned nodes, the data operation type of the above-mentioned in-memory database, the unique identity information of the above-mentioned data warehouse, the data operation type of the above-mentioned data to be synchronized, and the expiration time of the above-mentioned data to be synchronized stored in the above-mentioned relational database, to obtain the above-mentioned configuration table.

[0142] Optionally, the above-mentioned configuration table includes at least an SQL statement, and the above-mentioned SQL statement is used to process the data in the above-mentioned data warehouse. According to the above-mentioned configuration table, the key-value pairs corresponding to the data in the above-mentioned data warehouse are obtained to obtain the original data, including: sending a request information to the above-mentioned data warehouse, wherein the above-mentioned request information is used to request synchronization of the data in the above-mentioned data warehouse; modifying the predetermined field in the above-mentioned SQL statement to obtain a select statement, the above-mentioned predetermined field includes a key and a key value, and the above-mentioned key is used to uniquely identify the above-mentioned key value; using the above-mentioned select statement to obtain the key-value pairs corresponding to the data in the above-mentioned data warehouse to obtain the original data.

[0143] Optionally, performing format conversion processing on the original data to obtain the data to be synchronized includes: converting the data format of the key value corresponding to the original data into the data format of the memory database to obtain the data to be synchronized.

[0144] Optionally, the above-mentioned memory database includes multiple clusters, any one of the above-mentioned clusters includes multiple storage nodes, and the above-mentioned data to be synchronized is written into the above-mentioned memory database, including: obtaining the slot interval range corresponding to each of the above-mentioned storage nodes, and obtaining multiple target ranges, wherein the above-mentioned storage nodes correspond one-to-one to the above-mentioned slot interval ranges; determining the synchronization address according to the above-mentioned data to be synchronized and the above-mentioned target range; synchronizing the above-mentioned data to be synchronized to the above-mentioned synchronization address of the above-mentioned memory database.

[0145] Optionally, the synchronization address is determined based on the data to be synchronized and the target range, including: obtaining the key corresponding to the data to be synchronized, and determining whether the key is within the target range; if the key is within the target range, determining that the address of the storage node corresponding to the target range is the synchronization address.

[0146] Optionally, synchronizing the data to be synchronized to the synchronization address of the memory database includes: grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; and batch writing the multiple data groups into different storage partitions of the same synchronization address.

[0147] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0157] 1) The data synchronization method of the present application is applied to a data synchronization system, which includes a data warehouse and an in-memory database. First, a configuration table is obtained, which is at least used to characterize the correspondence between the data in the data warehouse and the data in the in-memory database; then, according to the configuration table, the key-value pairs corresponding to the data in the data warehouse are obtained to obtain the original data, and the key-value pairs include the key and the key value corresponding to the key; then, the original data is formatted and processed to obtain the data to be synchronized, and the data format of the data to be synchronized is the same as the data format of the in-memory database; finally, the data to be synchronized is written into the in-memory database. By obtaining the correspondence between the data of the pre-configured data warehouse and the in-memory database, that is, the configuration table, and then determining the data to be synchronized according to the configuration, there is no need to perform data synchronization through a transfer device or file landing. Data updates in the data warehouse can be synchronized to the in-memory database of the production system in a timely manner. The problem of untimely data synchronization between the data warehouse and the in-memory database is solved.

[0158] 2) The data synchronization device of the present application is applied to a data synchronization system, which includes a data warehouse and a memory database. The first acquisition unit acquires a configuration table, which is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database; the second acquisition unit acquires the key-value pair corresponding to the data in the data warehouse according to the configuration table to obtain the original data, and the key-value pair includes a key and a key value corresponding to the key; the processing unit performs format conversion processing on the original data to obtain the data to be synchronized, and the data format of the data to be synchronized is the same as the data format of the memory database; the writing unit writes the data to be synchronized into the memory database. By acquiring the corresponding relationship between the data of the pre-configured data warehouse and the memory database, that is, the configuration table, and then determining the data to be synchronized according to the configuration, there is no need to perform data synchronization through a transfer device or file landing. The data update in the data warehouse can be synchronized to the memory database of the production system in a timely manner. The problem of untimely data synchronization between the data warehouse and the memory database is solved.

[0159] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The method is applied to a data synchronization system, which includes a data warehouse and an in-memory database, including: Acquire a configuration table, wherein the configuration table is at least used to characterize the corresponding relationship between the data in the data warehouse and the data in the memory database; According to the configuration table, a key-value pair corresponding to the data in the data warehouse is obtained to obtain original data, wherein the key-value pair includes the key and the key value corresponding to the key; Performing format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database; The data to be synchronized is written into the memory database.

2. The method according to claim 1, characterized in that The memory database includes multiple clusters, any one of the clusters includes multiple nodes, the data synchronization system also includes a relational database, and obtaining a configuration table includes: The unique identity information of each cluster, the key of each cluster, the network protocol address of each node, the data operation type of the memory database, the unique identity information of the data warehouse, the data operation type of the data to be synchronized and the expiration time of the data to be synchronized stored in the relational database are obtained to obtain the configuration table.

3. The method according to claim 1, characterized in that: The configuration table at least includes an SQL statement, and the SQL statement is used to process the data in the data warehouse. According to the configuration table, the key-value pairs corresponding to the data in the data warehouse are obtained to obtain the original data, including: Sending a request message to the data warehouse, wherein the request message is used to request synchronization of data in the data warehouse; Modify a predetermined field in the SQL statement to obtain a select statement, wherein the predetermined field includes a key and a key value, and the key is used to uniquely identify the key value; The select statement is used to obtain the key-value pairs corresponding to the data in the data warehouse to obtain the original data.

4. The method according to claim 1, characterized in that: The original data is format converted to obtain data to be synchronized, including: The data format of the key value corresponding to the original data is converted into the data format of the memory database to obtain the data to be synchronized.

5. The method according to claim 1, characterized in that The memory database includes a plurality of clusters, any one of the clusters includes a plurality of storage nodes, and writing the data to be synchronized into the memory database includes: Obtaining a slot interval range corresponding to each of the storage nodes to obtain a plurality of target ranges, wherein the storage nodes correspond to the slot interval ranges one by one; Determining a synchronization address according to the data to be synchronized and the target range; The data to be synchronized is synchronized to the synchronization address of the memory database.

6. The method according to claim 5, characterized in that Determining a synchronization address according to the data to be synchronized and the target range includes: Obtaining a key corresponding to the data to be synchronized, and determining whether the key is within the target range; In the case where the key is within the target range, the address of the storage node corresponding to the target range is determined as the synchronization address.

7. The method according to claim 5, characterized in that Synchronizing the data to be synchronized to the synchronization address of the memory database includes: Grouping the data to be synchronized corresponding to the same storage node to obtain multiple data groups; Writing the plurality of data groups in batches into different storage partitions of the same synchronization address.

8. A data synchronization device, characterized in that: The device is applied to a data synchronization system, which includes a data warehouse and a memory database, including: A first acquisition unit, configured to acquire a configuration table, wherein the configuration table is at least used to characterize a correspondence between the data in the data warehouse and the data in the memory database; A second acquisition unit is used to acquire a key-value pair corresponding to the data in the data warehouse according to the configuration table to obtain original data, wherein the key-value pair includes the key and a key value corresponding to the key; A processing unit, configured to perform format conversion processing on the original data to obtain data to be synchronized, wherein the data format of the data to be synchronized is the same as the data format of the memory database; A writing unit is used to write the data to be synchronized into the memory database.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Data synchronization method and device, electronic equipment and medium

    CN113971195A

  • Method and device for synchronizing database to data warehouse and electronic device

    CN114185907A

  • Data writing method and device, computer equipment and storage medium

    CN116466884A

  • Database data synchronization method and device, equipment and medium

    CN117453808A