Database data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data synchronization, and discloses a database data synchronization method and device, computer equipment and a storage medium, and the method comprises the following steps: creating an array data table in a ClickHouse database; obtaining target data from the MongoDB database; analyzing the target data to obtain a name, a data type and a numerical value of each row of data in the target data; and synchronizing the target data to an array data table of the ClickHouse database based on the name, the data type and the numerical value of each row of data. According to the method, the data of the MongoDB database is synchronized by utilizing the array data table of the ClickHouse database, and the array fields are used, so that multiple pieces of data with the same data type can be more efficiently synchronized and managed, the data query and analysis performance is remarkably improved, meanwhile, high flexibility is provided, the method can adapt to changes of different data volumes and data types, and the method is suitable for large-scale popularization and application. And the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a database data synchronization method, device, computer equipment and storage medium. Background Art

[0002] As a document-based database, MongoDB is suitable for synchronizing data with non-fixed fields. However, with the development of business, some online analytical processing tasks are not suitable for direct implementation in MongoDB. ClickHouse, as an open source database specifically for online analytical processing, has excellent performance in the aforementioned tasks. Since ClickHouse does not officially provide a synchronization engine for MongoDB, how to synchronize database data is a problem that needs to be solved.

[0003] Currently, in order to synchronize MongoDB data to ClickHouse, you need to predefine all the fields in the MongoDB table in ClickHouse. For example, if any data table of MongoDB has three fields: ID (integer type), Name (text type), and Score (floating point type), ClickHouse needs to create a data table including these three fields first, and then synchronize the data. However, once a new field is inserted into a MongoDB data table, ClickHouse must rebuild the table. Moreover, there are differences between the fields of different MongoDB data tables. When synchronizing data in different data tables, ClickHouse needs to repeatedly build tables, resulting in low data synchronization efficiency. Summary of the invention

[0004] In view of this, the present invention provides a database data synchronization method, apparatus, computer equipment and storage medium to solve the problem of low data synchronization efficiency.

[0005] In a first aspect, the present invention provides a method for synchronizing database data, the method comprising:

[0006] Create an array data table in the ClickHouse database. The array data table includes multiple array fields and array value fields corresponding to each array field.

[0007] Get the target data from the MongoDB database;

[0008] Parse the target data to obtain the name, data type and value of each row of data in the target data;

[0009] Based on the name, data type, and value of each row of data in the target data, synchronize the target data to the array data table of the ClickHouse database.

[0010] The database data synchronization method provided by the embodiment of the present invention synchronizes the data of the MongoDB database by utilizing the array data table of the ClickHouse database, so that it can flexibly accommodate fields of different types and numbers, reduce the creation of redundant fields, and allow users to query data originally belonging to the MongoDB database from a single ClickHouse database without knowing the actual synchronization details. The use of array fields can more efficiently synchronize and manage multiple data with the same data type, thereby realizing data migration and synchronization from the MongoDB database to the ClickHouse database. The array field design and query optimization characteristics of the ClickHouse database significantly improve the performance of data query and analysis, while providing a high degree of flexibility, which can adapt to changes in different data volumes and data types, and improve data synchronization efficiency.

[0011] In an optional implementation, obtaining target data from a MongoDB database includes:

[0012] Establish a connection between the Gorm library and the MongoDB database;

[0013] Query target data from the MongoDB database;

[0014] Map the target data to the structure of the Gorm library.

[0015] The database data synchronization method provided in the embodiment of the present invention can quickly and accurately obtain target data from the MongoDB database through the efficient query and connection management functions of the Gorm library. The use of the structure makes data processing more standardized and orderly. The target data is mapped to the structure to ensure the accuracy of the target data.

[0016] In an optional implementation, parsing the target data to obtain the name, data type, and value of each row of data in the target data includes:

[0017] Traverse the fields of each row of data in the target data in the structure of the Gorm library;

[0018] Get the name, data type, and value of the field in each row of data based on the reflection mechanism.

[0019] The database data synchronization method provided in the embodiment of the present invention can automatically process all fields in the structure through traversal and reflection mechanisms to ensure the integrity and consistency of the data. The automated processing reduces human errors and improves the accuracy and reliability of data processing.

[0020] In an optional implementation, based on the name, data type, and value of each row of data in the target data, the target data is synchronized with an array data table of the ClickHouse database, including:

[0021] For any row of data in the target data, based on the data type, an array field with the same data type is determined from the array data table;

[0022] Synchronize the data name to the array field, and synchronize the data value to the array value field corresponding to the array field.

[0023] The database data synchronization method provided in the embodiment of the present invention can more efficiently synchronize and manage multiple data with the same data type by matching and synchronizing the array data table of the ClickHouse database with the target data in the MongoDB database according to the data type, thereby improving the data synchronization efficiency.

[0024] In an optional implementation, the length and number of fields of the array data table are variable.

[0025] The database data synchronization method provided by the embodiment of the present invention makes data management more flexible and adapts to changes in different data volumes through a dynamically expanded array data table with variable fields and length designs.

[0026] In an optional embodiment, the method further includes:

[0027] Execute data query statements in the ClickHouse database to obtain any row of data in the target data.

[0028] The database data synchronization method provided by the embodiment of the present invention can query any row of target data belonging to the MongoDB database by executing a data query statement in the ClickHouse database, thereby realizing data synchronization.

[0029] In a second aspect, the present invention provides a database data synchronization device, the device comprising:

[0030] Create a module for creating an array data table in the ClickHouse database. The array data table includes multiple array fields and array value fields corresponding to each array field.

[0031] The acquisition module is used to obtain target data from the MongoDB database;

[0032] A parsing module is used to parse the target data and obtain the name, data type and value of each row of data in the target data;

[0033] The synchronization module is used to synchronize the target data to the array data table of the ClickHouse database based on the name, data type, and value of each row of data in the target data.

[0034] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the database data synchronization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the database data synchronization method of the first aspect or any corresponding embodiment thereof.

[0036] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the database data synchronization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flowchart of a database data synchronization method according to an embodiment of the present invention;

[0039] Figure 2 is a flowchart of another database data synchronization method according to an embodiment of the present invention;

[0040] Figure 3 is a structural block diagram of a database data synchronization device according to an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0043] Some OLAP tasks are not suitable for direct implementation in MongoDB, and ClickHouse, as an open source database specifically for OLAP, has excellent performance in the aforementioned tasks. Since ClickHouse does not officially provide a synchronization engine for MongoDB, how to synchronize database data is a problem that needs to be solved. Usually, all fields in the MongoDB table are predefined in ClickHouse, and then data synchronization is performed. However, once a new field is inserted into a MongoDB data table, ClickHouse must rebuild the table. Moreover, there are differences in the fields of different MongoDB data tables. When synchronizing data in different data tables, ClickHouse needs to repeatedly build tables, resulting in low data synchronization efficiency. The database data synchronization method provided in the embodiment of the present invention synchronizes the data of the MongoDB database by utilizing the array data table of the ClickHouse database. By using the array field, multiple data with the same data type can be synchronized and managed more efficiently, thereby realizing data migration and synchronization from the MongoDB database to the ClickHouse database. The array field design and query optimization characteristics of the ClickHouse database significantly improve the performance of data query and analysis, while providing a high degree of flexibility, which can adapt to changes in different data volumes and data types and improve data synchronization efficiency.

[0044] According to an embodiment of the present invention, an embodiment of a database data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] In this embodiment, a database data synchronization method is provided, which can be used in electronic devices. Figure 1 is a flowchart of a database data synchronization method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0046] Step S101, create an array data table in the ClickHouse database, the array data table includes multiple array fields and array value fields corresponding to each array field. Specifically, assuming that the data to be synchronized this time is host data, and there is an association with the existing host data table in the ClickHouse database, the host data can be synchronized to the host data table of ClickHouse. Assuming that the data to be synchronized this time is application data, it has nothing to do with all the existing data in the ClickHouse database. A blank array data table can be created to synchronize the application data to the blank array data table. It should be noted that the above assumptions are all examples. In actual situations, the array data table can be a blank data table or a data table with existing data, which can be set according to actual needs. The array data table is composed of multiple array fields and array value fields corresponding to each array field, such as array field: string_names, and the corresponding array value field: string_values; array field: int_names, and the corresponding array value field: int_values; array field: float_names, and the corresponding array value field: float_values. Optionally, the process of creating an array data table in the ClickHouse database can be implemented by the following code:

[0047] CREATE TABLE common_table(

[0048] `table_name`String COMMENT"MongoDB table name",

[0049] `string_names`Array(String)COMMENT"Array field of string type",

[0050] `string_values`Array(String)COMMENT "Numeric array field of string type"

[0051] ,

[0052] `number_names`Array(Int64)COMMENT"Array field of integer type",

[0053] `number_values`Array(Int64)COMMENT "Numerical array field of integer type")

[0054] Step S102, obtaining target data from the MongoDB database. Specifically, extract the target data to be synchronized from the MongoDB database. Assume that the code corresponding to the target data is as follows: [

[0056] {"ip":"1.1.1.1","cpu":8,"os":"Centos"},

[0057] {"ip":"2.2.2.2","cpu":16,"os_version":"ubuntu18.4","os":"ubuntu"},

[0058] {"ip":"3.3.3.3","memory":32,"address":"guangzhou"} ]

[0060] Step S103, parse the target data to obtain the name, data type and value of each row of data in the target data. Specifically, referring to the code of the target data above, the target data in the MongoDB database is stored by row, and each row of data is also parsed during parsing to obtain the name, value and data type of the corresponding field in each row of data. For example, by parsing "ip":"3.3.3.3","memory":32,"address":"guangzhou", the name, data type and value of the data shown below can be obtained:

[0061] Name: ip, Data type: string, Value: 3.3.3.3;

[0062] Name: memory, Data type: int, Value: 32;

[0063] Name: address, Data type: string, Value: guangzhou.

[0064] Step S104, based on the name, data type and value of each row of data in the target data, synchronize the target data to the array data table of the ClickHouse database. Specifically, based on the name, data type and value of each row of data, the array in the ClickHouse database is used for data synchronization, which can flexibly accommodate different types and numbers of fields in the MongoDB database, and even if the target data obtained from the MongoDB database changes, it can be efficiently synchronized to the ClickHouse database.

[0065] The database data synchronization method provided by the embodiment of the present invention synchronizes the data of the MongoDB database by utilizing the array data table of the ClickHouse database, so that it can flexibly accommodate fields of different types and numbers, reduce the creation of redundant fields, and allow users to query data originally belonging to the MongoDB database from a single ClickHouse database without knowing the actual synchronization details. The use of array fields can more efficiently synchronize and manage multiple data with the same data type, thereby realizing data migration and synchronization from the MongoDB database to the ClickHouse database. The array field design and query optimization characteristics of the ClickHouse database significantly improve the performance of data query and analysis, while providing a high degree of flexibility, which can adapt to changes in different data volumes and data types, and improve data synchronization efficiency.

[0066] In this embodiment, a database data synchronization method is provided, which can be used for the above electronic device. Figure 2 is a flowchart of another database data synchronization method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0067] Step S201: Create an array data table in the ClickHouse database. The array data table includes multiple array fields and array value fields corresponding to each array field. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0068] In an optional implementation, the length and number of fields of the array data table are variable. Specifically, the length and fields of the array data table can be adaptively adjusted to meet data synchronization requirements. For example, the length of the current array data table is 50, that is, a maximum of 50 groups of data can be placed under each array field. If 50 groups of data are already placed under any array field during data synchronization, the length of the array data table can be increased. In addition, if the current array data table only has the array field: string_names, and the corresponding array value field: string_values; array field: int_names, and the corresponding array value field: int_values. If there is floating point type data in the target data, the fields float_names and float_values ​​can be added to the array data table. Through the dynamically expanded array data table, its variable field and length design make data management more flexible and adaptable to changes in different data volumes.

[0069] Step S202, obtaining target data from the MongoDB database.

[0070] Specifically, the above step S202 includes:

[0071] Step S2021, establish a connection between the Gorm library and the MongoDB database. Specifically, Gorm is a Go language ORM (Object-Relational Mapping) library. By establishing a connection between the Gorm library and the MongoDB database, the target data can be quickly and accurately obtained from the MongoDB database.

[0072] Step S2022, query the target data from the MongoDB database. Specifically, the Gorm library queries the target data to be synchronized from the MongoDB database according to the query conditions.

[0073] Step S2023, mapping the target data to the structure of the Gorm library. Specifically, mapping the target data to the structure of the Gorm library ensures accurate acquisition of the target data. Optionally, acquiring data through the Gorm library is a prior art and will not be repeated here.

[0074] Step S203, parsing the target data to obtain the name, data type and value of each row of data in the target data.

[0075] Specifically, the above step S203 includes:

[0076] Step S2031, traverse the fields of each row of data in the target data in the structure of the Gorm library. Specifically, traverse the instance of the structure in the Gorm library to obtain the fields of each row of data.

[0077] Step S2032, based on the reflection mechanism, obtain the name, data type and value of the field in each row of data. Specifically, use Golang's reflection mechanism to traverse each field of the structure and obtain the name, data type and value of the field. Optionally, the name, data type and value of the target data obtained by parsing can be found in the above step S103.

[0078] Step S204, based on the name, data type and value of each row of data in the target data, synchronize the target data to the array data table of the ClickHouse database.

[0079] Specifically, the above step S204 includes:

[0080] Step S2041, for any row of data in the target data, based on the data type of the data, determine the array field consistent with the data type from the array data table. Specifically, assume that any three rows of data in the target data parsed by the above steps are: name: ip, data type: string, value: 3.3.3.3; name: memory, data type: int, value: 32; name: address, data type: string, value: guangzhou. First, synchronize the data named ip, whose data type is string, and determine the array field string_names that is consistent with string from the array data table of the ClickHouse database. Then, synchronize the data named memory, whose data type is int, and determine the array field int_names that is consistent with int from the array data table of the ClickHouse database. Then, synchronize the data named address, whose data type is string, and determine the array field string_names that is consistent with string from the array data table of the ClickHouse database.

[0081] Step S2042, synchronize the name of the data to the array field, and synchronize the value of the data to the array value field corresponding to the array field. Specifically, first synchronize the name ip in the data named ip to string_names, and synchronize the value 3.3.3.3 to string_values. Then, synchronize the name memory in the data named memory to int_names, and synchronize the value 32 to int_values. Then, synchronize the name address in the data named address to string_names, located below ip, and synchronize the value guangzhou to string_values. Efficient synchronization and management of multiple data with the same data type is achieved. Optionally, the array data table can be seen in the table below.

[0082]

[0083] It should be noted that when synchronizing multiple data, for one data, the array field is first determined in step S3041, and then the data synchronization is implemented in step S3042. After the synchronization of one data is completed, the synchronization of the next data is executed.

[0084] Step S205, execute the data query statement in the ClickHouse database to obtain any row of data in the target data. Specifically, since the target data in the MongoDB database has been synchronized to the ClickHouse database, any data in the target data can be obtained by executing the data query statement in the ClickHouse database. For example, if you want to query the data with the ip "3.3.3.3", first set the table name of the array data table to be queried, that is, table_name is "host", and then find out the position of the ip in string_names, thereby determining the position of the ip value in the string_values ​​array, and accurately filtering out the data with the ip "3.3.3.3" from a general ClickHouse array data table, that is, allowing the data originally belonging to the MongoDB database to be queried from a single ClickHouse database without knowing the actual synchronization details. The data query statement is as follows:

[0085] SELECT

[0086] string_values[indexOf(string_names,'ip')]as ip,

[0087] FROM common_table WHERE table_name='host'AND ip='3.3.3.3'

[0088] The database data synchronization method provided by the embodiment of the present invention synchronizes the data of the MongoDB database by utilizing the array data table of the ClickHouse database, so that it can flexibly accommodate fields of different types and numbers, reduce the creation of redundant fields, and allow users to query data originally belonging to the MongoDB database from a single ClickHouse database without knowing the actual synchronization details. The use of array fields can more efficiently synchronize and manage multiple data with the same data type, thereby realizing data migration and synchronization from the MongoDB database to the ClickHouse database. The array field design and query optimization characteristics of the ClickHouse database significantly improve the performance of data query and analysis, while providing a high degree of flexibility, which can adapt to changes in different data volumes and data types, and improve data synchronization efficiency.

[0089] In this embodiment, a database data synchronization device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0090] This embodiment provides a database data synchronization device, such as Figure 3 As shown, including:

[0091] A creation module 301 is used to create an array data table in the ClickHouse database. The array data table includes multiple array fields and an array value field corresponding to each array field.

[0092] The acquisition module 302 is used to acquire target data from the MongoDB database.

[0093] The parsing module 303 is used to parse the target data to obtain the name, data type and value of each row of data in the target data.

[0094] The synchronization module 304 is used to synchronize the target data to the array data table of the ClickHouse database based on the name, data type and value of each row of data in the target data.

[0095] In some optional implementations, the acquisition module 302 includes:

[0096] The creation unit is used to establish a connection between the Gorm library and the MongoDB database.

[0097] The query unit is used to query target data from the MongoDB database.

[0098] Mapping unit, used to map target data into the structure of Gorm library.

[0099] In some optional implementations, the parsing module 303 includes:

[0100] The traversal unit is used to traverse the fields of each row of data in the target data in the structure of the Gorm library.

[0101] The acquisition unit is used to obtain the name, data type, and value of the field in each row of data based on the reflection mechanism.

[0102] In some optional implementations, the synchronization module 304 includes:

[0103] The determination unit is used to determine, for any row of data in the target data, an array field consistent with the data type from the array data table based on the data type of the data.

[0104] The synchronization unit is used to synchronize the name of the data to the array field, and synchronize the value of the data to the array value field corresponding to the array field.

[0105] In some optional implementations, the length and number of fields of the array data table are variable.

[0106] In some optional embodiments, the device further comprises:

[0107] The query module is used to execute data query statements in the ClickHouse database to obtain any row of data in the target data.

[0108] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0109] The database data synchronization device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0110] The embodiment of the present invention also provides a computer device having the above Figure 3 The database data synchronization device shown.

[0111] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0112] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0113] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0114] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0116] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0117] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0118] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0119] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0120] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for synchronizing database data, characterized in that: The method comprises: Create an array data table in the ClickHouse database, wherein the array data table includes multiple array fields and an array value field corresponding to each array field; Get the target data from the MongoDB database; Parse the target data to obtain the name, data type and value of each row of data in the target data; Based on the name, data type and value of each row of data in the target data, the target data is synchronized to the array data table of the ClickHouse database.

2. The method according to claim 1, characterized in that The step of obtaining target data from the MongoDB database includes: Establish a connection between the Gorm library and the MongoDB database; Query the target data from the MongoDB database; Map the target data to the structure of the Gorm library.

3. The method according to claim 2, characterized in that The parsing of the target data to obtain the name, data type and value of each row of data in the target data includes: Traverse the fields of each row of data in the target data in the structure of the Gorm library; Get the name, data type, and value of the field in each row of data based on the reflection mechanism.

4. The method according to claim 1, characterized in that The step of synchronizing the target data with the array data table of the ClickHouse database based on the name, data type and value of each row of data in the target data includes: For any row of data in the target data, based on the data type of the data, determining an array field consistent with the data type from the array data table; The name of the data is synchronized to the array field, and the value of the data is synchronized to the array value field corresponding to the array field.

5. The method according to claim 1, characterized in that The length and number of fields of the array data table are variable.

6. The method according to claim 1, characterized in that The method further comprises: Execute a data query statement in the ClickHouse database to obtain any row of data in the target data.

7. A database data synchronization device, characterized in that: The device comprises: A creation module is used to create an array data table in the ClickHouse database, wherein the array data table includes multiple array fields and an array value field corresponding to each array field; The acquisition module is used to obtain target data from the MongoDB database; A parsing module, used to parse the target data to obtain the name, data type and value of each row of data in the target data; A synchronization module is used to synchronize the target data to the array data table of the ClickHouse database based on the name, data type and value of each row of data in the target data.

8. A computer device, characterized in that: include: A synchronizer and a processor, wherein the synchronizer and the processor are communicatively connected to each other, the synchronizer synchronizes computer instructions, and the processor executes the database data synchronization method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is synchronized with computer instructions, and the computer instructions are used to enable a computer to execute the database data synchronization method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the database data synchronization method according to any one of claims 1 to 6.

Images