Database synchronization method and system and computer storage medium

Abstract

The invention discloses a database synchronization method and system and a computer storage medium, and the method comprises the steps: obtaining and analyzing to-be-synchronized data of a source database, and determining a corresponding synchronization data field; grouping the synchronous data fields according to at least one of accumulated data volume and accumulated time of the to-be-synchronized data, and determining a plurality of corresponding request bodies according to each group of synchronous data fields; and generating a corresponding data synchronization request according to each request body, and synchronously or asynchronously sending the data synchronization request to a target database. According to the method, the data volume of single synchronization is reduced, the network transmission burden is reduced, and the synchronization process can be performed in parallel, so that the synchronization efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a database synchronization method, system and computer storage medium. Background Art

[0002] With the advent of the big data era, enterprises' demands for database storage and query are becoming increasingly diverse and complex. Traditional relational databases, such as MySQL, perform well in terms of transactional and data consistency when processing large-scale data, but often fail to meet the complex aggregation query requirements of high concurrency and low latency.

[0003] To overcome these limitations, companies began to seek to store data in a database, such as MongoDB, which has become the preferred data storage solution for many companies due to its flexible document model, scalability, and high-performance write operations. At the same time, they use another database that performs better on complex aggregate queries, such as ClickHouse, to perform data analysis and queries. The coordination between the two requires data synchronization operations.

[0004] However, there are significant differences between MongoDB's document model and ClickHouse's column storage model. Different types of data (such as nested objects, arrays, etc.) require complex conversion during the synchronization process. Especially when the amount of data in MongoDB is huge, synchronizing all data at one time may cause system performance to degrade and affect the normal operation of other businesses. Summary of the invention

[0005] One of the purposes of the present invention is to provide a database synchronization method to solve the technical problem that the prior art does not support synchronization of large-scale complex data types, resulting in slow synchronization speed, low efficiency and poor reliability.

[0006] One of the objectives of the present invention is to provide a database synchronization system.

[0007] One of the objects of the present invention is to provide a computer storage medium.

[0008] In order to achieve one of the above-mentioned purposes of the invention, the present invention provides a database synchronization method, including: acquiring and parsing the data to be synchronized of the source database, and determining the corresponding synchronization data fields; grouping the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized, and determining a corresponding number of request bodies according to each group of synchronization data fields; generating a corresponding data synchronization request according to each request body, and sending the data synchronization request to the target database synchronously or asynchronously.

[0009] As a further improvement of an embodiment of the present invention, the data to be synchronized in the source database is parsed to determine the corresponding synchronization data field, including: when the data to be synchronized is a nested object, the nested object is flattened to determine the corresponding synchronization data field, the synchronization data field contains a number of key-value pairs, and the key in the key-value pair is formed based on the parent key and child key of the nested object.

[0010] As a further improvement of an embodiment of the present invention, the flattening processing of the nested object and determining the corresponding synchronization data field includes: traversing all child objects of the nested object, determining the key-value pairs of each nested level, combining the parent key and child key of each level key-value pair into a new key, and the value corresponding to the lowest level of the nested object is the new value; splicing the new key and the new value to form a new key-value pair, and updating the synchronization data field based on the new key-value pair.

[0011] As a further improvement of an embodiment of the present invention, before the data to be synchronized is a nested object, it includes: traversing the data objects containing key-value pairs in the data to be synchronized, and determining whether the corresponding values ​​are of dictionary type; if so, determining that the data to be synchronized contains nested objects; if not, determining that the data to be synchronized contains non-nested objects.

[0012] As a further improvement of an embodiment of the present invention, the data to be synchronized in the source database is parsed to determine the corresponding synchronization data field, including: when the data to be synchronized is a non-nested object, the key and value of each key-value pair in the non-nested object are converted into a type according to the target storage type defined by the target database, and updated to the synchronization data field.

[0013] As a further improvement of an embodiment of the present invention, before the method is performed based on at least one of the cumulative data volume and the cumulative time of the data to be synchronized, the method includes: detecting the cumulative data volume of the data to be synchronized stored in the source database within a unit time, and judging whether the cumulative data volume meets a preset first threshold; grouping the synchronization data fields based on at least one of the cumulative data volume and the cumulative time of the data to be synchronized includes: if so, grouping the synchronization data fields according to the cumulative data volume of the data to be synchronized.

[0014] As a further improvement of an embodiment of the present invention, before the method is performed based on at least one of the accumulated data volume and the accumulated time of the data to be synchronized, the method includes: obtaining the historical time of the last completion of data synchronization, and judging whether the time difference between the current time and the historical time is greater than or equal to a preset second threshold; grouping the synchronization data fields based on at least one of the accumulated data volume and the accumulated time of the data to be synchronized includes: if so, grouping the synchronization data fields according to the time difference.

[0015] As a further improvement of an embodiment of the present invention, the generating of a corresponding data synchronization request according to each request body respectively includes: obtaining and parsing log information of a source database, determining request parameters, wherein the request parameters include at least one of an address, a port, authentication information and a synchronization operation type of a target database; and generating a corresponding data synchronization request according to the request parameters and the request body.

[0016] As a further improvement of an embodiment of the present invention, the method of determining a corresponding number of request bodies based on each group of synchronization data fields includes: converting all key-value pairs in each group of synchronization data fields into request data in a preset format, and determining the corresponding request body based on the request data, wherein the preset format includes at least one of CSV and XML.

[0017] As a further improvement of an implementation mode of the present invention, the source database is MongoDB, and the target database is ClickHouse.

[0018] To achieve one of the above-mentioned purposes of the invention, the present invention also provides a database synchronization method, which is applied to a target database, comprising: responding to a first data synchronization request from a source database, determining a first synchronization data field and a first synchronization operation type corresponding to a first synchronization request, wherein the first data synchronization request is determined based on any one of the database synchronization methods; synchronizing the first synchronization data field to the target database according to the first synchronization operation type.

[0019] To achieve one of the above-mentioned purposes of the invention, the present invention also provides a database synchronization system, including: a source database, used to generate data to be synchronized; a processing module, used to parse the data to be synchronized and determine the corresponding synchronization data fields; used to group the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized, and determine a corresponding number of request bodies according to each group of synchronization data fields; a sending module, used to generate a corresponding data synchronization request according to each request body, and send the data synchronization request to a target database synchronously or asynchronously; a target database, used to respond to the data synchronization request and synchronize the data to be synchronized to the target database according to the data synchronization request.

[0020] To achieve one of the above-mentioned objects of the invention, the present invention also provides a computer storage medium, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned database synchronization method.

[0021] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0022] The present invention adopts a database synchronization method, which can reduce unnecessary data transmission and processing and improve the accuracy and efficiency of synchronization by obtaining and parsing the data to be synchronized of the source database and determining the corresponding synchronization data fields; at the same time, according to the accumulated data volume and / or accumulated time of the data to be synchronized, the synchronization data fields generated by the analysis are grouped, and the request body is constructed according to the synchronization data fields of each group, which not only reduces the data volume of a single synchronization and reduces the network transmission burden, but also allows the synchronization process to be carried out in parallel, thereby improving the synchronization efficiency. In addition, by sending data synchronization requests to the target database synchronously or asynchronously, the sending strategy can be flexibly adjusted according to the load situation to avoid excessive pressure on the system during peak hours. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the structure of a database synchronization system in one embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of the steps of a database synchronization method in one embodiment of the present invention.

[0025] Figure 3 It is a partial schematic diagram of step S1 in one embodiment of the present invention.

[0026] FIG. 4( a ) is a partial schematic diagram of step S2 in one embodiment of the present invention.

[0027] FIG. 4( b ) is a partial schematic diagram of step S2 in another embodiment of the present invention.

[0028] Figure 5 It is a partial schematic diagram of step S2 in yet another embodiment of the present invention.

[0029] Figure 6 It is a schematic diagram of the steps of a database synchronization method in one embodiment of the present invention.

[0030] Figure 7 It is a schematic diagram of the principle structure of a database synchronization method in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0032] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0033] like Figure 1 As shown, an embodiment of the present invention provides a database synchronization system 100.

[0034] In one implementation, the database synchronization system 100 implements data synchronization between a source database and a target database according to a database synchronization method, thereby improving data synchronization efficiency and accuracy.

[0035] In a specific embodiment, the database synchronization method shown can be implemented with reference to any technical solution provided below.

[0036] The database synchronization system 100 refers to determining the data change information of the source database based on the monitoring information of the database transaction log, or the configuration information of the trigger or the feedback information of calling the API (Application Programming Interface); processing the data change information according to the data format defined by the target database to determine the corresponding processed change data; transmitting the processed change data to the target database based on network communication, or message middleware or file transfer; and updating the processed change data to the target database in response to receiving the processed change data.

[0037] The database synchronization system 100 includes a source database 11 for generating data to be synchronized.

[0038] Data to be synchronized refers to data that has been changed in the source database but has not yet been copied to the target database to maintain data consistency.

[0039] In one embodiment, the data to be synchronized includes at least one of creating a table, deleting a table, adding a record, deleting a record, and updating a record.

[0040] In one embodiment, the data to be synchronized of the source database is determined based on at least one of a transaction log, a change data capture table, a message queue, a trigger, and a tracking table of the source database.

[0041] The database synchronization system 100 includes a processing module 12 .

[0042] In one embodiment, the processing module 12 is used to parse the data to be synchronized and determine the corresponding synchronization data field.

[0043] In one embodiment, the processing module 12 is used to group the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized, and determine a number of corresponding request bodies according to each group of synchronization data fields.

[0044] The database synchronization system 100 includes a sending module 13 .

[0045] In one embodiment, the sending module 13 is used to generate a corresponding data synchronization request according to each request body.

[0046] In one embodiment, the sending module 13 is used to send the data synchronization request to the target database synchronously or asynchronously.

[0047] The database synchronization system 100 includes a target database 14 .

[0048] In one embodiment, the target database 14 is used to respond to the data synchronization request and synchronize the to-be-synchronized data to the target database 14 according to the data synchronization request.

[0049] like Figure 2 As shown, an embodiment of the present invention provides a database synchronization method.

[0050] The database synchronization method is applied to a database synchronization system.

[0051] The database synchronization system includes a source database and a target database. The source database is used to generate data to be synchronized, and the target database is used to receive the data to be synchronized and perform data synchronization operations accordingly.

[0052] In one embodiment, the database synchronization system can be as follows Figure 1 The configuration is performed as described above, and the corresponding technical solution is set and referenced in the generation method provided by the present invention. Of course, the database synchronization system applied by the database synchronization method provided by the present invention is not limited to this configuration structure.

[0053] like Figure 2 As shown, a database synchronization method provided in one embodiment of the present invention includes the following steps.

[0054] Step S1, obtaining and parsing the data to be synchronized in the source database, and determining the corresponding synchronization data fields;

[0055] Step S2, grouping the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized, and determining a number of corresponding request bodies according to each group of synchronization data fields;

[0056] Step S3: Generate a corresponding data synchronization request according to each request body, and send the data synchronization request to the target database synchronously or asynchronously.

[0057] In this way, by acquiring and parsing the data to be synchronized in the source database and determining the corresponding synchronization data fields, unnecessary data transmission and processing can be reduced, and the accuracy and efficiency of synchronization can be improved; at the same time, according to the cumulative data volume and / or cumulative time of the data to be synchronized, the synchronization data fields generated by the analysis are grouped, and the request body is constructed according to the synchronization data fields of each group, which not only reduces the amount of data for a single synchronization and reduces the network transmission burden, but also allows the synchronization process to be carried out in parallel, thereby improving synchronization efficiency. In addition, by sending data synchronization requests to the target database synchronously or asynchronously, the sending strategy can be flexibly adjusted according to the load situation to avoid excessive pressure on the system during peak hours.

[0058] In one embodiment, the source database is MongoDB and the target database is ClickHouse.

[0059] MongoDB is a document-based database that stores data in the form of documents similar to JSON. MongoDB can be used to cope with large data volumes, high performance, and flexibility requirements. MongoDB uses collections to organize documents, and each document consists of key-value pairs. A collection in the database is similar to a table in a relational database, and a data record in a collection is similar to a row in a relational database, stored in BSON (Binary JSON) format.

[0060] MongoDB stores data as a document. The data structure consists of key-value pairs. Documents are similar to JSON objects. Field values ​​can contain other documents, arrays, and document data. In MongoDB, each document stored in a collection requires a unique _id field as the primary key. If the inserted document ignores the _id field, the MongoDB driver will automatically generate an _id for the field.

[0061] In one embodiment, the data to be synchronized is stored in a document format similar to JSON. In other words, the data to be synchronized may include several "key-value pairs", where the key is a string and the value can be a string, a number, a Boolean value, an array, an object or null.

[0062] In one embodiment, the data to be synchronized includes at least one of a non-nested object and a nested object.

[0063] A non-nested object is an object that does not contain other objects or arrays as its own properties (or fields, or member variables). In other words, a non-nested object contains basic data types (such as integers, floating-point numbers, strings, Boolean values, etc.) or value types as its properties. A nested object value is an object that contains one or more other objects as its own properties, and these inner objects can also continue to nest other objects.

[0064] In a specific embodiment, a non-nested object may be understood as a simple object that does not contain other objects or other arrays as values, and a nested object may be understood as an object that contains other objects or other arrays as values.

[0065] For example, a non-nested object is {key1:value1, key2:value2}, and a nested object is key1:[{key2:value2,key3:value3},{key4:value4}].

[0066] ClickHouse is an open source column-based database that uses a column-based storage format for online analytical processing. Using column-based storage means organizing by columns, and only involving some columns of the data table during the query process, so that the system can only scan and load necessary column data, thereby reducing I / O operations, helping to implement aggregate queries, and improving query speed.

[0067] Of course, the source database of the present invention is not limited to MongoDB, as long as it is a document-type database similar to MongoDB, such as CouchDB and CouchBase. Similarly, the target database is not limited to ClickHouse, as long as it is a database that meets the column storage format similar to ClickHouse, such as Greenplum, and the present invention does not make specific restrictions on this.

[0068] Synchronous sending in step S3 means that after sending the data synchronization request, the sender (source database) will wait until the target database synchronizes the request and returns a response result before continuing to perform subsequent operations. Asynchronous sending means that after sending the data synchronization request, the sender will not wait for the response of the target database, but will immediately continue to perform subsequent operations. You can choose whether to send synchronously or asynchronously according to the actual situation, and the present invention does not make specific restrictions on this.

[0069] In one embodiment, log information of a source database is obtained and parsed to determine corresponding data to be synchronized.

[0070] In a specific embodiment, before the data to be synchronized is a nested object, the method may further include the following steps.

[0071] Step P11, traverse the data objects containing key-value pairs in the data to be synchronized, and determine whether the corresponding value is of dictionary type;

[0072] If so, jump to step P12A to determine that the data to be synchronized contains nested objects;

[0073] If not, jump to step P12B to determine that the data to be synchronized is a non-nested object.

[0074] In this way, by detecting whether the value is of dictionary type to determine whether it is a nested object, it is easy to identify and process, and the processing logic is simplified.

[0075] Of course, the present invention does not necessarily make a judgment only through the dictionary type, and other structural types, such as arrays, lists, sets, tuples, etc. can also be used, and no specific restrictions are made on this.

[0076] In one embodiment, when the data to be synchronized is a nested object, flatten the nested object to determine the corresponding synchronization data fields, and the synchronization data fields include several key-value pairs, and the keys in the key-value pairs are formed by combining the parent keys and subkeys of the nested object.

[0077] In this way, by flattening the data to be synchronized containing nested structures, redundant data in the nested structures can be removed, and simple key-value pairs can be generated, making storage more efficient and the data structure clearer and easier to understand.

[0078] The flattening process refers to making the originally complex or multi-level things simpler and more direct by reducing or eliminating the hierarchical structure. The synchronization data fields refer to the data elements that are extracted from the source database, processed, and finally synchronized to the target database. It is the basic component of the database table and is used to store specific types of data.

[0079] In the above embodiment, these data elements can exist in the form of key-value pairs, where the key can be the name of the data field and the value is the data content corresponding to the name. For example, name: "Zhang xxx", the name of the synchronization data field is "name", and the corresponding synchronized data content is "Zhang xxx".

[0080] As Figure 3 shown, in a specific embodiment, the flattening of the nested object to determine the corresponding synchronization data fields includes:

[0081] Step S11, traverse all the sub-objects of the nested object, determine the key-value pairs at each nested level, combine the parent key and sub-key of each level of key-value pair into a new key, and the value corresponding to the lowest level of the nested object is the new value;

[0082] Step S12: forming a new key-value pair according to the new key and the new value, and updating the synchronization data field based on the new key-value pair.

[0083] In this way, by traversing all child objects of the nested object and combining the parent key and child key of each level key-value pair to form a new key-value pair, the conversion from a complex nested structure to a simple key-value pair set is achieved, which retains the relevance and integrity of the data while removing unnecessary nested levels, providing clear and easy-to-handle data fields for subsequent synchronization operations.

[0084] In a specific embodiment, a separator is used to connect the parent key and the child key to form a new key-value pair, and the separator includes "." or " / " or "_".

[0085] All sub-objects refer to all lower-level objects directly or indirectly contained by the current object in a nested object structure. In a nested data structure, an object may contain multiple sub-objects, and these sub-objects themselves may contain more sub-objects, forming a tree-like or graph-like structure.

[0086] In the process of converting nested objects into key-value pairs, the properties of each object (or sub-object) are considered as part of the key-value pair. The level refers to the depth or position of the object (or sub-object) in the nested structure. The subkey refers to the property name (or key) of the current object itself. In the flattening process, the subkey is used to identify the specific data item in the current key-value pair, and the parent key is used to identify the position or context of the current key-value pair in the original nested structure. If the current object is a sub-object, its parent key is the property name (or key) of the object that contains it.

[0087] For example, if key1: {key2: value2, key3: value3}, the key2 and key3 attributes in the key1 object are combined with their parent key key1 to form new key-value pairs key1.key2 and key1.key3.

[0088] In one embodiment, when the data to be synchronized is a non-nested object, the key and value of each key-value pair in the non-nested object are converted according to the target storage type defined by the target database, and updated to the synchronization data field.

[0089] For example, in the source database, there is a table named users, which contains records such as {_id:'3dccae44d0d05c0a97143080912d73b7',Amount:'50600.00',AgencyCode:'44463'}, where the value of Amount is a string type. The target database also has a corresponding table named users, and the type of the corresponding Amount field is a floating point type. When synchronizing the Amount field in the source database to the target database, the field needs to be converted to a floating point type.

[0090] As shown in FIG. 4( a ), in one embodiment, the method of the present invention may further include the following steps.

[0091] Step P211, detecting the cumulative amount of data to be synchronized stored in the source database within a unit time, and determining whether the cumulative amount of data meets a preset first threshold;

[0092] Based on this, step S2 may specifically include the following steps:

[0093] If yes, jump to step S2', and group the synchronization data fields according to the accumulated data volume of the data to be synchronized.

[0094] In this way, by grouping the data to be synchronized based on the accumulated data volume, the writing pressure on the target database can be reduced, and the performance bottleneck caused by synchronizing a large amount of data at one time can be avoided.

[0095] As shown in FIG. 4( b ), in one embodiment, the method of the present invention may further include the following steps.

[0096] Step P221, obtaining the historical time of the last completed data synchronization, and determining whether the time difference between the current time and the historical time is greater than or equal to a preset second threshold;

[0097] Based on this, step S2 may specifically include the following steps:

[0098] If yes, jump to step S2", and group the synchronization data fields according to the time difference.

[0099] In this way, by grouping the data to be synchronized based on the accumulated time and implementing periodic triggering of synchronization tasks, it is possible to ensure that the data in the target database remains up to date. In addition, the frequency of data synchronization can be flexibly controlled by adjusting the second threshold.

[0100] The cumulative data volume value is the total amount of data newly added to the source database or to be synchronized to the target database within a certain period of time. The total amount of data includes at least one of the recorded data and the data volume.

[0101] The cumulative time refers to the length of time from the last successful completion of the data synchronization operation to the current time point, and the time length is used to measure the timeliness and frequency of data synchronization. In other words, even if the accumulated data volume in the source database does not reach the preset first threshold, in order to ensure the timeliness and freshness of the data, the synchronization operation will be triggered according to the cumulative time.

[0102] In one embodiment, the above two embodiments can be combined, that is, the synchronization data fields are grouped according to the accumulated data volume and accumulated time of the data to be synchronized. Step S2' and step S2" are derived steps of step S2.

[0103] like Figure 5 As shown, in one embodiment, the step S2 of generating a corresponding data synchronization request according to each request body may specifically include the following steps.

[0104] Step S21, obtaining and parsing the log information of the source database, and determining the request parameters, wherein the request parameters include at least one of the address, port, authentication information and synchronization operation type of the target database;

[0105] Step S21: Generate a corresponding data synchronization request according to the request parameters and the request body.

[0106] In this way, parsing the log information of the source database helps to identify the data that needs to be synchronized and avoid unnecessary data transmission and repeated processing.

[0107] The target database address refers to the network address used to connect and manage the database, and the network address includes an IP address or a host name, which is used to locate the location of the database server in the network. The database port refers to a specific network port used to connect to the database server. The authentication information is data used to verify the identity of the user, ensuring that only authorized users can access the database. The authentication information includes a user name and password, and at least one of the identity authentication methods.

[0108] In a specific embodiment, log information of a source database is obtained and parsed to determine data synchronization information in a key-value pair format. The data synchronization information includes at least one of an operation type (abbreviated as OP), a database and collection name (abbreviated as NS), an operation execution time (abbreviated as Wall), and specific business data (abbreviated as O).

[0109] In a specific embodiment, the synchronization operation type OP includes at least one of an insert operation 'i', a delete operation 'd', an update operation 'u' and a data definition language operation 'c', wherein the data definition language (DDL) operation is used to operate databases, tables, columns, etc.

[0110] Database and collection name NS indicates the name of the database and collection (or table) where the operation occurs. For databases that do not support the concept of collection (such as relational databases), it may only contain the database name.

[0111] The operation execution time Wall refers to the timestamp or execution time recorded in the log file. Based on the operation execution time and the completion time of the last synchronization operation, the data to be synchronized in the source database is determined.

[0112] Business data O refers to the specific data content involved in the operation, and the corresponding business data is determined according to the operation type. Specifically, if the operation type is an insert operation or an update operation, the business data includes the record to be inserted or updated; if the operation type is a delete operation, the business data includes the identifier of the deleted record; if the operation type is a DDL operation, the business data includes the specific parameters or description information of the operation.

[0113] For example, parsing log information to determine operation events is as follows. Of course, the log information is not limited to one operation event, but can include multiple operation events of different synchronous operation types at the same time:

[0114] operation_event = {

[0115] "OP":"i",

[0116] "NS":"database_name.collection_name",

[0117] "Wall":"2023-01-15T12:00:00Z",

[0118] "O":{"name":"John Doe","age":30}

[0119] }.

[0120] In one embodiment, all key-value pairs in each group of synchronization data fields are converted into request data in a preset format, and the corresponding request body is determined according to the request data.

[0121] In a specific embodiment, the preset format includes at least one of CSV and XML. Thus, by converting the data into a unified format (such as CSV or XML), data standardization can be achieved, which is convenient for subsequent processing, analysis and exchange. At the same time, formats such as CSV and XML have good readability.

[0122] Specifically, the synchronization data fields are converted into csv format data separated by delimiters (such as commas). When the number of synchronization data fields (such as the number of rows) reaches a preset first threshold (such as 100,000) or the cumulative time reaches a second threshold (such as 5 seconds), the synchronization data fields are grouped and each group of sub-synchronization data fields is sent to the target database.

[0123] In a specific embodiment, based on an HTTP (HyperText Transfer Protocol) data request, a corresponding data synchronization request is generated according to the request parameters and the request body.

[0124] Specifically, determine the request method (such as GET or POST), the address of the target database (such as http: / / api.example.com) and the specific resource path (such as / users / 123); determine the request parameters according to the request method, and set the request header; determine the corresponding request body according to the format type defined in the request header (such as Content-Type) and each group of request data; generate a corresponding data synchronization request according to each request body, and send the data synchronization request to the target database synchronously or asynchronously.

[0125] like Figure 6 As shown, a database synchronization method provided in one embodiment of the present invention includes the following steps.

[0126] Step M1, in response to a first data synchronization request from a source database, determining a first synchronization data field and a first synchronization operation type corresponding to synchronization, wherein the first data synchronization request is determined based on any one of the database synchronization methods;

[0127] Step M2: synchronize the first synchronization data field to the target database according to the first synchronization operation type.

[0128] In this way, by responding to the first data synchronization request of the source database, the first synchronization data field and the first synchronization operation type to be synchronized are determined to ensure the targeted data synchronization, avoid unnecessary data transmission and processing, and synchronize the first synchronization data field to the target database according to the first synchronization operation type to ensure the integrity and consistency of the data.

[0129] In one embodiment, the target database can respond to several data synchronization requests from the source database and determine several groups of synchronization data fields and corresponding synchronization operation types respectively; according to each group of synchronization operation types, the corresponding synchronization data fields are synchronized to the target database synchronously or asynchronously.

[0130] The above-mentioned various implementation modes, embodiments or specific examples provided by the present invention can be combined with each other to finally form multiple better implementation modes.

[0131] Figure 7 A schematic diagram showing the principle structure of a database synchronization method in a preferred embodiment is shown below. Figure 7 , summarizing the processing process of this preferred embodiment.

[0132] Obtain and parse the log information of MongoDB (i.e., the source database) to determine the data to be synchronized; parse and process the data to be synchronized to obtain the corresponding synchronization data field information; based on the small batch aggregation method, group the synchronization data fields, and generate a corresponding data synchronization request according to each group of synchronization data fields, and send the data synchronization request to ClickHouse (i.e., the target database) synchronously or asynchronously.

[0133] Specifically, the data to be synchronized in MongoDB exists in a complex structure, such as nested JSON documents or arrays. These complex structures may face difficulties in data understanding and increased processing complexity when synchronized directly. Based on this, parsing these data into simple key-value pair fields (synchronization data fields) makes the data easier to understand and process, and can also speed up subsequent data synchronization.

[0134] After conversion to simple synchronized data fields, data not only becomes more intuitive and easier to manage, but also lays the foundation for subsequent storage and query. Especially when synchronized data fields are used for columnar storage, such as columnar databases such as ClickHouse, data storage and query performance will be significantly improved. This is because simple synchronized data fields are easier to store in columns, and databases based on columnar storage are easy to implement aggregate queries.

[0135] One embodiment of the present invention provides a computer-readable storage medium.

[0136] In one embodiment, a computer readable storage medium stores a computer program executed by the processor mentioned above, or a database synchronization method in any of the technical solutions mentioned above.

[0137] When the processor executes the computer program, it can execute the description of the database synchronization method in any of the above technical solutions, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0138] The computer-readable storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0139] In summary, the database synchronization method, system and computer storage medium provided by the present invention. By acquiring and parsing the data to be synchronized of the source database and determining the corresponding synchronization data fields, unnecessary data transmission and processing can be reduced, and the accuracy and efficiency of synchronization can be improved; at the same time, according to the cumulative data volume and / or cumulative time of the data to be synchronized, the synchronization data fields generated by the analysis are grouped, and the request body is constructed according to the synchronization data fields of each group, which not only reduces the amount of data for a single synchronization and reduces the network transmission burden, but also allows the synchronization process to be carried out in parallel, thereby improving synchronization efficiency. In addition, by sending data synchronization requests to the target database synchronously or asynchronously, the sending strategy can be flexibly adjusted according to the load conditions to avoid excessive pressure on the system during peak hours.

[0140] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0141] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A database synchronization method, characterized in that: include: Obtain and parse the data to be synchronized from the source database and determine the corresponding synchronization data fields; According to at least one of the accumulated data volume and accumulated time of the data to be synchronized, the synchronization data fields are grouped, and a number of corresponding request bodies are determined according to each group of synchronization data fields; A corresponding data synchronization request is generated according to each request body, and the data synchronization request is sent to the target database synchronously or asynchronously.

2. The database synchronization method according to claim 1, characterized in that: The step of parsing the data to be synchronized in the source database and determining the corresponding synchronization data fields includes: When the data to be synchronized is a nested object, the nested object is flattened to determine a corresponding synchronization data field, the synchronization data field includes a plurality of key-value pairs, and the key in the key-value pair is formed based on a combination of a parent key and a child key of the nested object.

3. The database synchronization method according to claim 2, characterized in that: The flattening of the nested objects to determine the corresponding synchronization data fields includes: Traversing all child objects of the nested object, determining the key-value pairs of each nested level, combining the parent key and the child key of each level key-value pair into a new key, and the value corresponding to the lowest level of the nested object is the new value; A new key-value pair is formed by concatenating the new key and the new value, and a synchronization data field is updated based on the new key-value pair.

4. The database synchronization method according to claim 2, characterized in that: Before the data to be synchronized is a nested object, the method includes: Traverse the data objects containing key-value pairs in the data to be synchronized, and determine whether the corresponding values ​​are of dictionary type; If so, it is determined that the data to be synchronized contains a nested object; If not, it is determined that the data to be synchronized includes non-nested objects.

5. The database synchronization method according to claim 1, characterized in that: The step of parsing the data to be synchronized in the source database and determining the corresponding synchronization data fields includes: When the data to be synchronized is a non-nested object, the key and value of each key-value pair in the non-nested object are converted according to the target storage type defined by the target database, and updated to the synchronization data field.

6. The database synchronization method according to claim 1, characterized in that: Before at least one of the accumulated data volume and accumulated time of the data to be synchronized, the method includes: Detecting the cumulative amount of data to be synchronized stored in the source database within a unit time, and determining whether the cumulative amount of data meets a preset first threshold; The grouping of the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized comprises: If so, the synchronization data fields are grouped according to the accumulated data volume of the data to be synchronized.

7. The database synchronization method according to claim 1, characterized in that: Before at least one of the accumulated data volume and accumulated time of the data to be synchronized, the method includes: Obtain the historical time of the last completed data synchronization, and determine whether the time difference between the current time and the historical time is greater than or equal to a preset second threshold; The grouping of the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized comprises: If so, the synchronization data fields are grouped according to the time difference.

8. The database synchronization method according to claim 1, characterized in that: Generating a corresponding data synchronization request according to each request body respectively includes: Obtain and parse the log information of the source database to determine the request parameters, where the request parameters include at least one of the address, port, authentication information, and synchronization operation type of the target database; Generate a corresponding data synchronization request according to the request parameters and the request body.

9. The database synchronization method according to claim 1, characterized in that: Determining a number of corresponding request bodies according to each group of synchronization data fields includes: All key-value pairs in each group of synchronization data fields are converted into request data in a preset format, and a corresponding request body is determined according to the request data, wherein the preset format includes at least one of CSV and XML.

10. The database synchronization method according to claim 1, characterized in that: The source database is MongoDB, and the target database is ClickHouse.

11. A database synchronization method, characterized in that: The method is applied to the target database and includes: In response to a first data synchronization request from a source database, determining a first synchronization data field and a first synchronization operation type corresponding to a to-be-synchronized data field, wherein the first data synchronization request is determined based on the database synchronization method according to any one of claims 1 to 10; The first synchronization data field is synchronized to the target database according to the first synchronization operation type.

12. A database synchronization system, characterized in that: include: Source database, used to generate data to be synchronized; A processing module, used to parse the data to be synchronized and determine the corresponding synchronization data fields; used to group the synchronization data fields according to at least one of the accumulated data volume and accumulated time of the data to be synchronized, and determine a number of corresponding request bodies according to each group of synchronization data fields; A sending module, used to generate a corresponding data synchronization request according to each request body, and send the data synchronization request to the target database synchronously or asynchronously; The target database is used to respond to the data synchronization request and synchronize the data to be synchronized to the target database according to the data synchronization request.

13. A computer storage medium comprising: at least one processor; A memory storing a computer program executable on the processor, wherein the processor executes the steps of the database synchronization method according to claims 1 to 10 when executing the program.

Images