Database data synchronization method and device

Abstract

The present invention discloses a database data synchronization method and device, which relates to the field of data processing technology. A specific implementation of the method includes: in response to a selection operation on a source table in a source database, obtaining first connection information of the source table; in response to a selection operation on a target table in a target database, obtaining second connection information of the target table; writing a synchronization configuration file based on the first connection information, the second connection information, and a preset virtual field name; through the synchronization configuration file, blending each row of data in the source table into a text data form, and setting the preset virtual field name as the field name of each text data; reading each text data from the source table, and in response to the field name of each text data being the preset virtual field name, converting each text data into row data in a preset format, and inserting the row data in the preset format under the preset virtual field name of the target table. This implementation introduces the concept of virtual fields to synchronize all fields of the source table to the target table.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a database data synchronization method and device. Background Art

[0002] In the big data world, data synchronization refers to the process of copying data from one source database to another, ensuring data consistency and real-time performance across different databases. The primary purpose of data synchronization is to aggregate data sources from various locations into a data warehouse for unified management, related computations, and ultimately, the production of valuable business data.

[0003] MongoDB, an open source document-based database management system, is a type of NoSQL database. It uses file storage and stores data in the BSON (Binary JSON) format similar to JSON. It has high performance, high scalability and a flexible data model.

[0004] However, due to MongoDB's flexible data model, it is difficult to map all fields to the data warehouse. Often, the data of the fields required by the business is synchronized. As a result, different businesses using different fields will synchronize the same MongoDB data multiple times, which not only brings query pressure to the MongoDB database, but also affects business development efficiency to a certain extent. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a database data synchronization method and apparatus, which can at least solve the problem in the prior art that a MongoDB database cannot synchronize all fields to a data warehouse at one time.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for synchronizing database data is provided, comprising:

[0007] In response to a selection operation on a source table in a source database, obtaining first connection information of the source table; and in response to a selection operation on a target table in a target database, obtaining second connection information of the target table;

[0008] Compile a synchronization configuration file according to the first connection information, the second connection information and the preset virtual field name;

[0009] By using the synchronization configuration file, each row of data in the source table is blended into a text data form, and the preset virtual field name is set as the field name of each text data; and

[0010] Each text data is read from the source table, and in response to the field name of each text data being the preset virtual field name, each text data is converted into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table.

[0011] Optionally, the first connection information includes a user name and password for logging into the source database, a connection address for connecting to the source database, a database name of the source database, and a collection name of a source table;

[0012] The second connection information includes the database name of the target database and the table name of the target table.

[0013] Optionally, the step of blending each row of data in the source table into text data format includes:

[0014] For each row of data, a key-value pair is constructed using the column field name as the key name and the column value as the key value. The key-value pairs of all columns are combined to form text data of the preset type.

[0015] Optionally, each row of data includes multiple columns of data, and each column of data corresponds to a different field name;

[0016] The step of blending each row of data in the source table into a text data format further includes: determining whether there is target column data with a field name of a row identifier, and if so, not processing the target column data;

[0017] The method further includes: reading target column data under the row identifier field name from the source table, and storing the target column data under the identifier field name of the target table.

[0018] Optionally, the method further includes:

[0019] Receive a query data request, determine a data field in the query data request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

[0020] To achieve the above object, according to another aspect of an embodiment of the present invention, a database data synchronization device is provided, comprising:

[0021] A selection module is configured to obtain first connection information of a source table in response to a selection operation on a source table in a source database; and obtain second connection information of a target table in response to a selection operation on a target table in a target database;

[0022] A writing module, configured to write a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name;

[0023] a blending module, configured to blend each row of data in the source table into a text data format using the synchronization configuration file, and set the preset virtual field name as the field name of each text data; and

[0024] The synchronization module is used to read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table.

[0025] Optionally, the first connection information includes a user name and password for logging into the source database, a connection address for connecting to the source database, a database name of the source database, and a collection name of a source table;

[0026] The second connection information includes the database name of the target database and the table name of the target table.

[0027] Optionally, the blending module is used to:

[0028] For each row of data, a key-value pair is constructed using the column field name as the key name and the column value as the key value. The key-value pairs of all columns are combined to form text data of the preset type.

[0029] Optionally, each row of data includes multiple columns of data, and each column of data corresponds to a different field name;

[0030] The blending module is further configured to: determine whether target column data with a field name of row identifier exists, and if so, not process the target column data;

[0031] The synchronization module is further configured to read target column data under a row identifier field name from a source table, and store the target column data under an identifier field name of a target table.

[0032] Optionally, the device further includes an acquisition module, configured to:

[0033] Receive a query data request, determine a data field in the query data request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

[0034] To achieve the above objective, according to another aspect of an embodiment of the present invention, a database data synchronization electronic device is provided.

[0035] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned database data synchronization methods.

[0036] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned database data synchronization methods is implemented.

[0037] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product according to an embodiment of the present invention includes a computer program, which, when executed by a processor, implements the database data synchronization method provided by an embodiment of the present invention.

[0038] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: by modifying the source code of the Flinkx data synchronization tool and introducing the concept of virtual fields, during the configuration phase, it is only necessary to write a synchronization configuration file treating the virtual fields as ordinary fields. During the data synchronization operation phase, the entire row of document content in the MongoDB collection is converted into JSON data, and the fields are set as virtual fields for subsequent synchronization operations, thereby achieving the purpose of synchronizing all MongoDB field data to the data warehouse at one time.

[0039] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0041] Figure 1 This is a diagram of the data synchronization architecture;

[0042] Figure 2 This is a schematic diagram of the main process of a database data synchronization method according to an embodiment of the present invention;

[0043] Figure 3 is a flowchart of a specific database data synchronization method according to an embodiment of the present invention;

[0044] Figure 4 is a flowchart of an optional database data synchronization method according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of main modules of a database data synchronization device according to an embodiment of the present invention;

[0046] Figure 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0047] Figure 7 It is a schematic diagram of the structure of a computer system of a mobile device or server suitable for implementing the embodiments of the present invention. DETAILED DESCRIPTION

[0048] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. In the technical solution of the present invention, the collection, analysis, use, transmission, storage and other aspects of user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, are not shared, disclosed or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken for user personal information to prevent illegal access to such personal information data, ensure that persons with access to personal information data comply with the provisions of relevant laws and regulations, and ensure the security of user personal information.

[0050] Once such personal information is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting the data. Where applicable, including in certain relevant applications, protect user privacy by de-identifying the data, for example, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of the data stored (e.g., collecting location data at the city level rather than at the specific address level), controlling how the data is stored, and / or other methods of de-identification.

[0051] It should be noted that this solution is designed specifically for MongoDB. MongoDB is a popular non-relational database system, renowned for its flexible data model, high performance, and scalability. MongoDB's flexibility stems from its schema-less data model, which allows it to store virtually any data format. Its high performance is due to its in-memory storage and data locality, making data retrieval and writing faster. Its scalability comes from its distributed architecture, which allows for the simple addition of more machines to store more data.

[0052] As shown below, the monitoring data of a certain computer room is stored in MongoDB. It can be seen that the same field in the same collection in MongoDB can be an array, a dictionary object, or a specific value. Data storage and use are very flexible, and it is not required that the fields in each row of data remain consistent. This is also a feature of all NoSQL databases.

[0053] ("_id":1,"name":"192.168.0.201","type":"CPU","max":"100%","min":"5%","usage":["3.2Ghz","2.8Ghz","2.5Ghz","4.4Ghz"])

[0054] ("_id":2,"name":"192.168.0.202","type":"Memory","free":"2GB","usage":["shared":"1GB","cache":"1GB])

[0055] ("_id":3,"name":"192.168.0.203","type":"disk","total":"2048GB","usage":"1024GB")

[0056] Data warehouses can store large amounts of historical data and handle complex analytical tasks such as query, join, and aggregation operations, with good data consistency and integrity. They synchronize different data sources, including MongoDB, MySQL, Oracle, SqlServer, and other databases from different business sources, into a unified data warehouse, and then conduct comprehensive data analysis and reporting in a unified place. This not only improves data utilization, but also allows different business tables to be linked to explore multi-dimensional business value. Figure 1 shown.

[0057] This solution only considers field synchronization. A field is a collection of columns in a table in the database, similar to selecting all columns in a column in Excel. Data, on the other hand, is a collection of all rows in a table in the database. Each row includes many fields and columns, similar to selecting all rows and columns in Excel. Different businesses generally only need to use some columns from certain tables, not all of them.

[0058] Currently, data in Oracle, MySQL, and SQL Server is stored in rows and columns, which is very standardized. The traditional approach of synchronizing "full-field data" is relatively mature and simple. This approach synchronizes the entire row and column to the data warehouse, allowing users to select the columns they need in the data warehouse. This allows queries to be made only in the data warehouse, without querying the original database, and without affecting the original database. MongoDB, on the other hand, stores data in JSON, which is flexible and has structures such as arrays and dictionaries. However, traditional approaches do not support the synchronization of arrays and dictionaries, making it impossible to synchronize all rows and columns to the data warehouse. Therefore, only the columns required by the business side are synchronized to the data warehouse.

[0059] Currently, the common steps in the industry for synchronizing MongoDB data to a data warehouse are as follows: 1. Analyze which fields are needed based on business needs; 2. Write the synchronization configuration, the main task of which is to enumerate the required MongoDB fields and then write the conversion rules; 3. Use tools such as Flinkx (or SparkSQL) in conjunction with the synchronization configuration to run the extraction. Therefore, the synchronization configuration mainly needs to configure the following: 1. The source library table information and target library table information to be synchronized; 2. The source field information to be synchronized and the corresponding target field information; 3. From the source library table information, after some custom transformSql processing, batch insert into the target library table. Among them, the content in the configuration item job.content.reader.parameter.column, because the data structure of MongoDB is arbitrary and will increase and change with the business dynamics, rather than a relational two-dimensional structure with fixed rows and columns, it is necessary to write the required fields each time according to business needs.

[0060] However, the same field name in the same MongoDB collection can be a value, an array, or a dictionary object. Current industry solutions cannot synchronize data for this field, making it impossible to enumerate all fields. Using the aforementioned monitoring data as an example, the usage field can be an array or a dictionary value in MongoDB, but a field in the data warehouse can only be a single structure—an array, a dictionary, or a value. If it's defined as a value, arrays and dictionaries cannot be synchronized, making it impossible to enumerate all fields.

[0061] Furthermore, because different businesses require different fields, existing operations will result in multiple synchronizations. Using the aforementioned monitoring data as an example, for the "usage" column, 1) Business 1 requires the average of the maximum values ​​in the usage array for all CPUs in the computer room. Therefore, Business 1's synchronization task can only be configured for type = cpu , and transformSQL requires a filter condition to filter out cpu . 2) Business 2 requires the average cache memory usage of all memory in the computer room. Therefore, Business 2's synchronization task can only be configured for type = memory , and transformSQL requires a filter condition to filter out memory .

[0062] As can be seen from this, even if different business parties require the same fields, the content in the same fields may have different meanings and structures, and additional fields may be added. The only way to map the content in the usage column is to split it into specific columns based on the business. Therefore, both Business 1 and Business 2 need to synchronize the same MongoDB monitoring data table, which requires two configurations and two queries. Multiple query operations in the original MongoDB database put IO query pressure on the database and may also affect the normal insertion of monitoring data.

[0063] See also Figure 2 , which shows a main flow chart of a database data synchronization method provided by an embodiment of the present invention, including the following steps:

[0064] S201: In response to a selection operation on a source table in a source database, obtaining first connection information of the source table; and in response to a selection operation on a target table in a target database, obtaining second connection information of the target table;

[0065] S202: Compile a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name;

[0066] S203: Blending each row of data in the source table into a text data format using the synchronization configuration file, and setting the preset virtual field name as the field name of each text data;

[0067] S204: Read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table.

[0068] In the above implementation, MongoDB, an open-source document-based database management system, is a type of NoSQL database. It uses a file-based storage approach, storing data in the BSON (Binary JSON) format, which is similar to JSON. It offers high performance, high scalability, and a flexible data model. Compared to traditional relational databases, MongoDB performs better when processing large amounts of data and supports distributed architectures such as replica sets and sharding, enabling high data availability and horizontal scalability. Furthermore, MongoDB provides rich query capabilities and a powerful aggregation framework, making it suitable for a variety of application scenarios, including web applications, mobile applications, and log management.

[0069] The current practice is to enumerate the fields in a MongoDB collection in the configuration file. However, due to MongoDB's "schemaless" data model (the field structure of each row of data can be completely different), it is impossible to enumerate all fields. The typical practice is to synchronize only the fields required, which makes it impossible to synchronize all fields at once. Furthermore, different businesses require different fields, so this operation results in multiple synchronizations, which not only puts query pressure on the MongoDB database but also affects business development efficiency to a certain extent.

[0070] For steps S201 and S202, a visual interface and guidance method are set. The user clicks "Select MongoDB Library" to display all tables in MongoDB. The user then selects Table A and uses it as the source table. The user clicks "Select Data Warehouse" to display all tables in the data warehouse. The user then selects Table B or creates a new Table B and uses it as the target table.

[0071] First, obtain the first connection information for source table A, including the MongoDB database login username and password, the connection address for connecting to the MongoDB database, the MongoDB database name, other connection information, and the collection name of source table A. It should be noted that the collection name here is a conceptual term for the data structure in the MongoDB database, similar to a table name. This analogy between MongoDB and the more familiar MySQL concept is shown in Table 1:

[0072] Table 1

[0073] MysQL MongoDB surface gather OK document List Field

[0074] And, obtain the second connection information of the target table B, including the library name of the data warehouse and the table name of the target table B.

[0075] This solution modifies the Flinkx data synchronization tool source code to incorporate the concept of "virtual fields." During the configuration phase, you simply write synchronization configuration information for virtual fields, treating them like ordinary fields. A synchronization configuration file is generated based on the first and second connection information described above. The synchronization configuration information for source table A includes the following (only partial code is shown here):

[0076] "column": / / Field enumeration in MongoDB

[0077] "name":"_id","type":"string"

[0078] "name":"flinkBson","type":"string"

[0079] Since only the _id and flinkBson fields are considered here, all fields in source table A can be enumerated. Similarly, the synchronization configuration information for target table B includes only partial code shown here:

[0080] "column": / / Hive corresponding MongoDB field enumeration

[0081] "name":"id","type":"string"

[0082] "name":"flinkBson","type":"string"

[0083] It can be seen that the mapping relationship between source table A and target table B is set: source table A's _id -> target table B's id, source table A's flinkBson -> target table B's flinkBson.

[0084] In step S203, flinkBson advantageously includes all columns of source table A. Traditional methods, however, only capture business-related columns and cannot exhaustively capture all columns. For example, the usage and cpu fields are arrays, which cannot exhaust all fields. This solution uses a synchronized configuration file to convert each row of data in source table A into text data and sets flinkBson as the field name for each text data row.

[0085] Take the aforementioned MongoDB monitoring data as an example, assuming that it is all in source table A:

[0086] ("_id":1,"name":"192.168.0.201","type":"CPU","max":"100%","min":"5%","usage":["3.2Ghz","2.8Ghz","2.5Ghz","4.4Ghz"])

[0087] ("_id":2,"name":"192.168.0.202","type":"Memory","free":"2GB","usage":["shared":"1GB","cache":"1GB])

[0088] ("_id":3,"name":"192.168.0.203","type":"disk","total":"2048GB","usage":"1024GB")

[0089] Its storage form is shown in Table 2:

[0090] Table 2

[0091] _id name type usage max min free total

[0092] After processing the synchronization configuration file, the results are shown in Table 3:

[0093] Table 3

[0094]

[0095] From this we can see that the processing of each row of data includes two parts, in which the _id and the corresponding data are processed twice:

[0096] 1. Do not process _id (row identifier);

[0097] 2. Combine the entire row of data. This involves constructing key-value pairs using the column names as keys and the column values ​​as values. This combination of key-value pairs across all columns generates JSON text data. Because each column is stored in key-value format, the order of the key-value pairs within the text data is unrestricted and can be sorted arbitrarily.

[0098] For step S204, the Flinkx specification requires a one-to-one correspondence between the fields of source table A and target table B: _id of source table A -> id of target table B, and flinkBson of source table A -> flinkBson of target table B. Therefore, text data can be read from source table A one by one and stored in target table B one by one.

[0099] 1. If the data field name is _id, it will be directly stored under the id of the target table B;

[0100] 2. If the data field is in flinkBson format, it needs to be converted to JSON-formatted row data that meets the data warehouse storage requirements. The JSON-formatted row data is then inserted into the flinkBson field of target table B. In source table A, flinkBson combines each row data into a JSON text file, which is then presented as text / string in target table B.

[0101] The results are shown in Table 4:

[0102] Table 4

[0103]

[0104] All synchronization tasks first query source table A using TransformSQL, then insert the queried data into target table B to synchronize data between source table A and target table B. To prevent the impact of too many queries on source table A, this solution also uses batch transfer. For example, first query 100 data items and insert them into target table B, then query another 100 data items and insert them into target table B, and repeat this cycle.

[0105] In view of this, this solution can use the iterator included in the Flinkx tool to implement a loop reading text data from source table A and storing it in target table B. If the iterator loop ends, the entire data synchronization process ends.

[0106] The method provided in the preceding embodiment introduces the concept of virtual fields from a development perspective and incorporates the logic of virtual fields into the Flinkx source code. This ensures that no matter how complex the MongoDB data is, it is ultimately divided into only two fields: _id and flinkBson. This simplifies the configuration and enables the use of virtual fields to synchronize all MongoDB fields to the Hive data warehouse.

[0107] See also Figure 3 , shows a schematic flow chart of a specific database data synchronization method according to an embodiment of the present invention, including the following steps:

[0108] S301: In response to a selection operation on a source table in a source database, obtaining first connection information of the source table; and in response to a selection operation on a target table in a target database, obtaining second connection information of the target table;

[0109] S302: Compile a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name;

[0110] S303: judging, through the synchronization configuration file, whether there is target column data with a field name of row identifier; if so, not processing the target column data;

[0111] S304: For each row of data in the source table, a key-value pair is constructed using the column name as the key name and the column value as the key value, and the key-value pairs of all columns are combined to form text data of a preset type, and the preset virtual field name is set as the field name of each text data;

[0112] S305: Read the target column data under the row identifier field name from the source table, and store the target column data under the identifier field name of the target table;

[0113] S306: Read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table.

[0114] The method provided in the embodiment of the present invention modifies the source code of the Flinkx data synchronization tool and adds the concept of virtual fields, flinkBson. During the configuration phase, the virtual fields only need to be treated as ordinary fields in the synchronization configuration. During the data synchronization operation phase, the entire row of document content in the MongoDB collection is converted into JSON format data, and flinkBson is used as its field name for synchronization operations, thereby achieving the purpose of synchronizing all MongoDB field data to the data warehouse at one time.

[0115] See also Figure 4 , shows a schematic flow chart of an optional database data synchronization method according to an embodiment of the present invention, comprising the following steps:

[0116] S401: In response to a selection operation on a source table in a source database, obtaining first connection information of the source table; and in response to a selection operation on a target table in a target database, obtaining second connection information of the target table;

[0117] S402: Compile a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name;

[0118] S403: Blending each row of data in the source table into text data format using the synchronization configuration file, and setting the preset virtual field name as the field name of each text data;

[0119] S404: Read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field of the target table;

[0120] S405: Receive a data query request, determine a data field in the data query request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

[0121] In the above implementation, for steps S401-S404, since businesses generally require the use of the id field for associated queries, this solution modifies the source code of the Flinkx data synchronization tool and incorporates the concept of a virtual field, flinkBson. This ensures that even the most complex MongoDB data is ultimately divided into two fields: _id and flinkBson. Storing the id facilitates business processing and allows synchronization to proceed according to the existing synchronization logic. Specifically, the specific value of _id is obtained and passed to the subsequent insert step. For flinkBson, however, the entire row of data must be converted to JSON before being passed to the subsequent insert step.

[0122] In step S405, ETL stands for Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from the source to the destination. After full field synchronization, the data is stored in the Hive data warehouse as a "Text" field type.

[0123] When used in business ETL, you must use the Hive JSON parsing function "get_json_object". For example, if you need to use the "shared" field with the "Memory" type in the synchronized ods.test table, you can use the following syntax: select get_json_object(flinkBson, '$.usage.shared') as shared from ods.test where type = 'Memory'. This means inserting the data fields in the query request into get_json_object, such as flinkBson, usage, and shared.

[0124] This solution combines all MongoDB columns (including arrays, dictionaries, and values) into a single column for synchronization. However, if you need to read the field values ​​required by the business, these values ​​are encapsulated in FlinkBson and cannot be read directly. When users need to use this complex column, unlike the existing practice of having only one value per column per row, a special method is required to obtain the required column. This requires using Hive's native function, namely, get_json_object.

[0125] The method provided in the above embodiment can, after synchronizing all fields of the data in the source table of the source MongoDB database, use the data warehouse Hive function to read the field values ​​according to data usage, without the need to synchronize the source MongoDB database multiple times, thereby reducing the reading pressure.

[0126] The method provided in the embodiment of the present invention aims to synchronize all fields of MongoDB data to the Hive data warehouse. In addition to using the open source synchronization tool Flinkx, other tools can also be used. The entire solution introduces the concept of virtual fields from a development perspective, and combines the logic of virtual fields with the Flinkx source code. In terms of behavior dimension, for data in the same row, if the field is _id, the specific value of the field is obtained according to the original logic and then passed to the data warehouse. The entire row is blended into a text data. Later, when the identification field is flinkBson, the text data needs to be converted into JSON data and inserted into the data warehouse. After synchronization, the field value can be read using Hive functions according to data usage, without the need to synchronize the MongoDB database multiple times, thus reducing the reading pressure.

[0127] See also Figure 5 , which shows a schematic diagram of the main modules of a database data synchronization device 500 provided by an embodiment of the present invention, including:

[0128] The selection module 501 is configured to obtain first connection information of a source table in response to a selection operation on a source table in a source database; and obtain second connection information of a target table in response to a selection operation on a target table in a target database;

[0129] A writing module 502 is configured to write a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name;

[0130] A blending module 503 is configured to blend each row of data in the source table into a text data format using the synchronization configuration file, and set the preset virtual field name as the field name of each text data; and

[0131] The synchronization module 504 is used to read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table.

[0132] In the implementation device of the present invention, the first connection information includes a user name and password for logging into the source database, a connection address for connecting to the source database, a database name of the source database, and a collection name of a source table;

[0133] The second connection information includes the database name of the target database and the table name of the target table.

[0134] In the embodiment of the present invention, the mixing module 503 is used to:

[0135] For each row of data, a key-value pair is constructed using the column field name as the key name and the column value as the key value. The key-value pairs of all columns are combined to form text data of the preset type.

[0136] In the implementation device of the present invention, each row of data includes multiple columns of data, and each column of data corresponds to a different field name;

[0137] The blending module 503 is further configured to: determine whether there is target column data with a field name of row identifier, and if so, not process the target column data;

[0138] The synchronization module 504 is further configured to read target column data under a row identifier field name from the source table, and store the target column data under the identifier field name of the target table.

[0139] The implementation device of the present invention further includes an acquisition module, which is used to:

[0140] Receive a query data request, determine a data field in the query data request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

[0141] In addition, the specific implementation content of the device in the embodiment of the present invention has been described in detail in the above method, so the repeated content will not be described again here.

[0142] Figure 6 An exemplary system architecture 600 to which embodiments of the present invention may be applied is shown, including terminal devices 601 , 602 , 603 , a network 604 and a server 605 (only an example).

[0143] Terminal devices 601, 602, and 603 can be various electronic devices with display screens and support web browsing, and are installed with various communication client applications. Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc.

[0144] The network 604 is used to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0145] The server 605 may be a server that provides various services. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the device is generally set in the server 605 .

[0146] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0147] Reference below Figure 7 , which shows a schematic structural diagram of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0148] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0149] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0150] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0151] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided within a processor. For example, a processor may be described as comprising a selection module, a writing module, a blending module, and a synchronization module. The names of these modules do not, in some cases, limit the modules themselves. For example, a selection module may also be described as an "acquisition module."

[0154] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently and not incorporated into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to perform any of the above-described database data synchronization methods.

[0155] The computer program product of the present invention includes a computer program, and when the computer program is executed by a processor, the database data synchronization method in the embodiment of the present invention is implemented.

[0156] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A database data synchronization method, characterized in that: Applied to a non-relational database, the method includes: In response to a selection operation on a source table in a source database, obtaining first connection information of the source table; and in response to a selection operation on a target table in a target database, obtaining second connection information of the target table; Compile a synchronization configuration file according to the first connection information, the second connection information and the preset virtual field name; Determine, through the synchronization configuration file, whether there is target column data with a field name of a row identifier; if so, do not process the target column data; combine each row data in the source table into a text data format, and set the preset virtual field name as the field name of each text data; wherein each row data includes multiple column data, and each column data corresponds to a different field name; and Each text data is read from the source table, and in response to the field name of each text data being the preset virtual field name, each text data is converted into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table; the target column data located under the row identifier field name is read from the source table, and the target column data is stored in the identifier field name of the target table.

2. The method according to claim 1, characterized in that The first connection information includes a user name and password for logging into the source database, a connection address for connecting to the source database, a database name of the source database, and a collection name of a source table; The second connection information includes the database name of the target database and the table name of the target table.

3. The method according to claim 1, characterized in that The step of blending each row of data in the source table into text data format includes: For each row of data, a key-value pair is constructed using the column field name as the key name and the column value as the key value. The key-value pairs of all columns are combined to form text data of the preset type.

4. The method according to claim 1, wherein The method further comprises: Receive a query data request, determine a data field in the query data request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

5. A database data synchronization device, characterized in that: Applied to a non-relational database, the device includes: A selection module is configured to obtain first connection information of a source table in response to a selection operation on a source table in a source database; and obtain second connection information of a target table in response to a selection operation on a target table in a target database; A writing module, configured to write a synchronization configuration file according to the first connection information, the second connection information, and a preset virtual field name; a blending module, configured to determine, through the synchronization configuration file, whether there is target column data with a field name being a row identifier; if so, not process the target column data; blend each row data in the source table into a text data format, and set the preset virtual field name as the field name of each text data; wherein each row data includes multiple column data, and each column data corresponds to a different field name; and The synchronization module is used to read each text data from the source table, and in response to the field name of each text data being the preset virtual field name, convert each text data into row data in a preset format, so as to insert the row data in the preset format into the preset virtual field name of the target table; read the target column data located under the row identifier field name from the source table, and store the target column data into the identifier field name of the target table.

6. The device according to claim 5, characterized in that The first connection information includes a user name and password for logging into the source database, a connection address for connecting to the source database, a database name of the source database, and a collection name of a source table; The second connection information includes the database name of the target database and the table name of the target table.

7. The device according to claim 5, characterized in that The mixing module is used to: For each row of data, a key-value pair is constructed using the column field name as the key name and the column value as the key value. The key-value pairs of all columns are combined to form text data of the preset type.

8. The device according to claim 5, characterized in that The device further includes an acquisition module, configured to: Receive a query data request, determine a data field in the query data request, insert the data field into a parsing function to generate a structured query language SQL statement, and call the SQL statement to query and return data under the data field from a target table.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images