Database synchronization method and apparatus, computer device, storage medium, and product

Abstract

The application relates to the technical field of Internet, and provides a database synchronization method and device, computer equipment, a storage medium and a product, which comprises the following steps: receiving a synchronization request sent by a first database in a database cluster; if it is confirmed that the synchronization type in the synchronization request is a full-amount type, then full-amount synchronization is performed on the first database by a second database, so that the data table in the first database is consistent with the data table in the second database; if it is confirmed that the synchronization type in the synchronization request is an incremental type, then incremental synchronization is performed on the first database by the second database, so that the data table in the first database is consistent with the data table in the second database. The application meets different synchronization requirements of each database in the database cluster, avoids the situation that each database in the database cluster adopts full-amount synchronization, the data transmission pressure in the database cluster is too large, and the communication resources are nervous or even collapse.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a database synchronization method, apparatus, computer equipment, storage medium and product. Background Technology

[0002] A database cluster utilizes at least two or more database servers to form a virtual, single logical database image, providing transparent data services to clients like a single database system. Because each database in a database cluster needs to provide the same data services to different clients, there is a requirement for data synchronization between the databases in a current database cluster.

[0003] However, the inventors discovered that the current database clusters typically use full synchronization methods, which can lead to excessive data transmission pressure in the database cluster during data synchronization, causing the database cluster communication resources to become strained or even crash. Summary of the Invention

[0004] This application provides a database synchronization method, apparatus, computer equipment, storage medium, and product to solve the problem that excessive data transmission pressure in the database cluster during data synchronization can lead to insufficient communication resources or even crashes in the database cluster.

[0005] In a first aspect, this application provides a database synchronization method, which operates in a database cluster having two or more databases, the method comprising:

[0006] Receive a synchronization request sent by a first database in the database cluster; wherein the synchronization request is used to instruct data synchronization with a second database in the database cluster;

[0007] If it is confirmed that the synchronization type in the synchronization request is full, then the second database is invoked to perform a full synchronization of the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0008] If it is confirmed that the synchronization type in the synchronization request is incremental, then the second database is invoked to perform incremental synchronization with the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0009] In the above scheme, before receiving the synchronization request sent by the first database in the database cluster, the method further includes:

[0010] In the database cluster, a physical primary key and a last modified time are constructed in the database, and a management process is created to assign values ​​to the physical primary key and the last modified time. The physical primary key includes all primary key fields in the data table; the primary key field is a unique identifier for a data row in the data table; the last modified time represents the time when the data table was last modified; the management process responds to changes in the data table detected in the database; the management process is used to assign values ​​to the physical primary key of the data table according to the changed data table, thereby updating the primary key field in the physical primary key; the management process is also used to assign values ​​to the last modified time of the data table according to the time of the data table change.

[0011] In the above scheme, before calling the second database to perform a full synchronization of the first database, the method further includes:

[0012] A full control file is generated based on the data tables in the second database; wherein, the full control file is a file obtained by unloading data from the data tables in the second database; unloading data is the process of importing the data from the data tables into a text file.

[0013] In the above scheme, before calling the second database to perform incremental synchronization of the first database, the method further includes:

[0014] An incremental control file is generated based on the changed data rows in the data table of the second database; wherein, the incremental control file is a file obtained by unloading the changed data rows in the data table of the second database within a target time period; the target time period is the time period from the last time when the second database and the first database were fully synchronized or incrementally synchronized to the time when the synchronization request is received; unloading is the process of importing the data in the data table into a text file.

[0015] In the above scheme, generating the incremental control file based on the changed data rows in the data table of the second database includes:

[0016] Obtain the table structure, physical primary key, and last modification time of the data table in the second database, and generate a structure file based on the table structure and the physical primary key; wherein, the table structure defines the file name and configuration information of the table fields; the physical primary key includes all primary key fields in the data table; the primary key field is a unique identifier for a data row in the data table; the structure file includes: the table name, a list of primary keys with at least one primary key field, a list of tables with at least one table field, and the total number of table fields in the table list; the last modification time represents the time when the data table in the second database was last modified;

[0017] If it is determined that the last modification time is within the target time period, then the change records in the second database whose modification time is within the target time period are obtained. Based on the primary key list and the change records, the data rows in the second database that have changed are identified, and the data rows that have changed are summarized to obtain the change file.

[0018] Based on at least one table field in the table field list, a data row of the change file is disassembled to obtain at least one cell string corresponding to the table field. The cell strings are arranged sequentially, and adjacent cell strings are separated by a preset field separator to obtain the row string of the data row. The row strings of all data rows in the change file are summarized to obtain the data file.

[0019] The information content of the structure file is calculated to obtain the structure information content, and the information content of the data file is calculated to obtain the data information content. The structure file, the data file, the structure information content, the data information content, the field separator, and the data unloading mode are summarized to obtain the incremental control file; wherein, the data unloading mode represents the synchronization type between the first database and the second database.

[0020] In the above scheme, the step of calling the second database to perform a full synchronization of the first database includes:

[0021] Clear the data tables in the first database and send the full control file from the second database to the first database;

[0022] Based on the field separator in the full control file, the line strings in the full control file are decomposed to obtain at least one unit string; wherein, the line string records the unit strings of the data rows that have changed in the second database, and the unit strings are the data in the data rows that have changed that correspond to the table fields;

[0023] Based on the order of the table fields in the table field list in the full control file and the arrangement order of each unit string in a row string, the table fields are matched one-to-one with the unit strings to obtain a full row of a row string. The full rows of all row strings in the full control file are summarized to obtain a full table.

[0024] The full table is entered into the data table of the first database, so that the content of the data table of the first database is consistent with the content of the data table of the second database.

[0025] In the above scheme, the step of calling the second database to perform incremental synchronization of the first database includes:

[0026] Extract the incremental control file from the second database and send the incremental control file to the first database;

[0027] Based on the field separator in the incremental control file, the row string in the incremental control file is decomposed to obtain at least one unit string; wherein, the row string records the unit string of the data row that has changed in the second database, and the unit string is the data in the data row that has changed that corresponds to the table field;

[0028] Based on the order of the table fields in the list of table fields in the incremental control file, and the order of the unit strings in a row string, the table fields are matched one-to-one with the unit strings to obtain an incremental row of a row string. The incremental rows of all row strings in the incremental control file are summarized to obtain an incremental table.

[0029] Adjust the data rows in the data table of the first database according to the incremental table, so that the adjusted data table is consistent with the data table in the second database.

[0030] In the above scheme, adjusting the data rows in the data table of the first database according to the incremental table includes:

[0031] Extract the increment type of the increment row in the increment table; wherein, the increment type defines the change of the increment row in the data table of the second database; the increment type includes: deletion type, addition type, and change type;

[0032] Set one incremental row in the incremental table as the target row;

[0033] If the incremental type of the target row is confirmed to be deletion, then delete the data row in the data table of the second database that is consistent with the target row;

[0034] If it is confirmed that the incremental type of the target row is a new row, then the target row is inserted into the data table of the second database;

[0035] If the incremental type of the target row is confirmed to be change type, then the data row in the data table of the second database whose primary key field is consistent with the target row is set as the row to be adjusted, and the target row replaces the row to be adjusted.

[0036] Secondly, this application provides a database synchronization device, installed in a database cluster having two or more databases, the device comprising:

[0037] An input module is configured to receive a synchronization request sent by a first database in the database cluster; wherein the synchronization request is used to instruct data synchronization with a second database in the database cluster.

[0038] The full synchronization module is used to call the second database to perform a full synchronization of the first database if it is confirmed that the synchronization type in the synchronization request is full, so that the data tables in the first database are consistent with the data tables in the second database.

[0039] The incremental synchronization module is used to call the second database to perform incremental synchronization with the first database if it is confirmed that the synchronization type in the synchronization request is incremental, so that the data tables in the first database are consistent with the data tables in the second database.

[0040] The above plan also includes:

[0041] A configuration module is used to construct physical primary keys and last modified times in the database within the database cluster, and to create a management process for assigning values ​​to the physical primary keys and the last modified times. The physical primary key includes all primary key fields in the data table; the primary key field is a unique identifier for a data row in the data table; the last modified time represents the time when the data table was last modified; the management process responds to when changes are detected in the data tables in the database; the management process is used to assign values ​​to the physical primary keys of the data tables based on the changed data tables, thereby updating the primary key fields in the physical primary keys; the management process is also used to assign values ​​to the last modified times of the data tables based on the time of the data table changes.

[0042] Thirdly, this application provides a computer device, including: a processor and a memory communicatively connected to the processor;

[0043] The memory stores computer-executed instructions;

[0044] The processor executes computer execution instructions stored in the memory to implement the database synchronization method as described above.

[0045] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the aforementioned database synchronization method.

[0046] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned database synchronization method.

[0047] This application provides a database synchronization method, apparatus, computer equipment, storage medium, and product. By automatically calling a second database to perform full or incremental synchronization on a first database according to a synchronization request, the technical effect of data synchronization between the second and first databases is achieved. This satisfies the different synchronization needs of each database in a database cluster and avoids the situation where each database in the database cluster uses full synchronization, which would lead to excessive data transmission pressure in the database cluster, causing communication resource shortages or even crashes. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0050] Figure 2 A flowchart of Embodiment 1 of a database synchronization method provided in this application;

[0051] Figure 3 A flowchart illustrating Embodiment 2 of a database synchronization method provided in this application;

[0052] Figure 4 A schematic diagram of the program modules of a database synchronization device provided by the present invention;

[0053] Figure 5 This is a schematic diagram of the hardware structure of the computer device in the computer device of the present invention.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] Please see Figure 1 The specific application scenario of this application is as follows:

[0057] A database synchronization method is implemented on a cluster server 2 within a database cluster 3 containing two or more databases. The cluster server 2 receives a synchronization request from the first database 4 in the database cluster 3. If the synchronization type in the synchronization request is confirmed to be full, the cluster server 2 calls the second database 5 to perform a full synchronization on the first database 4, making the data tables in the first database 4 consistent with the data tables in the second database 5. If the synchronization type in the synchronization request is confirmed to be incremental, the cluster server 2 calls the second database 5 to perform incremental synchronization on the first database 4, making the data tables in the first database 4 consistent with the data tables in the second database 5.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] Example 1:

[0060] Please see Figure 2 This application provides a database synchronization method, which operates in a database cluster with two or more databases. The method includes:

[0061] S101: Receive a synchronization request sent by the first database in the database cluster; wherein the synchronization request is used to indicate data synchronization with the second database in the database cluster.

[0062] In this example, the synchronization request includes:

[0063] The first identification information is a unique identifier for the first database;

[0064] The second identification information serves as a unique identifier for the second database;

[0065] Synchronization type refers to the synchronization method used to synchronize data between the first database request and the second database.

[0066] Synchronization requests are set as needed or triggered periodically to automatically synchronize databases in a database cluster in real time or periodically.

[0067] S102: If it is confirmed that the synchronization type in the synchronization request is full, then the second database is called to perform a full synchronization of the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0068] In a preferred embodiment, before invoking the second database to perform a full synchronization of the first database, the method further includes:

[0069] A full control file is generated based on the data tables in the second database; the full control file is the file obtained by unloading data from the data tables in the second database; unloading data is the process of importing data from the data tables into a text file.

[0070] In this example, by unpacking the data tables of the second database and extracting data rows from the tables, a full control file is generated. This ensures that the data in the full control file does not have a specific data structure, allowing the first database, which has a different format, to synchronize with the second database using this full control file. For example, the full control file is in XML format, and its filename format is {table name}_{date}_{time}_{uuid}.ctl.

[0071] Specifically, the table structure and physical primary key of the data table in the second database are obtained, and a structure file is generated based on the table structure and physical primary key. The table structure defines the file name of the data table and the configuration information of the table fields. The physical primary key includes all primary key fields in the data table. The primary key field is the unique identifier of the data row in the data table. The structure file includes: the table name of the data table, a list of primary keys with at least one primary key field, a list of tables with at least one table field, and the total number of table fields in the table list.

[0072] Based on at least one table field in the table field list, a data row in the change file is broken down to obtain at least one cell string corresponding to the table field. The cell strings are arranged sequentially, and adjacent cell strings are separated by a preset field separator (e.g., X) to obtain a row string for a data row. Since the obtained string does not have a data structure, it is convenient to synchronize databases with different data structures. The information content of the structure file is calculated to obtain the structure information content, and the information content of the data file is calculated to obtain the data information content. The structure file, data file, structure information content, data information content, field separator, and data unloading mode (ALL) are summarized to obtain the full control file. Among them, the data unloading mode represents the synchronization type between the first database and the second database.

[0073] In this example, the table structure defines the data table filename, determines which fields the data table contains, and specifies the field name, data type, and width of each field. The filename format for the structure file is {table name}_{date}_{time}_{uuid}.ddl.

[0074] In a preferred embodiment, invoking the second database to perform a full synchronization of the first database includes:

[0075] Clear the data tables in the first database and send the full control file from the second database to the first database;

[0076] Based on the field separators in the full control file, the row strings in the full control file are decomposed to obtain at least one unit string; wherein, the row string records the unit strings of the data rows that have changed in the second database, and the unit strings are the data in the data rows that have changed that correspond to the table fields;

[0077] Based on the order of the table fields in the table field list in the full control file, and the order of the unit strings in a row string, the table fields are matched one-to-one with the unit strings to obtain a full row of a row string. The full table is obtained by summarizing the full rows of all row strings in the full control file.

[0078] Enter the full data table into the data table of the first database, so that the content of the data table in the first database is consistent with the content of the data table in the second database.

[0079] In this example, identification is performed by disassembling the row strings in the full control file according to the field delimiters in the full control file to obtain at least one unit string, resulting in the individual unit strings that make up the row string. Each table field serves as metadata for each unit string and corresponds to each unit string. Therefore, by following the order of the table fields in the table field list in the full control file and the order of the unit strings in a row string, the table fields and unit strings are matched one-to-one, resulting in a full row of a row string. At this point, the full row has both table fields and unit strings. Therefore, the database only needs to process the full row according to the data structure of its data table.

[0080] Based on the order of the table fields in the table field list in the full control file and the arrangement order of the cell strings in the row strings, a full table is obtained by matching the table fields with the cell strings one-to-one. The full table is then entered into the data table of the first database, making the content of the data table in the first database consistent with the content of the data table in the second database. The cell strings are then entered into the corresponding positions of the table fields in this data table. This achieves the technical effect of loading all rows into the data table and realizes the technical effect of data synchronization between different databases.

[0081] Optionally, before calling the second database to perform a full synchronization of the first database, the method also includes:

[0082] The amount of structural verification information is obtained by calculating the structure file in the full control file, and the amount of data verification information is obtained by calculating the data file in the full control file.

[0083] If the amount of structure verification information is confirmed to be consistent with the amount of structure information in the full control file, and the amount of data verification information is consistent with the amount of data information in the full control file, it means that the full control file has not been tampered with and the data has not been lost. Therefore, the second database can be called to perform a full synchronization of the first database.

[0084] If the amount of structure verification information is inconsistent with the amount of structure information in the full control file, and / or the amount of data verification information is inconsistent with the amount of data information in the full control file, it indicates that the full control file has been tampered with or data has been lost. In this case, the call to the second database to perform full synchronization of the first database will be stopped.

[0085] S103: If it is confirmed that the synchronization type in the synchronization request is incremental, then the second database is called to perform incremental synchronization with the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0086] In a preferred embodiment, before invoking the second database to perform incremental synchronization of the first database, the method further includes:

[0087] An incremental control file is generated based on the changed data rows in the data table of the second database. The incremental control file is a file obtained by unloading the changed data rows in the data table of the second database within a target time period. The target time period is the time period from the last time that the second database and the first database were fully or incrementally synchronized to the time when the synchronization request was received. Unloading is the process of importing the data in the data table into a text file.

[0088] In this example, by unpacking the data tables of the second database and extracting data rows from the tables, an incremental control file is generated. This ensures that the data in the incremental control file does not have a specific data structure, allowing the first database, which has a different format, to synchronize with the second database using this incremental control file. For example, the incremental control file is in XML format, and its filename format is {table name}_{date}_{time}_{uuid}.ctl.

[0089] For example, the incremental control file is in XML format, and the file name format of the incremental control file is {table name}_{date}_{time}_{uuid}.ctl.

[0090] In a preferred embodiment, an incremental control file is generated based on the changed data rows in the data tables of the second database, including:

[0091] Obtain the table structure, physical primary key, and last modification time of the data tables in the second database, and generate a structure file based on the table structure and physical primary key. The table structure defines the data table's filename and table field configuration information; the physical primary key includes all primary key fields in the data table; the primary key field is the unique identifier of a data row in the data table; the structure file includes: the table name, a list of primary keys with at least one primary key field, a list of tables with at least one table field, and the total number of table fields in the table list; the last modification time represents the time when the data table in the second database was last modified.

[0092] If the last modification time is determined to be within the target time period, then the change records in the second database whose modification time is within the target time period are obtained. Based on the primary key list and change records, the data rows in the second database that have changed are identified, and the data rows that have changed are summarized to obtain the change file.

[0093] Based on at least one table field in the table field list, a data row in the change file is broken down to obtain at least one string corresponding to the table field. The cell strings are arranged in sequence, and adjacent cell strings are separated by a preset field separator to obtain the row string of a data row. The row strings of all data rows in the change file are summarized to obtain the data file.

[0094] The information content of the structure file is calculated to obtain the structure information content, and the information content of the data file is obtained to obtain the data information content. The structure file, data file, structure information content, data information content, field separator, and unloading mode (INCREASE) are summarized to obtain the incremental control file. Among them, the unloading mode represents the synchronization type between the first database and the second database.

[0095] In this example, the table structure defines the data table filename, determines which fields the data table contains, and specifies the field name, data type, and width of each field. The filename format for the structure file is {table name}_{date}_{time}_{uuid}.ddl.

[0096] If the last modification time falls within the target time period, it indicates that the data table in the second database has changed within the target time period, and change information is generated.

[0097] If the last time period is not within the target time period, it indicates that the data tables in the second database have not changed within the target time period, and synchronization information is generated.

[0098] Based on the change information, retrieve change records from the second database that fall within the target time period. A change record is a data row in a data table of the second database that has been changed, along with the content of the changes to that data row. The change content includes change information and change type. Change information is the content of the changed data row after the change. Change type represents the type of operation performed on the changed data row, including deletion, addition, and change. Change time is the time when the changed data row was modified.

[0099] In a preferred embodiment, the process of incrementally synchronizing the first database with the second database includes:

[0100] Extract the incremental control file from the second database and send the incremental control file to the first database;

[0101] Based on the field separator in the incremental control file, the row string in the incremental control file is decomposed to obtain at least one unit string; wherein, the row string records the unit string of the data row that has changed in the second database, and the unit string is the data in the data row that has changed that corresponds to the table field;

[0102] Based on the order of the table fields in the table field list in the incremental control file, and the order of the unit strings in a row string, the table fields are matched one-to-one with the unit strings to obtain an incremental row of a row string. The incremental rows of all unit strings in the incremental control file are summarized to obtain the incremental table.

[0103] Adjust the data rows in the data table of the first database according to the incremental table, so that the adjusted data table is consistent with the data table of the second database.

[0104] Preferably, adjusting data rows in the data tables of the first database according to the incremental table includes:

[0105] Extract the increment type of the increment row in the increment table; where increment type defines the changes of the increment row in the data table of the second database; increment type includes: deletion type, addition type, and change type;

[0106] Set one incremental row in the incremental table as the target row;

[0107] If the incremental type of the target row is confirmed to be deletion, then delete the data row in the data table of the second database that is consistent with the target row;

[0108] If the incremental type of the target row is confirmed to be new, then the target row is inserted into the data table of the second database;

[0109] If the incremental type of the target row is confirmed to be change type, then the data row in the second database table whose primary key field is the same as the target row is set as the row to be adjusted, and the target row replaces the row to be adjusted.

[0110] For example, the increment type is represented by "+", "-", and " / "; where the deletion type is "-", so "-" is inserted at the beginning of the target line with the increment type of deletion to indicate that the increment type of the target line is deletion.

[0111] The new class is "+", so "+" is inserted at the beginning of the target row whose increment type is "new class" to indicate that the increment type of the target row is "new class".

[0112] The change type is " / ", therefore, " / " is inserted at the beginning of the target line whose increment type is change type to indicate that the increment type of the target line is change type.

[0113] Optionally, before invoking the second database to perform incremental synchronization with the first database, the method may also include:

[0114] The amount of structural verification information is obtained by calculating the structure file in the incremental control file, and the amount of data verification information is obtained by calculating the data file in the incremental control file.

[0115] If the amount of structural verification information is confirmed to be consistent with the amount of structural information in the incremental control file, and the amount of data verification information is consistent with the amount of data information in the incremental control file, it indicates that the incremental control file has not been tampered with and the data has not been lost. Therefore, the second database can be called to perform incremental synchronization of the first database.

[0116] If the amount of structural verification information is inconsistent with the amount of structural information in the incremental control file, and / or the amount of data verification information is inconsistent with the amount of data information in the incremental control file, it indicates that the incremental control file has been tampered with or the data has been lost. In this case, the incremental synchronization of the first database with the second database will be stopped.

[0117] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] Example 2:

[0119] Please see Figure 3 This application provides a database synchronization method, which operates in a database cluster with two or more databases. The method includes:

[0120] S201: Construct physical primary keys and last modified times in the database within the database cluster, and create a management process for assigning values ​​to the physical primary keys and last modified times. The physical primary key includes all primary key fields in the data table; the primary key field is the unique identifier of a data row in the data table; the last modified time represents the time when the data table was last modified; the management process responds when it detects changes to the data table in the database; the management process is used to assign values ​​to the physical primary keys of the data table based on the changed data table, thereby updating the primary key fields in the physical primary keys; the management process is also used to assign values ​​to the last modified times of the data table based on the time the data table was modified.

[0121] In this example, a physical primary key PkId is set to identify all primary key fields in the summary data table. Therefore, if you need to know the identity of a data row in the data table, you only need to use the physical primary key, without having to retrieve the data table one by one to check and identify it.

[0122] The Last Modified Time (UpdateTime) indicates when a data table has been modified. This allows the database to be synchronized based on the last modified time, improving the overall management efficiency of the database cluster and reducing the occurrence of frequent invalid synchronizations that could lead to resource waste.

[0123] By creating a management process for assigning values ​​to physical primary keys and last modified times, the database tables can be automatically monitored to identify changed primary key fields in a timely manner and record the time when the data table changes.

[0124] S202: Receive a synchronization request sent by the first database in the database cluster; wherein the synchronization request is used to indicate data synchronization with the second database in the database cluster.

[0125] This step is the same as S101 in Example 1, so it will not be described again here.

[0126] S203: If it is confirmed that the synchronization type in the synchronization request is full, then the second database is called to perform a full synchronization of the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0127] This step is the same as S102 in Example 1, so it will not be described again here.

[0128] S204: If it is confirmed that the synchronization type in the synchronization request is incremental, then the second database is called to perform incremental synchronization with the first database, so that the data tables in the first database are consistent with the data tables in the second database.

[0129] This step is the same as S103 in Example 1, so it will not be described again here.

[0130] Example 3:

[0131] Please see Figure 4 This application provides a database synchronization device 1, installed in a database cluster having two or more databases, comprising:

[0132] Input module 12 is used to receive a synchronization request sent by the first database in the database cluster; wherein the synchronization request is used to indicate data synchronization with the second database in the database cluster;

[0133] The full synchronization module 13 is used to call the second database to perform a full synchronization of the first database if the synchronization type in the synchronization request is confirmed to be full, so that the data tables in the first database are consistent with the data tables in the second database.

[0134] The incremental synchronization module 14 is used to call the second database to perform incremental synchronization with the first database if the synchronization type in the synchronization request is confirmed to be incremental, so that the data tables in the first database are consistent with the data tables in the second database.

[0135] Optionally, the database synchronization device 1 also includes:

[0136] Configuration module 11 is used to construct physical primary keys and last modified times in the database of the database cluster, and to create a management process for assigning values ​​to the physical primary keys and last modified times. The physical primary key includes all primary key fields in the data table; the primary key field is the unique identifier of a data row in the data table; the last modified time represents the time when the data table was last modified; the management process responds when it detects changes to the data table in the database; the management process is used to assign values ​​to the physical primary keys of the data table according to the changed data table, thereby updating the primary key fields in the physical primary keys; the management process is also used to assign values ​​to the last modified times of the data table according to the time of the data table change.

[0137] Example 4:

[0138] To achieve the above objectives, this application also provides a computer device 6, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions;

[0139] The processor executes the computer execution instructions stored in the memory to implement the aforementioned database synchronization method. The components of the database synchronization device can be distributed across different computer devices. Computer device 6 can be a smartphone, tablet, laptop, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple application servers), etc. The computer device in this embodiment includes, but is not limited to, a memory 61 and a processor 62 that can communicate with each other via a system bus. Figure 5 As shown. It should be noted that, Figure 5 Only computer devices with components are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead. In this embodiment, memory 61 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, memory 61 may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, memory 61 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Of course, memory 61 may also include both internal storage units and external storage devices of the computer device. In this embodiment, memory 61 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the database synchronization device in Embodiment 3. In addition, memory 61 can also be used to temporarily store various types of data that have been output or will be output. Processor 62, in some embodiments, can be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 62 is typically used to control the overall operation of the computer device. In this embodiment, processor 62 is used to run program code stored in memory 61 or process data, for example, to run a database synchronization device to implement the database synchronization method of the above embodiments.

[0140] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. It should be understood that the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disk, etc.

[0141] To achieve the above objectives, this application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores computer-executable instructions, and the program implements the corresponding function when executed by processor 62. The computer-readable storage medium of this embodiment is used to store computer-executable instructions for implementing the database synchronization method, and when executed by processor 62, it implements the database synchronization method of the above embodiment.

[0142] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0143] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0144] This application provides a computer program product, including a computer program that, when executed by a processor, implements the database synchronization method described above.

[0145] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A database synchronization method, characterized in that, Running in a database cluster with two or more databases, the method constructs physical primary keys and last modified times in the databases within the cluster, and creates management processes for assigning values ​​to the physical primary keys and the last modified times. The physical primary keys include all primary key fields in the data table; the primary key fields are unique identifiers for data rows in the data table; the last modified time represents the time when the data table was last modified; the management process responds to detecting changes in the data tables in the database; the management process is used to assign values ​​to the physical primary keys of the data tables based on the changed data tables, thereby updating the primary key fields in the physical primary keys; the management process is also used to assign values ​​to the last modified times of the data tables based on the time of the data table changes; the method includes: Receive a synchronization request sent by a first database in the database cluster; wherein the synchronization request is used to instruct data synchronization with a second database in the database cluster; If it is confirmed that the synchronization type in the synchronization request is full, then the second database is invoked to perform a full synchronization of the first database, so that the data tables in the first database are consistent with the data tables in the second database. If the synchronization type in the synchronization request is confirmed to be incremental, an incremental control file is generated based on the changed data rows in the data table of the second database. The incremental control file is obtained by unloading the changed data rows in the data table of the second database within a target time period. The target time period is the time from the last time the second database and the first database underwent full or incremental synchronization to the time when the synchronization request is received. Unloading the data involves importing the data from the data table into a text file. The step of generating an incremental control file based on the changed data rows in the data table of the second database includes: obtaining the table structure, physical primary key, and last modification time of the data table in the second database; and generating a structure file based on the table structure and the physical primary key. The table structure defines the filename and configuration information of the table fields. The physical primary key includes all primary key fields in the data table. The primary key field is a unique identifier for each data row in the data table. The structure file includes: the table name, a list of primary keys with at least one primary key field, a list of tables with at least one table field, and the total number of table fields in the table list. The last modification time represents the time when the data table in the second database was last modified. If it is determined that the last modification time is within the target time period, then the change records in the second database whose modification time is within the target time period are obtained. Based on the primary key list and the change records, the data rows in the second database that have changed are identified, and the data rows that have changed are summarized to obtain the change file. Based on at least one table field in the table field list, a data row of the change file is disassembled to obtain at least one cell string corresponding to the table field. The cell strings are arranged sequentially, and adjacent cell strings are separated by a preset field separator to obtain the row string of the data row. The row strings of all data rows in the change file are summarized to obtain the data file. The information content of the structure file is calculated to obtain the structure information content, and the information content of the data file is calculated to obtain the data information content. The structure file, the data file, the structure information content, the data information content, the field separator, and the data unloading mode are summarized to obtain the incremental control file; wherein, the data unloading mode represents the synchronization type between the first database and the second database. The second database is invoked to perform incremental synchronization with the first database, so that the data tables in the first database are consistent with the data tables in the second database.

2. The database synchronization method according to claim 1, characterized in that, Before invoking the second database to perform a full synchronization of the first database, the method further includes: A full control file is generated based on the data tables in the second database; wherein, the full control file is a file obtained by unloading data from the data tables in the second database; unloading data is the process of importing the data from the data tables into a text file.

3. The database synchronization method according to claim 1, characterized in that, The step of calling the second database to perform a full synchronization of the first database includes: Clear the data tables in the first database and send the full control file from the second database to the first database; Based on the field separator in the full control file, the line strings in the full control file are decomposed to obtain at least one unit string; wherein, the line string records the unit strings of the data rows that have changed in the second database, and the unit strings are the data in the data rows that have changed that correspond to the table fields; Based on the order of the table fields in the table field list in the full control file and the arrangement order of each unit string in a row string, the table fields are matched one-to-one with the unit strings to obtain a full row of a row string. The full rows of all row strings in the full control file are summarized to obtain a full table. The full table is entered into the data table of the first database, so that the content of the data table of the first database is consistent with the content of the data table of the second database.

4. The database synchronization method according to claim 1, characterized in that, The step of calling the second database to perform incremental synchronization of the first database includes: Extract the incremental control file from the second database and send the incremental control file to the first database; Based on the field separator in the incremental control file, the row string in the incremental control file is decomposed to obtain at least one unit string; wherein, the row string records the unit string of the data row that has changed in the second database, and the unit string is the data in the data row that has changed that corresponds to the table field; Based on the order of the table fields in the list of table fields in the incremental control file, and the order of the unit strings in a row string, the table fields are matched one-to-one with the unit strings to obtain an incremental row of a row string. The incremental rows of all row strings in the incremental control file are summarized to obtain an incremental table. Adjust the data rows in the data table of the first database according to the incremental table, so that the adjusted data table is consistent with the data table in the second database.

5. The database synchronization method according to claim 4, characterized in that, The step of adjusting the data rows in the data table of the first database according to the incremental table includes: Extract the increment type of the increment row in the increment table; wherein, the increment type defines the change of the increment row in the data table of the second database; the increment type includes: deletion type, addition type, and change type; Set one incremental row in the incremental table as the target row; If the incremental type of the target row is confirmed to be deletion, then delete the data row in the data table of the second database that is consistent with the target row; If the incremental type of the target row is confirmed to be a new row, then the target row is inserted into the data table of the second database; If the incremental type of the target row is confirmed to be change type, then the data row in the data table of the second database whose primary key field is consistent with the target row is set as the row to be adjusted, and the target row replaces the row to be adjusted.

6. A database synchronization device, characterized in that, Installed in a database cluster having two or more databases, the database synchronization device is used to implement the database synchronization method according to any one of claims 1-5, the device comprising: An input module is configured to receive a synchronization request sent by a first database in the database cluster; wherein the synchronization request is used to instruct data synchronization with a second database in the database cluster. The full synchronization module is used to call the second database to perform a full synchronization of the first database if it is confirmed that the synchronization type in the synchronization request is full, so that the data tables in the first database are consistent with the data tables in the second database. The incremental synchronization module is used to call the second database to perform incremental synchronization with the first database if it is confirmed that the synchronization type in the synchronization request is incremental, so that the data tables in the first database are consistent with the data tables in the second database.

7. The database synchronization device according to claim 6, characterized in that, Also includes: A configuration module is used to construct physical primary keys and last modified times in the database within the database cluster, and to create a management process for assigning values ​​to the physical primary keys and the last modified times. The physical primary key includes all primary key fields in the data table; the primary key field is a unique identifier for a data row in the data table; the last modified time represents the time when the data table was last modified; the management process responds to when changes are detected in the data tables in the database; the management process is used to assign values ​​to the physical primary keys of the data tables based on the changed data tables, thereby updating the primary key fields in the physical primary keys; the management process is also used to assign values ​​to the last modified times of the data tables based on the time of the data table changes.

8. A computer device, characterized in that, include: A processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the database synchronization method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the database synchronization method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the database synchronization method according to any one of claims 1-5.

Images