Data synchronization method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data synchronization method, device and equipment, a computer readable storage medium and a computer program product. The method can be applied to the field of financial science and technology. The method comprises the steps of obtaining a to-be-synchronized first data table in a source database, and obtaining form data through an information interaction interface; generating a sub-library condition screening table according to the form data; wherein the sub-database condition screening table comprises a mapping relationship between sub-databases and sub-database screening conditions; determining a synchronous data source corresponding to each sub-database from the first data table based on the sub-database condition screening table; and for each sub-database, determining to-be-synchronized data from the synchronization data source corresponding to the sub-database based on the to-be-synchronized field in the first data table, and synchronizing the to-be-synchronized data to the second data table of the sub-database. According to the method, resources of each database server can be effectively utilized, the load of a single database server is reduced, and the overall data processing speed and response time are improved.

Description

Technical Field

[0001] The present application relates to the technical field of fintech, and particularly to a data synchronization method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of fintech, full-volume data synchronization is a common synchronization method in data synchronization technology. It copies the entire data set from one database to another to ensure that the data in the two databases is exactly the same.

[0003] Currently, the solution for full-volume data synchronization is to synchronize all the existing data in the source database to different sub-databases in full volume. Each sub-database has all the existing data in the source database. In this way, the existing data in the source database will be redundantly stored in each sub-database. This redundant storage method of data will occupy additional database resources, resulting in low utilization rate of database resources. Summary of the Invention

[0004] Embodiments of the present application provide a data synchronization method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the utilization rate of database resources, and enhance the data processing speed and response time during the data synchronization process.

[0005] In a first aspect, embodiments of the present application provide a data synchronization method, including:

[0006] Obtain a first data table to be synchronized in a source database, and obtain form data through an information interaction interface; wherein, the form data is used to indicate the data sharding rule of the first data table;

[0007] Generate a sub-database condition screening table according to the form data; wherein, the sub-database condition screening table includes the mapping relationship between sub-databases and sub-database screening conditions, and the sub-database screening conditions corresponding to different sub-databases are different;

[0008] Based on the sub-database condition screening table, determine the synchronization data sources corresponding to each sub-database from the first data table;

[0009] For each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to a second data table in the sub-database.

[0010] In a second aspect, embodiments of the present application provide a data synchronization apparatus, including:

[0011] An acquisition module, configured to acquire a first data table to be synchronized in a source database, and acquire form data through an information interaction interface; wherein, the form data is used to indicate a data sharding rule of the first data table.

[0012] A processing module, configured to generate a sub-database condition screening table according to the form data; determine a synchronization data source corresponding to each sub-database from the first data table based on the sub-database condition screening table; for each sub-database, determine to-be-synchronized data from the synchronization data source corresponding to the sub-database based on the to-be-synchronized fields in the first data table, and synchronize the to-be-synchronized data to a second data table in the sub-database; wherein, the sub-database condition screening table includes a mapping relationship between a sub-database and a sub-database screening condition, and the sub-database screening conditions corresponding to different sub-databases are different.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the data synchronization method provided in the first aspect of the embodiments of the present application are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data synchronization method provided in the first aspect of the embodiments of the present application are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data synchronization method provided in the first aspect of the embodiments of the present application are implemented.

[0016] The technical solution provided in the embodiments of the present application acquires a first data table to be synchronized in a source database, acquires form data for indicating a data sharding rule of the first data table through an information interaction interface, generates a sub-database condition screening table according to the form data, determines a synchronization data source corresponding to each sub-database from the first data table based on the sub-database condition screening table, and determines to-be-synchronized data for each sub-database from the synchronization data source corresponding to each sub-database based on the to-be-synchronized fields in the first data table, and synchronizes each to-be-synchronized data to a second data table in each sub-database. The stock data in the source database is sharded and synchronized to different sub-databases through a unified sub-database condition screening table, and different sub-databases synchronize different stock data in the source database, which can effectively utilize the resources of each database server, reduce the load on a single database server, and improve the overall data processing speed and response time. Description of the Drawings

[0017] Figure 1 It is a schematic diagram of an application environment of the data synchronization method provided in the embodiments of the present application.

[0018] Figure 2 It is a schematic flowchart of a data synchronization method provided by an embodiment of the present application;

[0019] Figure 3 It is another schematic flowchart of a data synchronization method provided by an embodiment of the present application;

[0020] Figure 4 It is a schematic structural diagram of a data synchronization device provided by an embodiment of the present application;

[0021] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0022] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be further described in detail below through the following embodiments in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Those skilled in the art can adjust them as needed to suit specific application scenarios. Additionally, it should be noted that for the sake of description, only parts related to the present application rather than all structures are shown in the drawings.

[0023] The data synchronization solution provided by the embodiments of the present application can be applied to the field of fintech. For example, in banking services, customer information in the source database can be sharded according to factors such as the location of the customer, account type, or transaction frequency, and different customer information can be synchronized to different sub-databases to meet different business requirements.

[0024] Figure 1 It is a schematic diagram of an application environment of a data synchronization method provided by an embodiment of the present application. As Figure 1 shown, at least a first server 101, a second server 102, and a data synchronization platform 103 can be included in this application environment. Among them, the bidirectional arrows in the figure represent communication services, and the thick unidirectional arrows represent data synchronization service channels.

[0025] Specifically, the first server 101 can include an independently operating server. The first server 101 can include a network communication unit, a processor, a memory, etc. The first server 101 can be used as the carrier of the source database.

[0026] The second server 102 can include an independently operating server, or a server cluster composed of multiple servers. The second server 102 can include a network communication unit, a processor, a memory, etc. The second server 102 can be used as the carrier of the sub-database.

[0027] The data synchronization platform 103 is communicatively connected to the first server 101 and the second server 102 respectively, and is used to synchronize the data in the source database of the first server 101 to the sub-database in the second server 102.

[0028] It should be noted that the data synchronization method provided in the embodiments of the present application can be executed by a data synchronization device. The data synchronization device can be implemented in a software and / or hardware manner and can be integrated into the above data synchronization platform.

[0029] Figure 2 It is a schematic flowchart of a data synchronization method provided in the embodiments of the present application. As Figure 2 shown, the method may include:

[0030] S201. Obtain the first data table to be synchronized in the source database, and obtain form data through the information interaction interface.

[0031] The above first data table refers to the data table that needs to be sharded. In the specific implementation process, the first data table to be sharded / synchronized can be selected from the source database in combination with the actual banking business requirements. For example, the first data table can be a table with a large amount of data, a table with fast data growth, a table that does not require global transaction support, etc.

[0032] As an optional implementation manner, the process of obtaining the first data table to be synchronized in the source database includes: obtaining the operation log file of the source database; analyzing the operation log file to obtain the data tables in the source database whose access volume exceeds a preset threshold; and determining the data tables whose access volume exceeds the preset threshold as the first data tables.

[0033] Among them, the operation log file records all operations and events of the database. By parsing the operation log file, key information related to data table access is extracted, such as query statements, execution time, table names accessed, etc. Statistical analysis is performed on the extracted key information to obtain the access times / access volume of each data table. Further, the access volume of each data table is compared with the preset threshold, and the data tables whose access volume exceeds the preset threshold are screened out and determined as the first data tables. Optionally, after the first data tables are determined, the user can also confirm the first data tables, that is, perform the sharding storage operation on the first data tables after obtaining the sharding storage instruction of the user for the first data tables. By sharding and storing the data tables with high access volume, the read / write requests of the data tables with high access volume are dispersed, which can relieve the access pressure of the source database and improve the concurrent processing ability of the database system at the same time.

[0034] The above information interaction interface can be a web page, a desktop application interface, or any other form of graphical user interface. Among them, a form is set on the information interaction interface, and the form is used to collect the data sharding rules of the user for the first data table. The form allows the user to set the following form data:

[0035] Database sharding field: The user can input or select a field through the form, and the information interaction interface obtains the selected field by the user and uses this field as the basis for data sharding of the first data table. For example, the company ID is used as the database sharding field.

[0036] Database sharding screening conditions: The user can input specific screening conditions based on the database sharding field in the form. These screening conditions define the association relationship between the data in the first data table and the sharded databases, that is, which data should be assigned to which specific sharded database. For example, the data with the company ID between 1 and 25 is assigned to sharded database 1, and the data with the company ID between 26 and 50 is assigned to sharded database 2, and so on.

[0037] The information interaction interface obtains the form data input by the user in the form and sends the form data to the data synchronization platform. The form data is used to indicate the data sharding rules of the first data table.

[0038] Optionally, before obtaining the form data through the information interaction interface, the data synchronization platform also needs to verify the user permissions. When it is determined that the user has the operation permissions, the user is allowed to set the data sharding rules. In addition, the data synchronization platform can also check the connectivity of the source database based on the connection information of the source database, and check the connectivity of the sharded databases based on the connection information of the sharded databases. Among them, the above connection information can include the address information of the database and the communication port, etc.

[0039] S202. Generate a database sharding condition screening table according to the form data.

[0040] Among them, the database sharding condition screening table includes the mapping relationship between the sharded databases and the database sharding screening conditions. The database sharding screening conditions corresponding to different sharded databases are different, and the same sharded database can correspond to at least one database sharding screening condition.

[0041] After obtaining the form data input by the user, a database sharding condition screening table can be generated according to the form data. Specifically, a data table can be created in the source database, and this data table is used to store the data sharding rules of the first data table. The association relationship between the user-defined database sharding screening conditions and the sharded databases is inserted into this data table, so as to obtain the database sharding condition screening table.

[0042] Exemplarily, the database sharding condition screening table can be as shown in Table 1 below:

[0043] Sub-library screening conditions Sub-database Company ID is in [1, 25] Sub-database 1 Company ID is in [26, 50] Sub-database 2 Company ID is in [51, 75] Sub-database 3 …… …… Company ID is in [126, 150] Sub-database n

[0044] Table 1

[0045] S203. Determine the synchronization data sources corresponding to each sub-database from the first data table based on the sub-database condition screening table.

[0046] Use the sub-database screening conditions in the sub-database condition screening table to screen the first data table in the source database to determine which data should be synchronized to which specific sub-database, that is, obtain the synchronization data sources corresponding to each sub-database, and mark the synchronization data sources corresponding to each sub-database in the first data table to prepare for subsequent data synchronization operations.

[0047] As an optional implementation manner, the above S203 may include: monitoring the running state of the source database to obtain running state data; and when it is determined that the source database is overloaded according to the running state data, determining the synchronization data sources corresponding to each sub-database from the first data table based on the sub-database condition screening table.

[0048] Specifically, a database monitoring tool is used to monitor the running state of the database to obtain the running state data of the source database. Among them, the running state data includes, but is not limited to, the load condition of the database, the number of active connections, the query execution time and frequency, the CPU usage rate, the memory usage rate, and the disk input / output, etc. By analyzing these running state data, it is judged whether the source database is within the normal workload range. For example, if the CPU usage rate of the source database continuously exceeds the preset threshold, or the disk input / output is in a high-load state for a long time, it can be determined that the source database is overloaded.

[0049] If it is determined that the source database is overloaded based on these running state data, the sharding synchronization operation of the first data table can be triggered, that is, the operation of determining the synchronization data sources corresponding to each sub-database from the first data table based on the sub-database condition screening table is executed, and the data in the first data table is shard-synchronized to different sub-databases to reduce the load pressure of the source database and improve the performance of the source database. At the same time, it can also avoid the source database from frequently performing data synchronization and reduce the impact on the business; in addition, sharding synchronization can disperse the data of the source database into multiple sub-databases, making the utilization of database resources more balanced and improving the overall utilization efficiency of database resources.

[0050] S204. For each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to the second data table of the sub-database.

[0051] Among them, the to-be-synchronized fields are fields in the first data table, and the to-be-synchronized fields can be set in combination with actual banking business requirements. For example, the to-be-synchronized fields can be company ID, company transaction amount, transaction time, and industrial and commercial information of the company, etc.

[0052] Extract the to-be-synchronized data from the marked synchronization data source based on the to-be-synchronized fields, and synchronize the extracted to-be-synchronized data to the second data table in the sub-database, achieving the purpose of synchronizing different data shards in the first data table to different sub-databases. In some examples, the to-be-synchronized data can be synchronized to the second data table in the sub-database at a specified time. The above-mentioned specified time can be a low business peak period. Synchronizing the to-be-synchronized data to the sub-database during the low business peak period can reduce the impact of data synchronization on the business.

[0053] Optionally, data synchronization is a type of data insertion for the sub-database. When the index is enabled, the speed of data synchronization will be affected and reduced by the index. Therefore, before synchronizing the data, the index of the sub-database can be turned off. After the data synchronization is completed, rebuild the sub-database index. By this method, the data synchronization speed can be improved.

[0054] Optionally, after the data synchronization is completed, data verification can also be performed on the second data table in the sub-database to determine whether the data in the second data table is complete and error-free and consistent with the source database.

[0055] The data synchronization method provided by the embodiments of the present application obtains the first data table to be synchronized in the source database, and obtains form data for indicating the data sharding rule of the first data table through the information interaction interface, generates a sub-database condition screening table according to the form data, determines the synchronization data sources corresponding to each sub-database from the first data table based on the sub-database condition screening table, and determines the to-be-synchronized data of each sub-database from the synchronization data sources corresponding to each sub-database based on the to-be-synchronized fields in the first data table, and synchronizes each to-be-synchronized data to the second data table in each sub-database. By means of the unified sub-database condition screening table, the stock data in the source database is sharded and synchronized to different sub-databases, and different sub-databases synchronize different stock data in the source database, which can effectively utilize the resources of each database server, reduce the load on a single database server, and improve the overall data processing speed and response time.

[0056] In one embodiment, optionally, as Figure 3 shown, the above S202 may include:

[0057] S301. Perform validity verification on the form data.

[0058] After obtaining the form data, it is possible to check the format, logical consistency of the form data, and whether it meets the requirements for sharding conditions. For example, the above form data may include sharding fields and sharding filtering conditions, and it is possible to verify whether the name of the sharding field is included in the first data table, whether the range of the sharding filtering conditions is reasonable, and whether there is a conflict with the business rules, etc.

[0059] Among them, if the verification passes, then execute the following S302; if the verification fails, then execute the following S303.

[0060] S302. Generate a sharding condition filtering table according to the form data.

[0061] If the validity verification of the form data passes, a data table can be created in the source database. This data table is used to store the data sharding rules of the first data table, and the user-defined sharding filtering conditions and the association relationship of the sharding database are inserted into this data table, thereby obtaining a sharding condition filtering table.

[0062] In some cases, if a sharding condition filtering table has been stored in the source database, the existing sharding condition filtering table can be updated according to the form data.

[0063] S303. Output modification suggestion information for the form data through the information interaction interface, obtain the modified form data, and use the modified form data as the new form data to continue executing the step of validating the form data.

[0064] During the validity verification process of the form data, all detected problems can be recorded and these problems can be classified into errors. For example, the error categories can be format errors, numerical range errors, logical errors, integrity errors, or data consistency errors, etc. For each type of error, modification suggestion information is generated and feedback to the user through the information interaction interface so that the user can adjust the form data according to the modification suggestion information. For example, for a format error, the modification suggestion information provided can be an example of the correct format; for a numerical range error, the modification suggestion information provided can be the acceptable range of values; for a logical error, the modification suggestion information provided can be to explain the reason for the logical error and provide an example of the correct logical expression; for an integrity error, the modification suggestion information provided can be to indicate which required fields are missing and require completion.

[0065] After the user modifies the form data and submits the form again, the data synchronization platform obtains the modified form data and uses the modified form data as the new form data, and repeats the process of the above S301 - S303 until the validity verification of the obtained form data passes. The data synchronization platform generates / updates the sharding condition filtering table based on the form data with passed validity verification.

[0066] In this way, it can be ensured that the collected form data is accurate and valid, and can be used to generate / update the sub-database condition filtering table, so as to achieve precise data sharding.

[0067] In some embodiments, optionally, the process of writing the data to be synchronized into the second data table of the sub-database in S204 above may include: based on the field mapping relationship between the first data table and the second data table, obtaining the matching fields of the fields to be synchronized in the second data table; obtaining the data requirement information of the matching fields; based on the data requirement information, processing the data to be synchronized, and synchronizing the processed data to the second data table.

[0068] Specifically, the above field mapping relationship is used to represent the mapping relationship between the fields in the first data table and the second data table. By identifying the table structures of the first data table and the second data table, the field mapping relationship between the first data table and the second data table can be obtained.

[0069] Based on the above field mapping relationship, identify the matching fields in the second data table corresponding to the fields to be synchronized in the first data table, and ensure that each field to be synchronized can find a corresponding matching field in the second data table, so as to facilitate data synchronization. For example, the field to be synchronized "user ID" corresponds to the field "customer number" in the second data table.

[0070] For each matching field, obtain its data requirement information, including data type, format, length, range, etc. According to the data requirement information of the matching fields, process the data to be synchronized. The processing methods here include but are not limited to data desensitization processing, data formatting, field value calculation, and data type conversion. After processing the data to be synchronized, if the processed data meets the structure and data requirements of the second data table, the processed data can be synchronized to the second data table.

[0071] Exemplarily, data type conversion means converting the data to be synchronized into the data type required by the matching field. Suppose the field to be synchronized "date of birth" in the first data table is of string type, while the data type required by the matching field of this field to be synchronized in the second data table is of date type. Then, before data synchronization, the data to be synchronized needs to be converted from string to date type.

[0072] Another example is that field value calculation means that the field value of a field in the second data table is calculated based on the field values of one or more fields in the first data table. For example, the field "age" in the second data table is calculated from the field "date of birth" in the first data table.

[0073] Data formatting means that if the matching field requires a specific format, the data to be synchronized needs to be formatted accordingly. For example, convert phone numbers from different formats to a unified international standard format.

[0074] Data desensitization processing means that when synchronizing data involving user privacy, the data to be synchronized needs to be desensitized. For example, partially hide the email address.

[0075] In this embodiment, based on the data requirement information of the matching field of the field to be synchronized in the second data table, the data to be synchronized is processed, and the processed data is synchronized to the second data table, so that the synchronized data can meet the requirements of the second data table, providing an accurate data basis for subsequent financial decisions and improving the accuracy and efficiency of financial decisions. For example, when evaluating the credit risk of a customer, accurate data can accurately evaluate the repayment ability of the borrower, thereby reducing the risk of non-performing loans.

[0076] Figure 4 It is a schematic structural diagram of a data synchronization device provided by an embodiment of the present application. As Figure 4 shown, the device may include: an acquisition module 401 and a processing module 402.

[0077] Specifically, the acquisition module 401 is used to acquire the first data table to be synchronized in the source database and acquire form data through the information interaction interface; wherein, the form data is used to indicate the data sharding rule of the first data table;

[0078] The processing module 402 is used to generate a sub-database condition screening table according to the form data; determine the synchronization data sources corresponding to each sub-database from the first data table based on the sub-database condition screening table; for each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to the second data table of the sub-database; wherein, the sub-database condition screening table includes the mapping relationship between the sub-database and the sub-database screening condition, and the sub-database screening conditions corresponding to different sub-databases are different.

[0079] Based on the above embodiment, optionally, the processing module 402 is further used to perform validity verification on the form data; if the verification passes, generate a sub-database condition screening table according to the form data; if the verification fails, output modification suggestion information for the form data through the information interaction interface, acquire the modified form data, and use the modified form data as the new form data, and continue to execute the step of performing validity verification on the form data.

[0080] Based on the above embodiments, optionally, the processing module 402 is further configured to obtain a matching field of the fields to be synchronized in the second data table based on the field mapping relationship between the first data table and the second data table; obtain data requirement information of the matching field; process the data to be synchronized based on the data requirement information, and synchronize the processed data to the second data table.

[0081] Optionally, the processing method for the data to be synchronized includes at least one of the following:

[0082] Data desensitization processing, data formatting, field value calculation, and data type conversion.

[0083] Based on the above embodiments, optionally, the processing module 402 is further configured to monitor the running state of the source database to obtain running state data; and when it is determined that the source database is overloaded according to the running state data, determine the synchronization data sources corresponding to each sub-database from the first data table based on the table screening conditions for sub-databases.

[0084] Based on the above embodiments, optionally, the obtaining module 401 is further configured to obtain the running log file of the source database; analyze the running log file to obtain the data tables in the source database whose access volume exceeds a preset threshold; and determine the data tables whose access volume exceeds the preset threshold as the first data table.

[0085] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present application is shown as Figure 5 shown. The device includes a processor 50, a memory 51, an input device 52, and an output device 53. The number of processors 50 in the device may be one or more. Figure 5 Taking one processor 50 as an example; the processor 50, the memory 51, the input device 52, and the output device 53 in the device may be connected through a bus or other means. Figure 5 Taking the connection through the bus as an example.

[0086] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data synchronization method in the embodiments of the present application (for example, the obtaining module 401 and the processing module 402 in the data synchronization device). The processor 50 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 51, that is, implements the data synchronization method provided in any of the above embodiments.

[0087] The memory 51 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created during the data synchronization process, etc. In addition, the memory 51 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 51 may further include a memory remotely provided with respect to the processor 50, and these remote memories may be connected to the device / terminal / server through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The input device 52 may be used to receive input digital or character information, and generate key signal inputs related to user settings and function controls of the electronic device. The output device 53 may include a display device such as a display screen.

[0089] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0090] Obtain the first data table to be synchronized in the source database, and obtain form data through the information interaction interface; wherein, the form data is used to indicate the data sharding rule of the first data table;

[0091] Generate a sub-database condition screening table according to the form data; wherein, the sub-database condition screening table includes the mapping relationship between the sub-databases and the sub-database screening conditions, and the sub-database screening conditions corresponding to different sub-databases are different;

[0092] Based on the sub-database condition screening table, determine the synchronization data sources corresponding to each sub-database from the first data table;

[0093] For each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to the second data table in the sub-database.

[0094] In one embodiment, a computer program product is further provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0095] Obtain the first data table to be synchronized in the source database, and obtain form data through the information interaction interface; wherein, the form data is used to indicate the data sharding rule of the first data table;

[0096] Generate a sub-library condition screening table according to the form data; wherein, the sub-library condition screening table includes the mapping relationship between the sub-databases and the sub-library screening conditions, and the sub-library screening conditions corresponding to different sub-databases are different;

[0097] Based on the sub-library condition screening table, determine the synchronization data sources corresponding to each sub-database from the first data table;

[0098] For each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to the second data table of the sub-database.

[0099] The data synchronization device, electronic device, computer-readable storage medium, and computer program product provided in the above embodiments can execute the data synchronization method provided in any embodiment of the present application, and have the corresponding functional modules and beneficial effects for executing the method. The technical details not described in detail in the above embodiments can be referred to the data synchronization method provided in any embodiment of the present application.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general-purpose hardware, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk, or optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0101] It should be noted that the various units and modules included in the above embodiments are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present application.

[0102] Note that the above is only a preferred embodiment of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments here, and various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments can be included, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data synchronization method, characterized in that, including: obtaining a first data table to be synchronized in a source database, and obtaining form data through an information interaction interface; wherein, the form data is used to indicate a data sharding rule of the first data table; generating a sub-database condition screening table according to the form data; wherein, the sub-database condition screening table includes a mapping relationship between sub-databases and sub-database screening conditions, and the sub-database screening conditions corresponding to different sub-databases are different; determining a synchronization data source corresponding to each sub-database from the first data table based on the sub-database condition screening table; for each sub-database, determining data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronizing the data to be synchronized to a second data table in the sub-database.

2. The method according to claim 1, characterized in that, The generating a sub-database condition screening table according to the form data includes: performing validity verification on the form data; if the verification passes, generating a sub-database condition screening table according to the form data; if the verification fails, outputting modification suggestion information for the form data through the information interaction interface, obtaining the modified form data, and using the modified form data as new form data to continue performing the step of performing validity verification on the form data.

3. The method according to claim 1, wherein The writing the data to be synchronized into the second data table in the sub-database includes: obtaining a matching field of the field to be synchronized in the second data table based on a field mapping relationship between the first data table and the second data table; obtaining data requirement information of the matching field; processing the data to be synchronized based on the data requirement information, and synchronizing the processed data to the second data table.

4. The method according to claim 3, wherein The processing methods for the data to be synchronized include at least one of the following: data desensitization processing, data formatting, field value calculation, and data type conversion.

5. The method according to any one of claims 1 to 4, characterized in that, The determining a synchronization data source corresponding to each sub-database from the first data table based on the sub-database condition screening table includes: monitoring an operating state of the source database to obtain operating state data; when it is determined that the source database is overloaded according to the operating state data, determining a synchronization data source corresponding to each sub-database from the first data table based on the sub-database condition screening table.

6. The method according to any one of claims 1 to 4, characterized in that The obtaining a first data table to be synchronized in a source database includes: obtaining an operating log file of the source database; analyzing the operating log file to obtain data tables in the source database with an access volume exceeding a preset threshold; determining the data tables with an access volume exceeding the preset threshold as the first data table.

7. A data synchronization device, characterized in that, including: an obtaining module, configured to obtain a first data table to be synchronized in a source database, and obtain form data through an information interaction interface; wherein, the form data is used to indicate a data sharding rule of the first data table; A processing module, configured to generate a sub-library condition screening table according to the form data; determine a synchronization data source corresponding to each sub-database from the first data table based on the sub-library condition screening table; for each sub-database, determine the data to be synchronized from the synchronization data source corresponding to the sub-database based on the fields to be synchronized in the first data table, and synchronize the data to be synchronized to the second data table of the sub-database; wherein, the sub-library condition screening table includes a mapping relationship between the sub-database and the sub-library screening condition, and the sub-library screening conditions corresponding to different sub-databases are different.

8. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images