Library-level synchronization method and system based on data synchronization and medium

Abstract

The invention discloses a library-level synchronization method and system based on data synchronization and a medium, mainly relates to the technical field of library-level synchronization, and is used for solving the problem that the existing data synchronization method cannot effectively perform library-level synchronization. Comprising the steps of obtaining data information of a source database and a target database needing data synchronization; determining a corresponding data conversion program; determining synchronization time for synchronizing the data in the source database to the target database; collecting synchronous data from the source database according to the synchronous time, and adding an exclusive lock to the synchronous data to be collected; unifying the data structure and the field type of the synchronous data into the data structure and the field type of the target database; a transaction processing mechanism is used, synchronous data are obtained in a data loading mode, and meanwhile the isolation level of an exclusive lock and a current transaction is set; after data loading is completed, the loaded data size and the expected data size are obtained, whether synchronous data are accurate and complete or not is verified, and a verification result is obtained.

Description

Technical Field

[0001] This application relates to the technical field of database synchronization, and in particular, to a library-level synchronization method, system, and medium based on data synchronization. Background Art

[0002] With the rapid development of computer technology, the synchronization and interoperability performance of data have become increasingly important for many software applications and system architectures. Especially in the fields of distributed systems, cloud computing, big data processing, etc., data synchronization technology has become the key to ensuring data consistency, reliability, and efficiency. However, existing data synchronization methods have some problems in certain scenarios.

[0003] Most current data synchronization methods focus on system-level or application-level synchronization. This common synchronization method may encounter performance bottlenecks and synchronization problems when facing complex distributed systems and multi-user environments. For example, when multiple processes or threads need to access and modify data in the same library, if library-level synchronization cannot be effectively performed, it may lead to data conflicts and inconsistencies.

[0004] Therefore, there is an urgent need for a library-level synchronization method, system, and medium based on data synchronization to solve the problem that existing data synchronization methods cannot effectively perform library-level synchronization. Summary of the Invention

[0005] In view of the above deficiencies of the prior art, this application provides a library-level synchronization method, system, and medium based on data synchronization to solve the problem that existing data synchronization methods cannot effectively perform library-level synchronization.

[0006] In a first aspect, this application provides a library-level synchronization method based on data synchronization, the method includes: Obtain the data information of the source database and the target database that need data synchronization; among them, the data information at least includes: data structure, field type, and data update frequency; when the data structures or field types of the source database and the target database are different, determine the corresponding data conversion program based on the data structure or field type; according to the data update frequency and real-time requirement of the source database, determine the synchronization time to synchronize the data in the source database to the target database; according to the synchronization time, collect the synchronization data from the source database, and at the same time add an exclusive lock to the synchronization data to be collected; when the synchronization data is collected, call the data conversion program, and then unify the data structure and field type of the synchronization data into the data structure and field type of the target database; based on the data volume of the synchronization data, determine the data loading method to load the synchronization data into the target database, and at the same time use the transaction processing mechanism to obtain the synchronization data through the data loading method, and set the exclusive lock and the isolation level of the current transaction; when the data loading is completed, obtain the loaded data volume and the expected data volume, and verify whether the synchronization data is accurate and complete to obtain the verification result.

[0007] In an implementation manner of the present application, when the data structures or field types of the source database and the target database are different, determine the corresponding data conversion program based on the data structure or field type, specifically including: When the data structures of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the data structure of the source database into the data structure of the target database; When the field types of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the field type of the source database into the field type of the target database.

[0008] In an implementation manner of the present application, according to the data update frequency and real-time requirement of the source database, determine the synchronization time to synchronize the data in the source database to the target database; according to the synchronization time, collect the synchronization data from the source database, and at the same time add an exclusive lock to the synchronization data to be collected, specifically including: When there is a real-time requirement, determine that the synchronization time of the source database is real-time synchronization; When there is no real-time requirement, determine that the synchronization time of the source database is scheduled synchronization; Furthermore, according to the preset relationship between the data update frequency and the scheduled interval, determine the scheduled interval corresponding to the current data update frequency, and then obtain the synchronization time; Obtain whether the source database supports incremental changes. When it supports incremental changes, obtain the incremental data from the previous synchronization time to the current synchronization time as the synchronization data; When it does not support incremental changes, obtain the synchronization data by means of full extraction.

[0009] In an implementation manner of the present application, before unifying the data structure and field type of the synchronization data into those of the target database, the method further includes: Adopting an intelligent error value correction method to remove invalid information and duplicate information in the synchronization data; Based on a preset error pattern library, identifying error values from the synchronization data, correcting the error values to preset reasonable values, and recording them in a preset correction log.

[0010] In an implementation manner of the present application, based on the data volume of the synchronization data, determining the data loading method for loading the synchronization data into the target database, specifically including: When the data volume of the synchronization data is less than a preset data volume, determining that the data loading method for loading the synchronization data into the target database is to load it item by item; When the data volume of the synchronization data is greater than or equal to the preset data volume, determining that the data loading method for loading the synchronization data into the target database is to load it in batches.

[0011] In an implementation manner of the present application, after the data loading is completed, obtaining the loaded data volume and the expected data volume, verifying whether the synchronization data is accurate and complete, and obtaining a verification result, specifically including: After the data loading is completed, obtaining the loaded data volume and the expected data volume; When the loaded data volume is less than the expected data volume, determining that there are problems such as network interruption or server downtime, and determining that the verification result is: the synchronization data is inaccurate and incomplete; When the loaded data volume is greater than the expected data volume, further comparing whether the loaded data exists in the original synchronization data in the source database. When the loaded data exists in the original synchronization data in the source database, determining that the verification result is: synchronization completed; When the loaded data volume is equal to the expected data volume, determining that the verification result is: synchronization completed.

[0012] In a second aspect, the present application provides a library-level synchronization system based on data synchronization, and the system includes: A determination module, configured to obtain the data information of the source database and the target database that need data synchronization; wherein, the data information at least includes: data structure, field type, and data update frequency; when the data structures or field types of the source database and the target database are different, determining the corresponding data conversion program based on the data structure or field type; An acquisition module, configured to determine the synchronization time for synchronizing the data in the source database to the target database according to the data update frequency and real-time requirement of the source database; according to the synchronization time, acquiring the synchronization data from the source database, and adding an exclusive lock to the synchronization data to be acquired at the same time; A unified module, which is used to call a data conversion program when synchronous data is collected, and then unify the data structure and field type of the synchronous data into the data structure and field type of the target database; An acquisition module, which is used to determine the data loading method for loading the synchronous data into the target database based on the data volume of the synchronous data, and at the same time use a transaction processing mechanism to obtain the synchronous data through the data loading method, and set an exclusive lock and the isolation level of the current transaction; A verification module, which is used to obtain the loaded data volume and the expected data volume after the data loading is completed, verify whether the synchronous data is accurate and complete, and obtain a verification result.

[0013] In an implementation manner of the present application, the determination module includes a conversion unit, which is used to select a data conversion program for converting the data structure of the source database into the data structure of the target database from a preset data conversion program database when the data structures of the source database and the target database are different; When the field types of the source database and the target database are different, select a data conversion program for converting the field type of the source database into the field type of the target database from a preset data conversion program database.

[0014] In an implementation manner of the present application, the acquisition module includes an acquisition unit, which is used to determine that the synchronization time of the source database is real-time synchronization when there is a real-time requirement; When there is no real-time requirement, determine that the synchronization time of the source database is scheduled synchronization; Furthermore, according to a preset relationship between the data update frequency and the scheduled interval, determine the scheduled interval corresponding to the current data update frequency, and then obtain the synchronization time; Obtain whether the source database supports incremental changes. When it supports incremental changes, obtain the incremental data from the previous synchronization time to the current synchronization time as the synchronous data; When it does not support incremental changes, obtain the synchronous data by means of full extraction.

[0015] In a third aspect, the present application provides a non-volatile computer storage medium, on which computer instructions are stored, and the computer instructions, when executed, implement a library-level synchronization method based on data synchronization as described in any one of the above.

[0016] Those skilled in the art can understand that the present application has at least the following beneficial effects: The present application provides a library-level synchronization method, system, and medium based on data synchronization, which can upload newly added data in the source database to the target database, achieving library-level synchronization and reducing the performance overhead caused by system-level or application-level synchronization. Additionally, during the process of obtaining and uploading synchronization data, exclusive locks and transaction processing mechanisms are used to avoid data conflicts and inconsistencies. Moreover, to avoid data incompatibility between two databases, the present application can call the corresponding data conversion program, thereby being able to handle synchronization between databases with different data structures or field types, improving the generality and flexibility of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of a library-level synchronization method based on data synchronization provided by an embodiment of the present application.

[0019] Figure 2 It is a schematic internal structure diagram of a library-level synchronization system based on data synchronization provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Those skilled in the art should understand that the embodiments described below are only the preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are only used to explain the technical principles of the present disclosure and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts should still fall within the protection scope of the present disclosure.

[0021] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, commodity or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity or device including the element.

[0022] The technical solutions proposed by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0023] An embodiment provides a library-level synchronization method based on data synchronization, as Figure 1 shown, the method provided by the embodiment of the present application mainly includes the following steps: Step 110: Obtain the data information of the source database and the target database that need data synchronization; when the data structures or field types of the source database and the target database are different, determine the corresponding data conversion program based on the data structure or field type.

[0024] It should be noted that the data information at least includes: data structure, field type, and data update frequency.

[0025] First of all, in this step, it is clear that the source database to be synchronized can be a relational database (such as MySQL, Oracle, etc.), a NoSQL database (such as MongoDB, redis, etc.). When selecting the source database, factors such as data structure and data update frequency need to be considered. Ensure that the selected source database can provide an appropriate data access interface or API for data extraction and synchronization operations. Different source databases may provide different data structures and field types, so it is necessary to ensure that the same data structures and field types exist in the target database as in the source database. In the case where the structures of the two databases are different, data mapping and conversion operations (data conversion programs) are required to convert the data in the source database into the data format of the target database.

[0026] In some embodiments, when the data structures or field types of the source database and the target database are different, determine the corresponding data conversion program based on the data structure or field type, specifically including: When the data structures of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the data structure of the source database into the data structure of the target database; when the field types of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the field type of the source database into the field type of the target database.

[0027] Those skilled in the art can understand that, in this application, through a preset database of data conversion programs, data conversion programs can be automatically selected and applied without manually writing conversion logic, thus significantly improving the efficiency of data synchronization. Additionally, this strategy enables the system to handle the synchronization between databases of different types and structures, enhancing data compatibility and enabling the system to be more widely applied to different business scenarios. Moreover, when the data structure or field type of the source database or target database changes, only the corresponding data conversion programs need to be updated or added, without the need for large-scale modification of the entire synchronization system. Additionally, once the data conversion programs are developed and pass the tests, they can be reused in multiple synchronization tasks, reducing the development and maintenance costs.

[0028] Step 120: Determine the synchronization time for synchronizing the data in the source database to the target database according to the data update frequency and real-time requirement of the source database; at the synchronization time, collect the synchronization data from the source database and simultaneously add an exclusive lock to the synchronization data to be collected.

[0029] This step can be specifically as follows: When there is a real-time requirement, determine the synchronization time of the source database as real-time synchronization; when there is no real-time requirement, determine the synchronization time of the source database as scheduled synchronization; then, according to the preset relationship between the data update frequency and the scheduled interval, determine the scheduled interval corresponding to the current data update frequency, and further obtain the synchronization time; obtain whether the source database supports incremental changes. When it supports incremental changes, obtain the incremental data from the previous synchronization time to the current synchronization time as the synchronization data; when it does not support incremental changes, obtain the synchronization data by means of full extraction.

[0030] More specifically, when performing data extraction, based on the incremental changes in the data, ensure that the synchronized data is up-to-date. Since the update mechanisms of different source databases are different, some source databases may experience incremental changes, that is, only some fields have changed. Therefore, when performing data extraction, adopt corresponding strategies for different source databases to facilitate the extraction of only the changed data. For source databases that support incremental changes, the timestamp of the last synchronization can be obtained by querying the historical records, and then the changed data can be filtered out based on the timestamp; for source databases that do not support incremental changes, a full extraction method needs to be adopted. Since multiple application programs may need to access the same source database simultaneously, attention needs to be paid to the issue of concurrent control when performing data extraction. Locks, transactions, etc. can be used to ensure data consistency and integrity. When a data extraction task starts, an exclusive lock is added to the data rows or tables to be extracted. In this way, if other application programs or tasks want to access this data, they must wait for the lock to be released. During the data extraction process, the entire extraction operation is placed in a transaction, which can ensure data consistency, that is, if an error occurs or conflicts with other concurrent operations during the extraction process, the database can roll back to the state before the transaction started.

[0031] Step 130: When the synchronized data is collected, call the data conversion program to unify the data structure and field types of the synchronized data into the data structure and field types of the target database.

[0032] For example, convert the string type in the source database to the text type, date type, etc. in the target database.

[0033] In addition, before unifying the data structure and field types of the synchronized data into the data structure and field types of the target database, the method can also clean the data. The specific cleaning method can be: Adopt an intelligent error value correction method to remove invalid information and duplicate information in the synchronized data; based on a preset error pattern library, identify error values from the synchronized data and correct the error values to preset reasonable values, and record them in a preset correction log at the same time. For example, null values, special characters, HTML tags, etc. can be removed. Since different databases may use different field names and naming specifications, it is necessary to map the field names and field values in the source database to the corresponding field names and field values in the target database. For example, map the "username" field in the source database to the "user_name" field in the target database.

[0034] Step 140: Based on the data volume of the synchronized data, determine the data loading method for loading the synchronized data into the target database. At the same time, use the transaction processing mechanism to obtain the synchronized data through the data loading method, and set the exclusive lock and the isolation level of the current transaction.

[0035] In some embodiments, based on the amount of synchronization data, determine the data loading method for loading the synchronization data into the target database, which specifically includes: When the amount of synchronization data is less than the preset amount of data, determine that the data loading method for loading the synchronization data into the target database is to load it one by one (this method is simple and direct, with less one-time occupancy of system resources, and can more accurately control the loading process of each piece of data, facilitating positioning and processing in case of problems); When the amount of synchronization data is greater than or equal to the preset amount of data, determine that the data loading method for loading the synchronization data into the target database is batch loading (batch loading can reduce the number of database interactions and improve data loading efficiency).

[0036] Those skilled in the art can understand that since the data loading may affect the consistency and integrity of the target database, a transaction processing mechanism needs to be used to ensure the atomicity and consistency of the data. Place the data loading operation in a transaction. If the loading of one piece of data fails, roll back the entire transaction to ensure data consistency. In addition, when performing data loading, use the locking method to ensure data consistency and integrity. In a high-concurrency environment, by setting the lock granularity and the transaction isolation level, both ensure data consistency and reduce the impact on system performance.

[0037] Step 150: After the data loading is completed, obtain the amount of loaded data and the expected amount of data, verify whether the synchronization data is accurate and complete, and obtain the verification result.

[0038] This step can be specifically as follows: After the data loading is completed, obtain the amount of loaded data and the expected amount of data; When the amount of loaded data is less than the expected amount of data, determine that there is a network interruption or server downtime problem, and determine that the verification result is: the synchronization data is inaccurate and incomplete; When the amount of loaded data is greater than the expected amount of data, further compare whether the loaded data exists in the original synchronization data in the source database. When the loaded data exists in the original synchronization data in the source database, determine that the verification result is: synchronization completed; When the amount of loaded data is equal to the expected amount of data, determine that the verification result is: synchronization completed.

[0039] Those skilled in the art can understand that this application determines whether the synchronization is successful by comparing whether the loaded data volume matches the expected data volume. If the actually loaded data volume is less than the expected data volume, the synchronization may fail due to problems such as network interruption or server downtime; if the actually loaded data volume is greater than the expected data volume, the synchronization may fail due to duplicate data or incorrect processing. It is also determined whether the synchronization is successful by comparing whether the loaded data content is the same as the original synchronization data in the source database. If it is found that some of the loaded data is different from the original synchronization data in the source library, the synchronization may fail due to problems such as data conversion errors or network latency; if it is found that some of the loaded data is exactly the same as the original data in the source library, it indicates that the synchronization has been successfully completed.

[0040] In addition, this application Figure 2 provides a library-level synchronization system based on data synchronization for the embodiments of this application. As Figure 2 shown, the system provided by the embodiments of this application mainly includes: A determination module 210, configured to obtain data information of the source database and the target database that need data synchronization; wherein, the data information at least includes: data structure, field type, and data update frequency; when the data structures or field types of the source database and the target database are different, based on the data structure or field type, determine the corresponding data conversion program.

[0041] The determination module 210 includes a conversion unit, configured to, when the data structures of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the data structure of the source database into the data structure of the target database; when the field types of the source database and the target database are different, select a data conversion program from the preset data conversion program database that converts the field type of the source database into the field type of the target database.

[0042] An acquisition module 220, configured to determine the synchronization time for synchronizing the data in the source database to the target database according to the data update frequency of the source database and the real-time requirement; according to the synchronization time, acquire synchronization data from the source database, and at the same time add an exclusive lock to the synchronization data to be acquired.

[0043] The acquisition module 220 includes an acquisition unit, configured to, when there is a real-time requirement, determine that the synchronization time of the source database is real-time synchronization; when there is no real-time requirement, determine that the synchronization time of the source database is scheduled synchronization; further, according to the preset relationship between the data update frequency and the scheduled interval, determine the scheduled interval corresponding to the current data update frequency, and then obtain the synchronization time; Obtain whether the source database supports incremental changes. When it supports incremental changes, obtain the incremental data from the previous synchronization time to the current synchronization time as the synchronization data; When it does not support incremental changes, obtain the synchronization data by means of full extraction.

[0044] The unified module 230 is used to call the data conversion program when the synchronization data is collected, and then unify the data structure and field type of the synchronization data into the data structure and field type of the target database.

[0045] The acquisition module 240 is used to determine the data loading method for loading the synchronization data into the target database based on the data volume of the synchronization data, and at the same time use the transaction processing mechanism to obtain the synchronization data through the data loading method, and set the exclusive lock and the isolation level of the current transaction.

[0046] The verification module 250 is used to obtain the loaded data volume and the expected data volume after the data loading is completed, verify whether the synchronization data is accurate and complete, and obtain the verification result.

[0047] In addition, the embodiment of the present application also provides a non-volatile computer storage medium, on which executable instructions are stored. When the executable instructions are executed, a library-level synchronization method based on data synchronization as described above is implemented.

[0048] So far, the technical solutions of the present disclosure have been described in combination with multiple previous embodiments. However, it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principle of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. A library-level synchronization method based on data synchronization, characterized in that: The method comprises: Acquire data information of a source database and a target database that need data synchronization; wherein the data information at least includes: data structure, field type and data update frequency; when the data structure or field type of the source database and the target database are different, determine a corresponding data conversion program based on the data structure or field type; Determine the synchronization time for synchronizing data from the source database to the target database based on the data update frequency and real-time requirements of the source database; collect synchronization data from the source database according to the synchronization time, and add an exclusive lock to the synchronization data to be collected; When the synchronization data is collected, the data conversion program is called to unify the data structure and field type of the synchronization data into the data structure and field type of the target database; Based on the amount of synchronized data, determine the data loading method for loading synchronized data into the target database, use the transaction processing mechanism to obtain synchronized data through the data loading method, and set the exclusive lock and isolation level of the current transaction; When the data loading is completed, obtain the amount of loaded data and the expected amount of data, verify whether the synchronized data is accurate and complete, and obtain the verification result.

2. The library-level synchronization method based on data synchronization according to claim 1, characterized in that: When the data structure or field type of the source database and the target database are different, the corresponding data conversion program is determined based on the data structure or field type, including: When the data structures of the source database and the target database are different, a data conversion program for converting the data structure of the source database into the data structure of the target database is selected from a preset data conversion program database; When the field types of the source database and the target database are different, a data conversion program for converting the field type of the source database into the field type of the target database is selected from a preset data conversion program database.

3. The library-level synchronization method based on data synchronization according to claim 1, characterized in that: Determine the synchronization time for synchronizing data from the source database to the target database based on the data update frequency and real-time requirements of the source database; According to the synchronization time, synchronized data is collected from the source database, and an exclusive lock is added to the synchronized data to be collected, including: When there is a real-time requirement, the synchronization time of the source database is determined to be real-time synchronization; When there is no real-time requirement, the synchronization time of the source database is determined to be scheduled synchronization; Then, according to the preset relationship between the data update frequency and the timing interval, the timing interval corresponding to the current data update frequency is determined, and the synchronization time is obtained; Get whether the source database supports incremental changes. If it supports incremental changes, get the incremental data from the last synchronization time to the current synchronization time as the synchronization data; When incremental changes are not supported, full extraction is used to obtain synchronized data.

4. The library-level synchronization method based on data synchronization according to claim 1, characterized in that: Before unifying the data structure and field type of the synchronization data into the data structure and field type of the target database, the method further includes: Adopt intelligent error value correction method to remove invalid and duplicate information in synchronized data; Based on the preset error pattern library, the error values ​​are identified from the synchronization data, and the error values ​​are corrected to the preset reasonable values, and recorded in the preset correction log.

5. The library-level synchronization method based on data synchronization according to claim 1, characterized in that: Based on the amount of synchronized data, determine the data loading method for loading synchronized data into the target database, including: When the amount of synchronized data is less than the preset amount of data, determining that the data loading method for loading synchronized data into the target database is to load data one by one; When the amount of synchronized data is greater than or equal to the preset amount of data, it is determined that the data loading mode for loading the synchronized data into the target database is batch loading.

6. The library-level synchronization method based on data synchronization according to claim 1, characterized in that: After the data is loaded, obtain the amount of loaded data and the expected amount of data, verify whether the synchronized data is accurate and complete, and obtain the verification results, including: When data loading is complete, obtain the amount of loaded data and the expected amount of data; When the amount of loaded data is less than the expected amount of data, it is determined that there is a network interruption or server downtime problem, and the verification result is determined to be: the synchronized data is inaccurate and incomplete; When the amount of loaded data is greater than the expected amount of data, further compare whether the loaded data exists in the original synchronized data in the source database. If the loaded data exists in the original synchronized data in the source database, determine that the verification result is: synchronization is completed; When the amount of loaded data is equal to the expected amount of data, the verification result is determined to be: synchronization is completed.

7. A library-level synchronization system based on data synchronization, characterized in that: The system comprises: A determination module is used to obtain data information of a source database and a target database that need data synchronization; wherein the data information at least includes: data structure, field type and data update frequency; when the data structure or field type of the source database and the target database are different, a corresponding data conversion program is determined based on the data structure or field type; The acquisition module is used to determine the synchronization time for synchronizing the data in the source database to the target database according to the data update frequency and real-time requirements of the source database; collect synchronization data from the source database according to the synchronization time, and add an exclusive lock to the synchronization data to be collected; A unification module is used to call a data conversion program when the synchronization data is collected, thereby unifying the data structure and field type of the synchronization data into the data structure and field type of the target database; The acquisition module is used to determine the data loading method for loading the synchronized data into the target database based on the amount of synchronized data, and use the transaction processing mechanism to obtain the synchronized data through the data loading method, and set the exclusive lock and the isolation level of the current transaction; The verification module is used to obtain the amount of loaded data and the expected amount of data after the data loading is completed, verify whether the synchronized data is accurate and complete, and obtain the verification result.

8. The library-level synchronization system based on data synchronization according to claim 7, characterized in that: The determination module includes a conversion unit, When the data structures of the source database and the target database are different, a data conversion program is selected from a preset data conversion program database to convert the data structure of the source database into the data structure of the target database; When the field types of the source database and the target database are different, a data conversion program for converting the field type of the source database into the field type of the target database is selected from a preset data conversion program database.

9. The library-level synchronization system based on data synchronization according to claim 7, characterized in that: The acquisition module includes an acquisition unit, Used to determine the synchronization time of the source database as real-time synchronization when there is a real-time requirement; When there is no real-time requirement, the synchronization time of the source database is determined to be scheduled synchronization; Then, according to the preset relationship between the data update frequency and the timing interval, the timing interval corresponding to the current data update frequency is determined, and the synchronization time is obtained; Get whether the source database supports incremental changes. If it supports incremental changes, get the incremental data from the last synchronization time to the current synchronization time as the synchronization data; When incremental changes are not supported, full extraction is used to obtain synchronized data.

10. A non-volatile computer storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the library-level synchronization method based on data synchronization as described in any one of claims 1-6 is implemented.

Images