Data verification method of database synchronization process, storage medium and device
By setting reference columns and updating verification benchmark parameters during database synchronization, the problem of low verification efficiency during database synchronization is solved, achieving efficient and accurate data verification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CETC JINCANG (BEIJING) TECH CO LTD
- Filing Date
- 2022-10-12
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies suffer from low data verification efficiency and excessive time consumption during database synchronization, making it impossible to complete the verification of all tables in a short period of time, resulting in low verification efficiency.
Set a reference column in the data table to be verified, obtain the reference value for data updates, and extract the maximum value to update the verification benchmark parameter as the update time increases, thereby determining the scope of synchronous verification.
By setting reference columns and updating verification baseline parameters, verification time can be effectively reduced, verification efficiency improved, the range of verifiable options expanded, and verification accuracy enhanced.
Smart Images
Figure CN115599861B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of database technology, in particular to a data checking method for database synchronization process, a storage medium and an equipment. BACKGROUND
[0002] In real-time data synchronization using data synchronization tools, there are three stages. In the first stage, the initialization loading of inventory data is performed to obtain the basic point of data synchronization. In the second stage, the incremental data synchronization is performed based on the synchronization basic point established by the initialization data loading. In the third stage, the source data and the target data of the data synchronization are compared and checked periodically to confirm that there is no data loss in the data synchronization process.
[0003] In the prior art, the method for checking data is usually to query and load all the tables to be checked from the source database and the target database into the memory, and then use a specific algorithm to check in the memory. Such a cycle is repeated until all the tables to be compared are compared.
[0004] However, the number of tables in the database and the amount of data in the table are very large, which results in a particularly long time for querying and checking a single table, so that the checking action of all tables cannot be completed in a short time, and the checking efficiency is low. SUMMARY
[0005] An object of the present application is to provide a data checking method for database synchronization process, a storage medium and an equipment which can solve any of the above problems.
[0006] A further object of the present application is to improve the accuracy of the checking.
[0007] In particular, the present application provides a data checking method for database synchronization process, comprising:
[0008] setting a reference column in the data table to be checked;
[0009] obtaining the data update in the data table and setting the reference value of the reference column corresponding to the updated data accordingly, so that the reference value of the reference column corresponds to the update time and increases with the update time;
[0010] extracting the maximum value from the reference value of the reference column at the moment when the synchronization checking of the data table is started;
[0011] After this checking, updating the checking reference parameter according to the maximum value, the checking reference parameter being used to determine the range of the synchronization checking.
[0012] Optionally, the checking reference parameter comprises a first checking reference value and a second checking reference value,
[0013] The step of updating the check reference value according to the maximum value comprises:
[0014] The first check reference value and the second check reference value are compared, and the maximum value is used to replace the smaller one of the first check reference value and the second check reference value.
[0015] Optionally, the step of synchronously checking the data table comprises:
[0016] The data corresponding to the reference value of the reference column which is greater than or equal to the first check reference value and less than or equal to the second check reference value is checked.
[0017] Optionally, the step of synchronously checking the data table comprises:
[0018] The data corresponding to the reference value of the reference column which is greater than the first check reference value or greater than the second check reference value is checked.
[0019] Optionally, the check reference parameter comprises a reference value,
[0020] The step of updating the check reference value according to the maximum value comprises:
[0021] The reference value is replaced by the maximum value.
[0022] Optionally, the step of synchronously checking the data table comprises:
[0023] The data corresponding to the reference value of the reference column which is greater than the reference value is checked.
[0024] Optionally, the step of setting a reference column in the data table to be checked comprises:
[0025] It is judged whether the data table has a self-owned data column which can realize the function of the reference column, if yes, the self-owned data column is taken as the reference column, if no, the reference column is created in the data table.
[0026] Optionally, the step of setting a reference column in the data table to be checked comprises:
[0027] A system table is created in the meta information library, and the system table is used to record the table name of the data table and the check reference parameter.
[0028] According to another aspect of the present application, there is also provided a machine readable storage medium having stored thereon a machine executable program which, when executed by a processor, implements the data checking method of the database synchronization process according to any one of the above.
[0029] According to still another aspect of the present application, there is also provided a computer device comprising a memory, a processor, and a machine executable program stored on the memory and running on the processor, and the processor implements the data checking method of the database synchronization process according to any one of the above aspects when executing the machine executable program.
[0030] The data checking method of the database synchronization process according to the present application sets a reference column in the data table to be checked; acquires data updates in the data table and sets the reference values of the reference column corresponding to the updated data accordingly, so that the reference values of the reference column correspond to the update times and increase with the update times; extracts the maximum value from the reference values of the reference column at the time when the data table starts the synchronization checking; and updates the checking reference parameter according to the maximum value after this checking, the checking reference parameter being used to determine the range of the synchronization checking. That is, the maximum value of the reference values of the reference column is selected at each checking, and the checking reference parameter is updated according to the maximum value by setting the reference column for the specific data table. Therefore, not only the specific data table can be checked in the checking process, but also the new incremental data after each checking can be checked. Thus, the checking time is effectively reduced, and the checking efficiency is improved.
[0031] Further, the data checking method of the database synchronization process according to the present application sets the checking reference parameter as the first checking reference value and the second checking reference value, so that the data corresponding to the reference values of the reference column greater than or equal to the first checking reference value and less than or equal to the second checking reference value can be selected for checking in the checking process, or the data corresponding to the reference values of the reference column greater than the first checking reference value or greater than the second checking reference value can be selected for checking. Therefore, not only the data not changing between the first checking reference value and the second checking reference value can be checked, but also the selectable range of the checking is expanded.
[0032] The above and other objects, advantages and features of the present application will become more apparent from the following detailed description of some embodiments thereof, when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS
[0033] Some embodiments of the present application will now be described in detail in connection with the accompanying drawings, in which:
[0034] Figure 1 is a schematic flow chart of the data checking method of the database synchronization process according to an embodiment of the present application;
[0035] Figure 2is a schematic flow chart of the step of setting a reference column in a data table to be verified according to an embodiment of the present application;
[0036] Figure 3 is a schematic flow chart of the synchronization verification process of a data table in the data verification method of the database synchronization process according to an embodiment of the present application;
[0037] Figure 4 is a schematic flow chart of the synchronization verification process of a data table in the data verification method of the database synchronization process according to another embodiment of the present application;
[0038] Figure 5 is a schematic flow chart of the data verification method of the database synchronization process according to another embodiment of the present application;
[0039] Figure 6 is a schematic diagram of a machine readable storage medium according to an embodiment of the present application;
[0040] Figure 7 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0041] Those skilled in the art should understand that the embodiments described below are only a part of the embodiments of the present application, and are not intended to limit the protection scope of the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the protection scope of the present application.
[0042] It should be noted that the logic and / or steps represented in the flow chart or otherwise described herein, for example, can be considered as an ordered listing of executable instructions for implementing logical functions, and can be specifically embodied in any computer readable medium for use by an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices.
[0043] Figure 1 is a schematic flow chart of the data verification method according to an embodiment of the present application.
[0044] As shown in Figure 1 , in the present embodiment, the data verification method of the database synchronization process can generally include:
[0045] Step S102, setting a reference column in a data table to be verified.
[0046] Step S104, obtaining the data update in the data table, and setting the reference value of the reference column corresponding to the updated data accordingly, so that the reference value of the reference column corresponds to the update time and increases with the update time.
[0047] Step S106, extracting the maximum value from the reference values of the reference column at the time when the synchronization check of the data table is started.
[0048] Step S108, updating the check reference parameter according to the maximum value after the current check, the check reference parameter being used to determine the range of the synchronization check.
[0049] Specifically, after the reference column is set for the data table, each reference value in the reference column corresponds to a group of data in the data table. When the data in the data table is updated, the reference value corresponding to the updated data changes accordingly, so that the reference value increases with the update time.
[0050] Specifically, the reference value can be the update time, so as to correspond to the update time of the data. Alternatively, the reference value can also be a numerical value, for example, the initial reference value of the reference column is 0, when one group of data in the data table is updated, the reference value corresponding to the updated data becomes 1. Then, when another group of data in the data table is updated, the reference value corresponding to the updated data becomes 2, and so on, so that the numerical value increases with the update time.
[0051] Specifically, the step of setting the reference column in the data table to be checked comprises: creating a system table in a meta information library, the system table being used to record the table name of the data table and the check reference parameter. That is, the system table can record the table name of the data table and the check reference parameter corresponding thereto, so as to facilitate the unified maintenance and search of the data table and the check reference parameter. The meta information library is a database used to record some auxiliary information in the data check.
[0052] Figure 2 is a schematic flow chart of the step of setting the reference column in the data table to be checked according to one embodiment of the present application.
[0053] As shown in Figure 2 , the step of setting the reference column in the data table to be checked can generally comprise:
[0054] Step S202, judging whether the data table has a self-owned data column capable of realizing the function of the reference column, if yes, executing step S204, if no, executing step S206.
[0055] Step S204, taking the self-owned data column as the reference column.
[0056] Step S206, creating the reference column in the data table.
[0057] Specifically, if the data table has a self-owned data column that can realize the function of the reference column, for example, the data table originally has a column recording the update time or other column with values increasing with the update time, the self-owned data column in the data table can be directly used as the reference column. If the data table does not have a data column that can realize the function of the reference column, a reference column is newly created in the data table. Preferably, the newly created reference column is a hidden column, and the reference value is the time of data update.
[0058] Figure 3 is a schematic flow chart of a data table in a data checking method of a database synchronization process according to an embodiment of the present application. Further, the checking reference parameters include a first checking reference value and a second checking reference value.
[0059] As shown in Figure 3 , the step of synchronously checking the data table in the data checking method can generally include:
[0060] Step S302, at the time when the data table starts the synchronous checking, the maximum value is extracted from the reference values of the reference column.
[0061] Step S304, the data table is checked according to the first checking reference value and the second checking reference value.
[0062] Step S306, the first checking reference value and the second checking reference value are compared, and the maximum value is used to replace the smaller one of the first checking reference value and the second checking reference value.
[0063] Specifically, at the time when the data table starts the synchronous checking, the maximum value of the reference values of the reference column is extracted. If the reference value is the update time, it is the latest time. If the reference value is a numerical value, it is the maximum numerical value.
[0064] In the process of synchronously checking the data table, the data table is checked according to the first checking reference value and the second checking reference value. Specifically, three ways can be selected for checking:
[0065] First, the data corresponding to the reference value of the reference column that is greater than or equal to the first checking reference value and less than or equal to the second checking reference value is checked.
[0066] Second, the data corresponding to the reference value of the reference column that is greater than the first checking reference value is checked.
[0067] Third, the data corresponding to the reference value of the reference column that is greater than the second checking reference value is checked.
[0068] For example, taking the specific checking process as an example, at the beginning of the checking, the first checking reference value is a, the second checking reference value is b, and a is smaller than b. At the same time, at the beginning of the checking, the maximum value c is extracted from the reference values of the reference column. Therefore, the data corresponding to the reference values of the reference column which are greater than a can be selected for checking. Alternatively, the data corresponding to the reference values of the reference column which are greater than b can be selected for checking. Alternatively, the data corresponding to the reference values of the reference column which are greater than or equal to a and less than or equal to b can be selected for checking. Alternatively, two or three of the above-mentioned ways can be selected for checking.
[0069] After the checking is completed, the smaller one of a and b is replaced by c, i.e., a is replaced by c.
[0070] Therefore, at the beginning of the next checking, the first reference value is c, the second reference value is b, and b is smaller than c. At the same time, at the beginning of the checking, the maximum value d is extracted from the reference values of the reference column. Therefore, the data corresponding to the reference values of the reference column which are greater than b can be selected for checking. Alternatively, the data corresponding to the reference values of the reference column which are greater than c can be selected for checking. Alternatively, the data corresponding to the reference values of the reference column which are greater than or equal to b and less than or equal to c can be selected for checking. Alternatively, two or three of the above-mentioned ways can be selected for checking.
[0071] After the checking is completed, the smaller one of b and c is replaced by d, i.e., b is replaced by d. Therefore, for the next checking, the first reference value is c, and the second reference value is d. The checking process is executed in this way repeatedly.
[0072] In the scheme of the embodiment, by setting the reference column for the specific data table, the maximum value of the reference values of the reference column is selected at each time of checking, and the checking reference parameters are updated according to the maximum value. Therefore, in the checking process, not only the specific data table can be checked, but also the new incremental data after each time of checking can be checked. Thus, the checking time is effectively reduced, and the checking efficiency is improved.
[0073] Further, by setting the checking reference parameters as the first checking reference value and the second checking reference value, the data corresponding to the reference values of the reference column which are greater than or equal to the first checking reference value and less than or equal to the second checking reference value can be selected for checking, or the data corresponding to the reference values of the reference column which are greater than the first checking reference value or greater than the second checking reference value can be selected for checking. Therefore, not only the data which does not change between the first checking reference value and the second checking reference value can be checked, but also the accuracy of the checking is improved. Moreover, the selectable range of the checking is expanded.
[0074] It should be noted that after the reference column is set, the initial maximum value in the reference column is recorded as the first check reference value. At this time, the second check reference value is empty. At the first time of checking thereafter, the maximum value in the reference value of the reference column is extracted. After the checking, because the second check reference value is empty, the first check reference value is greater than the second check reference value, and the maximum value is recorded as the second check reference value.
[0075] Figure 4 is a schematic flow chart of a data checking method in a database synchronization process according to another embodiment of the present application, in which a synchronization checking process is performed on a data table. In the embodiment, the check reference parameter includes a reference value.
[0076] As shown in Figure 4 , the step of performing a synchronization checking process on a data table in the data checking method can generally include:
[0077] Step S402: At the time when the synchronization checking process is started on the data table, the maximum value is extracted from the reference value of the reference column.
[0078] Step S404: The data whose reference value is greater than the reference value of the reference column is checked.
[0079] Step S406: The reference value is replaced by the maximum value.
[0080] Specifically, in the embodiment, the check reference parameter only has one reference value, the reference value is updated each time, and the data greater than the reference value is checked, so that the new incremental data can be checked, thereby improving the checking efficiency.
[0081] For example, taking a specific checking process as an example, at the beginning of the checking, the reference value is m, and at the same time, the maximum value n is extracted from the reference value of the reference column. Therefore, the data whose reference value is greater than m in the reference column can be checked.
[0082] After the checking is completed, m is replaced by n.
[0083] Therefore, at the beginning of the next checking, the reference value is n. At the same time, at the beginning of the checking, the maximum value o is extracted from the reference value of the reference column. Therefore, the data whose reference value is greater than n in the reference column can be checked.
[0084] After the checking is completed, n is replaced by o. Therefore, for the next checking, the reference value is o. The checking process is executed in this way.
[0085] Figure 5 is a schematic flow chart of a data checking method in a database synchronization process according to another embodiment of the present application.
[0086] As shown in Figure 5As shown, in the present embodiment, the data checking method of the database synchronization process can generally include:
[0087] Step S502, setting a reference column in the data table to be checked.
[0088] Step S504, judging whether the data table has a self-owned data column capable of realizing the function of the reference column, if yes, executing step S508, if no, executing step S506.
[0089] Step S506, creating a reference column in the data table. Step S508 is executed.
[0090] Step S508, obtaining the reference column and the checking reference parameter.
[0091] Specifically, in the process of setting the reference column, if the data table has a self-owned data column capable of realizing the function of the reference column, for example, the data table originally has a record update time column or other column with a value increasing with the update time. Then, the self-owned data column in the data table can be directly used as the reference column. If the data table does not have a data column capable of realizing the function of the reference column, a reference column is newly created in the data table. Preferably, the newly created reference column is a hidden column, and the reference value is the time of data update. In the case of a reference table, the reference table and the checking reference parameter are obtained, i.e., the reference table name and the checking reference parameter are recorded in the system table under the corresponding data table name.
[0092] Step S510, judging whether the checking reference parameter is empty, if yes, executing step S512, if no, executing step S514.
[0093] Step S512, performing full checking on the data table. Step S516 is executed.
[0094] Step S514, determining the checking range according to the checking reference parameter, and performing checking. Here, the description of the checking process in the above embodiment is referred to, and will not be repeated here.
[0095] Step S516, updating the checking reference parameter. Here, the description of updating the checking reference parameter according to the maximum value in the above embodiment is referred to, and will not be repeated here.
[0096] Specifically, when the checking reference parameter is obtained, if the checking reference parameter is empty, the first checking reference parameter has not been recorded. Then, the data table is checked once in full, the maximum value of the reference value in the reference column at this time is determined as the first checking reference parameter. In the case of the checking reference parameter being the first checking reference value and the second checking reference value, the first checking reference value is used. In the case of the checking reference parameter being the reference value, the reference value is used.
[0097] If the check reference parameter exists, the check range can be determined according to the check reference parameter, and the check is performed. After the check, the check reference parameter is also updated. Thereafter, the cycle can be performed.
[0098] The application further provides a machine readable storage medium and a computer device. Figure 6 Fig. 1 is a schematic diagram of a machine readable storage medium according to an embodiment of the application. Figure 7 Fig. 2 is a schematic diagram of a computer device according to an embodiment of the application.
[0099] The machine readable storage medium 10 has a machine executable program 11 stored thereon, and the machine executable program 11, when executed by a processor, implements the data check method of the database synchronization process of any of the above embodiments.
[0100] The computer device 20 can include a memory 210, a processor 220, and a machine executable program 11 stored on the memory 210 and running on the processor 220, and the processor 220, when executing the machine executable program 11, implements the statement processing method of the database system of any of the above embodiments.
[0101] For the description of the present embodiment, the machine readable storage medium 10 can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer disks (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disk read-only memories (CDROMs). In addition, the computer readable medium 10 can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example by optical scanning of the paper or other medium, followed by electronic conversion, interpretation or processing, if necessary, in other suitable manner, and then stored in a computer memory.
[0102] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.
[0103] The computer device 20 can include a processor 220 adapted to execute instructions stored in memory 210, which provides temporary storage for operations of the instructions during runtime. The processor 220 can be a single core processor, multi-core processor, computing cluster, or any number of other configurations. The memory 210 can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.
[0104] To this end, those skilled in the art will recognize that the foregoing description has been presented for the sake of illustration and description only. It is not intended to be exhaustive or to limit the application to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the application encompass all technically possible combinations of features covered by the claims.
Claims
1. A data verification method for a database synchronization process, comprising: Set a reference column in the data table that needs to be validated; The data update in the data table is obtained, and the reference value of the reference column corresponding to the updated data is set accordingly, so that the reference value of the reference column corresponds to the update time and increases with the update time; When the data table is synchronized and verified, the maximum value is extracted from the reference value of the reference column; After this verification, the verification benchmark parameter is updated according to the maximum value. The verification benchmark parameter is used to determine the range of synchronous verification. The verification benchmark parameter includes a first verification benchmark value and a second verification benchmark value, and the step of updating the verification benchmark value according to the maximum value includes: Compare the first verification benchmark value and the second verification benchmark value, and replace the smaller of the first verification benchmark value and the second verification benchmark value with the maximum value.
2. The data verification method for the database synchronization process according to claim 1, wherein, The steps for synchronizing and validating the data table include: The data corresponding to the reference values of the reference column that are greater than or equal to the smaller of the first verification benchmark value and the second verification benchmark value and less than or equal to the larger of the first verification benchmark value and the second verification benchmark value are verified.
3. The data verification method for the database synchronization process according to claim 1, wherein, The steps for synchronizing and validating the data table include: The data corresponding to the reference value of the reference column that is greater than the first verification benchmark value or the second verification benchmark value are verified.
4. The data verification method for the database synchronization process according to claim 1, wherein, The step of setting a reference column in the data table to be verified includes: Determine whether the data table has its own data column that can implement the function of the reference column. If yes, use the own data column as the reference column; otherwise, create the reference column in the data table.
5. The data verification method for the database synchronization process according to claim 1, wherein, The step of setting a reference column in the data table to be verified includes: A system table is created in the metadata database. The system table is used to record the table name and verification benchmark parameters of the data table.
6. A machine-readable storage medium having a machine-executable program stored thereon, wherein the machine-executable program, when executed by a processor, implements a data verification method for a database synchronization process according to any one of claims 1 to 5.
7. A computer device comprising a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements a data verification method for a database synchronization process according to any one of claims 1 to 5.
Citation Information
Patent Citations
Data synchronization verification method and device
CN113064909A