A method, device, computer device and storage medium for incremental data synchronization

Abstract

The application discloses a kind of incremental data synchronization method, device, computer equipment and storage medium, wherein, incremental data synchronization method part includes: recording the data of the first database of upstream business system at target time;Detect whether there is incremental mark in first database;If it is detected that there is incremental mark in first database, incremental data is identified;Incremental data is synchronized to intermediate database;Intermediate database is compared with the second database of downstream business system, and the incremental data that needs to be updated is obtained;For the incremental data that needs to be updated, set corresponding update mark;According to update mark, the data in second database is updated synchronously.Introduce intermediate database, and incremental data is synchronized to intermediate database, so that the data of upstream business system and the data of downstream business system are more consistent, solve the problem that the data of upstream business system and the data of downstream business system exist synchronization exception or missing data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method and device for incremental data synchronization, a computer device and a storage medium. BACKGROUND

[0002] With the rapid development of science and technology, the Internet technology is increasingly closely related to daily life and business life. At present, a data warehouse is used to store data of various business systems, each system has its own business logic, and the significance of the data warehouse lies in integrating data of all systems.

[0003] At present, data loading is implemented through Extract-Transform-Load (ETL), which is a process of extracting data from a source end, transforming the data, and loading the data to a destination end. Each table of a business system has primary key data. When primary key data of an upstream business system is modified, a table entering the data warehouse needs to be modified. When upstream data is deleted, the table of the data warehouse also needs to be synchronously deleted. When upstream data is newly added, the table of the data warehouse also needs to be synchronously added. That is, the extraction of data mainly includes three parts: deletion, addition, and modification. During incremental data synchronization, especially when a large number of data records are processed and the amount of data of each record is large, the existing system needs to develop deletion, addition, and modification functions in ETL development every time a table is synchronized, which is prone to synchronization exceptions or missing data when a large number of data records are processed. SUMMARY

[0004] The present application provides a method and device for incremental data synchronization, a computer device and a storage medium to solve the problem of incremental data synchronization.

[0005] A method for incremental data synchronization is applied to a business system, the business system includes an upstream business system and a downstream business system, the upstream business system and the downstream business system are systems having associated data, and the method includes:

[0006] Recording data of a first database of the upstream business system at a target time, wherein the target time is a cutoff time of a preset synchronization time interval;

[0007] Detecting whether there is an incremental mark in the first database, wherein the incremental mark is used to indicate identification information of data update;

[0008] If the incremental mark is detected in the first database, identifying incremental data according to the incremental mark;

[0009] synchronize the incremental data to an intermediate database;

[0010] compare the intermediate database with a second database of a downstream business system to obtain incremental data that needs to be updated, wherein the second database stores full data in the first database at a starting moment of a preset synchronization time interval;

[0011] set a corresponding update mark for the incremental data that needs to be updated;

[0012] synchronize and update data in the second database according to the update mark.

[0013] An apparatus for incremental data synchronization, comprising:

[0014] a recording module configured to record data of a first database of an upstream business system at a target moment, wherein the target moment is an ending moment of a preset synchronization time interval;

[0015] a detection module configured to detect whether there is an incremental mark in the first database, wherein the incremental mark is identification information indicating that there is data update;

[0016] an identification module configured to identify incremental data according to the incremental mark if it is detected that there is the incremental mark in the first database;

[0017] a first synchronization module configured to synchronize the incremental data to an intermediate database;

[0018] a comparison module configured to compare the intermediate database with a second database of a downstream business system to obtain incremental data that needs to be updated, wherein the second database stores full data in the first database at a starting moment of a preset synchronization time interval;

[0019] a setting module configured to set a corresponding update mark for the incremental data that needs to be updated;

[0020] a second synchronization module configured to synchronize and update data in the second database according to the update mark.

[0021] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements steps of the method for incremental data synchronization when executing the computer program.

[0022] A computer readable storage medium, storing a computer program, wherein the computer program is executable on a processor to implement steps of the method for incremental data synchronization.

[0023] The method, device, computer device and storage medium for incremental data synchronization, in one of the implemented schemes, synchronize the incremental data to an intermediate database, compare the intermediate database with a second database of a downstream business system, obtain incremental data that needs to be updated, set a corresponding update mark for the incremental data that needs to be updated, and finally synchronize and update data in the second database of the downstream business system according to the update mark. The intermediate database is introduced, and the incremental data is synchronized to the intermediate database, so that the data of the upstream business system and the data of the downstream business system are more consistent, and the problem of synchronization exception or missing data between the data of the upstream business system and the data of the downstream business system is solved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the present application, the drawings needed to be used in the description of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a schematic diagram of an application environment of the method for incremental data synchronization in an embodiment of the present application;

[0026] Figure 2 is a flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0027] Figure 3 is another flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0028] Figure 4 is another flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0029] Figure 5 is another flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0030] Figure 6 is another flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0031] Figure 7 is another flowchart of the method for incremental data synchronization in an embodiment of the present application;

[0032] Figure 8 is a principle block diagram of the device for incremental data synchronization in an embodiment of the present application;

[0033] Figure 9 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0035] The method for incremental data synchronization provided by the present application can be applied in the application environment as shown in Figure 1 , wherein the server communicates with the client through the network.

[0036] The method for incremental data synchronization comprises the following steps:

[0037] In an embodiment, as shown in Figure 2 , a method for incremental data synchronization is provided and applied in a business system, wherein the business system comprises an upstream business system and a downstream business system, and the upstream business system and the downstream business system are systems with associated data. The method is applied in the server in Figure 1 , and comprises the following steps:

[0038] S10: recording the data of a first database of the upstream business system at a target time, wherein the target time is the end time of a preset synchronization time interval.

[0039] The preset synchronization time interval is a preset time interval, for example, the preset synchronization time interval is 30 minutes, 60 minutes, 24 hours, etc.; and the end time means stopping at a certain time.

[0040] The first database in the upstream business system at the target time point stores user information, which can include user ID card, user name, user phone number, user gender, user hobby, etc.

[0041] If the starting time point is 12:00 and the preset synchronization time interval is 30 minutes, the target time point is the cutoff time point of 12:30, and it can be known that the first database of the upstream business system stores data at the cutoff time point of 12:30. If the starting time point is 12:00 and the preset synchronization time interval is 60 minutes, the target time point is the cutoff time point of 13:00, and it can be known that the first database of the upstream business system stores data at the cutoff time point of 13:00.

[0042] S20: detecting whether there is an incremental mark in the first database, wherein the incremental mark is used to indicate identification information of data update.

[0043] The incremental mark is a label automatically recognized and marked on the information to be updated by the upstream business system, and is used to indicate identification information of data update. Optionally, the first data can be an oracle database. In the oracle database, all information of a table is recorded in a basic table of the oracle itself, and there is a uniquely identified field in each table to control the uniqueness of each data. According to the oracle self-view, the primary key of the table can be queried, so that the case that the primary key in the oracle database is modified is marked with an incremental mark, thereby avoiding the problem of manually judging the primary key.

[0044] For example, the first database of the upstream business system at the cutoff time point of 12:30 stores the user ID card: 4419xxxxx10, the user name: A, the user phone number: 1355xxxx52, the user gender: male, and the user hobby: empty. It is detected whether there is an incremental mark in the first database.

[0045] S30: If it is detected that there is the incremental mark in the first database, incremental data is identified according to the incremental mark.

[0046] For example, the first database of the upstream business system at the cutoff time point of 12:30 stores the user ID card: 4419xxxxx10, the user name: A, the user phone number: 1355xxxx52, and the user gender: male. The user ID card, the user name, the user phone number, and the user hobby have the incremental mark. It is identified that the incremental data is the user ID card: 4419xxxxx10, the user name: A, the user phone number: 1355xxxx52, and the user hobby: empty according to the incremental mark.

[0047] S40: synchronizing the incremental data into the intermediate database.

[0048] For example, the incremental data of user ID: 4419xxxxx10, user name: A, user phone number: 1355xxxx52, and user hobby: null are synchronized into the intermediate database.

[0049] S50: comparing the intermediate database with a second database of a downstream business system to obtain incremental data that needs to be updated, wherein the second database stores full data in the first database at a starting time of a preset synchronization time interval.

[0050] The starting time represents a time before a certain time. For example, if the starting time is 12:00 and the preset synchronization time interval is 30 minutes, it can be known that the second database of the downstream business system stores data at the starting time of 12:00, that is, the second database stores full data in the first database at the starting time of 12:00. The second database of the downstream business system stores full data including user information. The second database stores full data in the first database at the starting time of the preset synchronization time interval. The user information includes user ID, user name, user phone number, user gender, and user hobby.

[0051] For example, the second database stores full data in the first database at the starting time of 12:00. The full data in the first database at the starting time of 12:00 includes user ID: 4312xxxxx10, user name: B, user phone number: null, user gender: male, and user hobby: music.

[0052] The intermediate database is compared with the second database by using ORACLE database technology to obtain incremental data that needs to be updated, which includes user ID, user name, user phone number, and user hobby. The incremental data that needs to be modified in the incremental data that needs to be updated includes user ID and user name. The incremental data that needs to be added includes user phone number. The incremental data that needs to be deleted includes user hobby.

[0053] S60: setting a corresponding update mark for the incremental data that needs to be updated.

[0054] The update mark includes a modification mark, an addition mark, and a deletion mark. For example, a corresponding modification mark is set for the user ID and user name that need to be modified. A corresponding addition mark is set for the user phone number that needs to be added. A corresponding deletion mark is set for the user hobby that needs to be deleted.

[0055] S70: synchronously updating the data in the second database according to the update mark.

[0056] For example, according to the modification mark, the user ID card 4312xxxxx10 is modified to 4419xxxxx10, and the user name B is modified to A. According to the increase mark, the user phone number is increased to 1355xxxx52. According to the deletion mark, the user hobby is deleted. The data stored in the second database of the downstream business system after the synchronous update is as follows: user ID card: 4419xxxxx10, user name: A, user phone number: 1355xxxx52, user gender: male, and user hobby: null.

[0057] In this embodiment, taking health insurance as an example, if the claim system is an upstream business system, then the health card business system, contract system, and security system associated with the claim system are corresponding downstream business systems. The upstream business system and the downstream business system are systems that have associated data. Understandably, the upstream business system and the downstream business system have systems of the same business type. If the data of the upstream business system is modified, then the corresponding downstream business systems need to be synchronously modified.

[0058] In Figure 2 In the corresponding embodiment, the incremental data is synchronized to the intermediate database, and the intermediate database is compared with the second database of the downstream business system to obtain the incremental data that needs to be updated. Then, the corresponding update mark is set for the incremental data that needs to be updated. Finally, the data in the second database of the downstream business system is synchronously updated according to the update mark. The introduction of the intermediate database makes the data of the upstream business system and the data of the downstream business system more consistent, and solves the problem of synchronization exception or missing data between the data of the upstream business system and the data of the downstream business system.

[0059] In an embodiment, as Figure 3 shown, after step S60 and before step S70, that is, after the corresponding update mark is set for the incremental data that needs to be updated and before the data in the second database of the downstream business system is synchronously updated according to the update mark, the method further includes the following steps:

[0060] S81: obtaining the original data corresponding to the incremental data that needs to be updated from the second database according to the update mark.

[0061] The update mark is used to indicate the identification information of the data modification, data increase, and data deletion.

[0062] For example, the incremental data needing to be updated includes the incremental data needing to be modified as the ID number and the user name, the incremental data needing to be added as the user phone number, and the incremental data needing to be deleted as the user hobby; the incremental data needing to be modified as the ID number and the user name and the incremental data needing to be added as the user phone number are obtained from the second database of the downstream business system according to the update mark, and the ID number corresponding to the original data of the user hobby needing to be deleted is 4312xxxxx10, the user name is B, the user phone number is empty, and the user hobby is music.

[0063] S82: Comparing the incremental field length of the incremental data needing to be updated with the original field length of the original data to obtain a comparison result of the incremental field length and the original field length.

[0064] Comparing the incremental field length of the incremental data needing to be updated with the original field length of the original data, if the incremental field length is longer than the original field length, a comparison result that the incremental field length is longer than the original field length is obtained; if the incremental field length is shorter than the original field length, a comparison result that the incremental field length is shorter than the original field length is obtained; if the incremental field length is shorter than the original field length, a comparison result that the incremental field length is shorter than the original field length is obtained; and if the incremental field length is the same as the original field length, a comparison result that the incremental field length is the same as the original field length is obtained.

[0065] For example, the ID number of the original data is 4312xxxxx10, and the original field length of the ID number of the original data is 11; the ID number of the incremental data needing to be updated is 4419xxxxx10, and the incremental field length of the ID number of the incremental data needing to be updated is 11; and comparing the incremental field length of the ID number of the incremental data needing to be updated with the original field length of the ID number of the original data, a comparison result that the incremental field length is the same as the original field length is obtained.

[0066] S83: Synchronously updating the original field length in the second database according to the comparison result of the incremental field length and the original field length.

[0067] According to the comparison result that the incremental field length is longer than the original field length, the original field length in the second database is increased to the same length as the incremental field length; according to the comparison result that the incremental field length is shorter than the original field length, the original field length in the second database is shortened to the same length as the incremental field length; and according to the comparison result that the incremental field length is the same as the original field length, the original field length in the second database remains unchanged.

[0068] In Figure 3In the corresponding embodiment, the length of the incremental field of the incremental data needing to be updated is compared with the length of the original field of the original data to obtain a comparison result of the length of the incremental field and the length of the original field, and the length of the original field in the second database is synchronously updated according to the comparison result of the length of the incremental field and the length of the original field, thereby solving the problem that the length of the field of the incremental data of the upstream business system is changed, the downstream business system is not timely notified, and the downstream system is reported as an error.

[0069] In an embodiment, as shown in FIG. 6, after step S60 and before step S70, that is, after the corresponding update mark is set for the incremental data needing to be updated, and before the data in the second database of the downstream business system is synchronously updated according to the update mark, the method further includes the following steps. Figure 4

[0070] S91: comparing the target table structure length of the first database with the original table structure length of the second database to obtain a comparison result of the target table structure length and the original table structure length.

[0071] The target table structure length of the first database is compared with the original table structure length of the second database, if the target table structure length is longer than the original table structure length, a comparison result that the target table structure length is longer than the original table structure length is obtained, if the target table structure length is shorter than the original table structure length, a comparison result that the target table structure length is shorter than the original table structure length is obtained, and if the target table structure length is the same as the original table structure length, a comparison result that the target table structure length is the same as the original table structure length is obtained.

[0072] For example, the first database includes a user gender without update, an ID number that has been modified, a user name that has been modified, and a user phone number that has been added, the target table structure length of the first database is 4, the second database includes a user gender without update, an ID number before modification, a user name before modification, and a user hobby before deletion, the original table structure length of the second database is 4, and the length of the target table structure length is compared with the length of the original table structure length to obtain a comparison result that the target table structure length is the same as the original table structure length.

[0073] S92: synchronously updating the original table structure length in the second database according to the comparison result of the target table structure length and the original table structure length.

[0074] ​According to the comparison result that the target table structure length is longer than the original table structure length, the original table structure length in the second database is increased to the same length as the target table structure length; according to the comparison result that the target table structure length is shorter than the original table structure length, the original table structure length in the second database is shortened to the same length as the target table structure length; and according to the comparison result that the target table structure length is the same as the original table structure length, the original table structure length in the second database remains unchanged.

[0075] In Figure 4 In the corresponding embodiment, the target table structure length of the first database is compared with the original table structure length of the second database to obtain a comparison result of the target table structure length and the original table structure length, and then the original table structure length in the second database is synchronously updated according to the comparison result of the target table structure length and the original table structure length, thereby solving the problem that the table structure length of the upstream changes and the downstream business system is not timely notified, resulting in an error of the downstream.

[0076] In an embodiment, as Figure 5 shown in FIG. 5, in step S50, that is, the comparison of the intermediate database and the second database of the downstream business system is performed to obtain the incremental data that needs to be updated, including the following steps.

[0077] S51: The comparison of the intermediate database and the second database of the downstream business system is performed to obtain the update type of the incremental data that needs to be updated, wherein the update type includes modification, addition or deletion.

[0078] S52: If the update type is modification, the incremental data that needs to be modified is obtained.

[0079] S53: If the update type is addition, the incremental data that needs to be added is obtained.

[0080] S54: If the update type is deletion, the incremental data that needs to be deleted is obtained.

[0081] For example, the intermediate database includes the identity card number, user name, user phone number and user hobby that need to be updated, and the second database includes the identity card number, user name, user phone number and user hobby before being updated. The comparison of the intermediate database and the second database of the downstream business system is performed to obtain the update type of the incremental data that needs to be updated.

[0082] If the update type is modification, the identity card number and user name that need to be modified are obtained; if the update type is addition, the user phone number that needs to be added is obtained; and if the update type is deletion, the user hobby that needs to be deleted is obtained.

[0083] In step S60, that is, the corresponding update mark is set for the incremental data that needs to be updated, including the following steps:

[0084] S61: setting a modified update mark for the incremental data that needs to be modified.

[0085] S62: setting an added update mark for the incremental data that needs to be added.

[0086] S63: setting a deleted update mark for the incremental data that needs to be deleted.

[0087] For example, setting a modified update mark for the identity card number that needs to be modified, setting an added update mark for the user phone number that needs to be added, and setting a deleted update mark for the user hobby that needs to be deleted.

[0088] In Figure 5 In the corresponding embodiment, different incremental data corresponding to different update types is obtained according to different update types, and the corresponding update mark is set according to the obtained incremental data, so that the data of the upstream business system and the data of the downstream business system are more consistent, and the problem of synchronization exception or missing data between the data of the upstream business system and the data of the downstream business system is solved.

[0089] In an embodiment, as Figure 6 shown, after step S40 and before step S50, that is, after the incremental data is synchronized to the intermediate database, before the intermediate database is compared with the second database of the downstream business system to obtain the incremental data that needs to be updated, the following steps are further included:

[0090] S101: adding a binary log into the intermediate database, wherein the change information of the incremental data in the second database is recorded in the binary log.

[0091] S102: obtaining the change information of the incremental data according to the binary log, wherein the change information includes the number of modified incremental data, the number of added incremental data, and the number of deleted incremental data.

[0092] The binary log can strictly record the time sequence and data snapshot of the dynamic change of the data in the second database, that is, the change process of the incremental data in the second database. The binary log always exists in the intermediate database, and as long as the incremental data changes, the binary log will record the detailed information of the change.

[0093] S103: sending the change information of the incremental data to the client for display.

[0094] The change information of the incremental data is sent to the client for display, which is beneficial to the client to detect the number of modified incremental data, the number of added incremental data and the number of deleted incremental data.

[0095] In Figure 6 In the corresponding embodiment, the binary log is added into the intermediate database, the change information of the incremental data is obtained according to the binary log, and finally the change information of the incremental data is sent to the client for display. The client can observe the number of data deletion, the number of data modification and the number of data addition, which is beneficial to strengthen the management of data.

[0096] In an embodiment, in step S40, that is, the synchronization of the incremental data into the intermediate database, includes the following steps:

[0097] S41: The incremental data is synchronized into the intermediate database by a comprehensive software Oracle Golden Gate which realizes real-time data integration and replication between heterogeneous IT environments.

[0098] The incremental data is synchronized into the intermediate database by a comprehensive software Oracle Golden Gate which realizes real-time data integration and replication between heterogeneous IT environments.

[0099] In an embodiment, the incremental data is directly synchronized into the intermediate database by Oracle Golden Gate, which improves the efficiency of subsequent data synchronization update.

[0100] In an embodiment, as Figure 7 shown, after step S70, that is, after the data in the second database of the downstream business system is synchronized and updated according to the update mark, the following steps are further included:

[0101] S111: When detecting the synchronization update exception, stop the operation of the synchronization update, and record the synchronization update progress.

[0102] Wherein, the detection of the synchronization update exception is accompanied by the whole incremental data synchronization process, that is, any problem in any step of the whole incremental data synchronization process can be regarded as a synchronization update exception. The synchronization progress is recorded in order to complete the synchronization in the fastest way when the incremental data synchronization service is restarted.

[0103] S112: When detecting that the synchronization update exception is repaired, continue to perform the operation of the synchronization update according to the synchronization update progress.

[0104] In Figure 7In the corresponding embodiment, the detection of the synchronization exception is accompanied by the entire data synchronization process, that is, any problem in the entire data synchronization process can be regarded as a synchronization exception. The synchronization progress is recorded to complete the synchronization in the fastest way when the data synchronization service is restarted; after the synchronization exception is repaired, the synchronization operation is continued according to the synchronization progress, so that the synchronization exception or missing data problem in the incremental data process can be quickly solved.

[0105] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the present application.

[0106] In an embodiment, an incremental data synchronization device is provided, which corresponds to the incremental data synchronization method in the above embodiment. As shown in the figure, the incremental data synchronization device includes a recording module 10, a detection module 20, an identification module 30, a first synchronization module 40, a comparison module 50, a setting module 60 and a second synchronization module 70. The functions of each module are described in detail as follows: Figure 8 The recording module 10 records the data of the first database of the upstream business system at the target time, wherein the target time is the end time of the preset synchronization time interval;

[0107] The detection module 20 detects whether there is an incremental mark in the first database, wherein the incremental mark is used to indicate the identification information of the data update;

[0108] The identification module 30 identifies the incremental data according to the incremental mark if it is detected that the first database has the incremental mark;

[0109] The first synchronization module 40 synchronizes the incremental data to the intermediate database;

[0110] The comparison module 50 compares the intermediate database with the second database of the downstream business system to obtain the incremental data that needs to be updated, wherein the second database stores the full data in the first database at the start time of the preset synchronization time interval;

[0111] The setting module 60 sets a corresponding update mark for the incremental data that needs to be updated;

[0112] The second synchronization module 70 synchronously updates the data in the second database according to the update mark.

[0113]

[0114] ​The specific limitation of the device for incremental data synchronization can refer to the limitation of the method for incremental data synchronization in the above, which will not be repeated here. Each module in the device for incremental data synchronization described above can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in the form of software, so as to call and execute the operation corresponding to each module by the processor.

[0115] In one embodiment, as shown in Figure 9 A computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0116] record the data of the first database of the upstream business system at a target time, wherein the target time is the end time of a preset synchronization time interval;

[0117] detect whether there is an incremental mark in the first database, wherein the incremental mark is identification information indicating that there is data update;

[0118] if it is detected that there is the incremental mark in the first database, identify the incremental data according to the incremental mark;

[0119] synchronize the incremental data to an intermediate database;

[0120] compare the intermediate database with a second database of a downstream business system to obtain incremental data that needs to be updated, wherein the second database stores full data in the first database at the start time of the preset synchronization time interval;

[0121] set a corresponding update mark for the incremental data that needs to be updated;

[0122] synchronize and update the data in the second database according to the update mark.

[0123] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0124] record the data of the first database of the upstream business system at a target time, wherein the target time is the end time of a preset synchronization time interval;

[0125] detect whether there is an incremental mark in the first database, wherein the incremental mark is identification information indicating that there is data update;

[0126] If it is detected that the first database has the incremental mark, incremental data is identified according to the incremental mark;

[0127] The incremental data is synchronized into an intermediate database;

[0128] The intermediate database is compared with a second database of a downstream business system to obtain incremental data that needs to be updated, wherein the second database stores full data in the first database at a starting time of a preset synchronization time interval;

[0129] A corresponding update mark is set for the incremental data that needs to be updated;

[0130] The data in the second database is synchronously updated according to the update mark.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments can be included. Any reference to a memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0133] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for incremental data synchronization, applied in a business system, the business system including an upstream business system and a downstream business system, wherein the upstream business system and the downstream business system are systems with related data, characterized in that, The method includes: Record the data of the first database of the upstream business system at the target time, wherein the target time is the cutoff time of the preset synchronization time interval; Detect whether there is an incremental marker in the first database, wherein the incremental marker is used to indicate that there is data update information; If the incremental marker is detected in the first database, the incremental data is identified based on the incremental marker; Synchronize the incremental data to the intermediate database; The intermediate database is compared with the second database of the downstream business system to obtain the incremental data that needs to be updated. The second database stores the full amount of data in the first database at the start time of the preset synchronization time interval. Set corresponding update flags for the incremental data that needs to be updated; The data in the second database is synchronously updated according to the update flag; The step of setting a corresponding update flag for the incremental data that needs to be updated includes: Set a modification update flag for the incremental data that needs to be modified; Set an update flag for the incremental data that needs to be added; Set a deletion update flag for the incremental data that needs to be deleted; Specifically, after setting the corresponding update flag for the incremental data that needs to be updated, and before synchronizing and updating the data in the second database of the downstream business system according to the update flag, the method further includes: According to the update flag, retrieve the original data corresponding to the incremental data that needs to be updated from the second database; The length of the incremental field of the incremental data that needs to be updated is compared with the length of the original field of the original data to obtain the comparison result of the length of the incremental field and the length of the original field. Based on the comparison between the incremental field length and the original field length, the original field length in the second database is updated synchronously.

2. The incremental data synchronization method as described in claim 1, characterized in that, After setting the corresponding update flag for the incremental data that needs to be updated, and before synchronizing and updating the data in the second database of the downstream business system according to the update flag, the method further includes: The target table structure length of the first database is compared with the original table structure length of the second database to obtain the comparison result of the target table structure length and the original table structure length. Based on the comparison between the target table structure length and the original table structure length, the original table structure length in the second database is updated synchronously.

3. The incremental data synchronization method as described in claim 1, characterized in that, The step of comparing the intermediate database with the second database of the downstream business system to obtain the incremental data that needs to be updated includes: The intermediate database is compared with the second database of the downstream business system to obtain the update type of the incremental data that needs to be updated, wherein the update type includes modification, addition or deletion; If the update type is modification, then the incremental data that needs to be modified is obtained; If the update type is "increment", then the incremental data that needs to be added is obtained; If the update type is deletion, then the incremental data that needs to be deleted is obtained.

4. The incremental data synchronization method as described in claim 1, characterized in that, After synchronizing the incremental data to the intermediate database, and before comparing the intermediate database with the second database of the downstream business system to obtain the incremental data that needs to be updated, the method further includes: A binary log is added to the intermediate database, wherein the binary log records the change information of the incremental data in the second database; The incremental data change information is obtained from the binary log, wherein the change information includes the number of modified incremental data, the number of added incremental data, and the number of deleted incremental data; The changes in the incremental data are sent to the client for display.

5. The incremental data synchronization method as described in claim 1, characterized in that, The step of synchronizing the incremental data to the intermediate database includes: The incremental data is synchronized to the intermediate database using Oracle GoldenGate, a comprehensive software that enables real-time data integration and replication between heterogeneous IT environments.

6. The incremental data synchronization method according to any one of claims 1-5, characterized in that, After synchronizing and updating the data in the second database of the downstream business system according to the update flag, the method further includes: When a synchronization update anomaly is detected, the synchronization update operation is stopped, and the synchronization update progress is recorded; Once the synchronization update anomaly is detected and repaired, the synchronization update operation continues according to the synchronization update progress.

7. An apparatus for incremental data synchronization, characterized in that, include: The recording module records the data of the first database of the upstream business system at the target time, wherein the target time is the cutoff time of a preset synchronization time interval; The detection module detects whether there is an incremental marker in the first database, wherein the incremental marker is used to indicate that there is data update information; If the identification module detects that the incremental marker exists in the first database, it identifies the incremental data based on the incremental marker. The first synchronization module synchronizes the incremental data to the intermediate database; The comparison module compares the intermediate database with the second database of the downstream business system to obtain the incremental data that needs to be updated. The second database stores the full amount of data in the first database at the start time of the preset synchronization time interval. The setting module sets corresponding update flags for the incremental data that needs to be updated; The second synchronization module updates the data in the second database synchronously according to the update flag; The step of setting a corresponding update flag for the incremental data that needs to be updated includes: Set a modification update flag for the incremental data that needs to be modified; Set an update flag for the incremental data that needs to be added; Set a deletion update flag for the incremental data that needs to be deleted; The device is also used for: According to the update flag, retrieve the original data corresponding to the incremental data that needs to be updated from the second database; The length of the incremental field of the incremental data that needs to be updated is compared with the length of the original field of the original data to obtain the comparison result of the length of the incremental field and the length of the original field. Based on the comparison between the incremental field length and the original field length, the original field length in the second database is updated synchronously.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the incremental data synchronization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the incremental data synchronization method as described in any one of claims 1 to 6.

Images