A method, device and medium for data synchronization

Abstract

The present application discloses a method, device and medium for data synchronization, the method comprising: determining a source data table of a source database, and determining a target data table and a temporary storage data table of a target database; determining the existence of the latest data according to the screening fields in the source data table; synchronizing the latest data to the target data table and the temporary storage data table according to the pre-established data mapping between the source database and the target database; determining the existence of difference data according to the target data table and the temporary storage data table, and marking the difference type of the difference data to the target data table, the difference type at least including: deletion type and modification type. After data synchronization, the difference type of the difference data is added to the target database to facilitate data retrieval or data recovery. At the same time, all comparison and calculation processes are performed in the target database without affecting the performance of the source database.

Description

Technical Field

[0001] The present application relates to the field of computer data science and technology, and in particular to a method, device and medium for data synchronization. Background Art

[0002] Synchronization is a very important concept in computer science. Synchronization generally refers to the process of achieving consistency between different data. Generally speaking, data synchronization is the process of achieving consistency between several data entities. The main information carrier in computer science and engineering is the database table. Synchronization is generally completed between different databases or different tables in the same database. Synchronization can keep data consistent and obtain the latest data.

[0003] Due to the particularity of certain businesses and data security factors, deleted data cannot be saved or restored in the database. When data is deleted by mistake, it is impossible to recover the data through corresponding technical means without destroying the stability of other data in the database. Summary of the invention

[0004] In order to solve the above problem, that is, to solve the problem that the deleted data cannot be saved and restored in the source database, the present application proposes a data synchronization method, device and medium, including:

[0005] In a first aspect, the present application provides a method for data synchronization, comprising: determining a source data table of a source database, and determining a target data table and a temporary storage data table of a target database; determining the existence of the latest data based on a filter field in the source data table; synchronizing the latest data to the target data table and the temporary storage data table based on a pre-established data mapping between the source database and the target database; determining the existence of difference data based on the target data table and the temporary storage data table, and marking the difference type of the difference data to the target data table, wherein the difference type includes at least: deletion type and modification type.

[0006] In one example, determining the existence of the latest data based on the filter field in the source data table specifically includes: determining the filter field corresponding to the source data in the source data table, and determining the data creation time of the source data based on the filter field; determining the existence of the latest data based on the data creation time.

[0007] In one example, according to the pre-established data mapping between the source database and the target database, the latest data is synchronized to the target data table and the temporary storage data table, specifically including: determining the pre-established data mapping between the source database and the target database; synchronizing the latest data increment to the target data table, and synchronizing the full amount of the source data to the temporary storage data table.

[0008] In one example, based on the target data table and the temporary storage data table, determining the existence of difference data, and marking the difference type of the difference data to the target data table, specifically includes: determining the filter fields of all target data in the target data table, and determining the filter fields of all temporary data in the temporary storage data table; determining the existence of difference data by comparing the filter fields of all target data with the filter fields of all temporary data; determining a data difference retrieval table pre-set in the target database, and storing the data number of the difference data in the data difference retrieval table; determining the difference type of the difference data through the filter fields of the difference data, and marking the difference type to the target data table, the difference type including at least: deletion type and modification type.

[0009] In one example, the difference type of the difference data is determined through the filtering field of the difference data, and the difference type is marked to the target data table, specifically including: determining the data modification time and data number of the difference data through the filtering field of the difference data; for any one of the multiple difference data, determining whether the data number corresponding to the difference data exists in the temporary storage data table; if so, determining whether the data modification time of the difference data is consistent with the corresponding data in the target data table; if so, deleting the data number of the difference data in the data difference retrieval table; if not, determining that the difference type of the difference data is a modification type, and marking the difference type of the difference data to the data corresponding to the target data table; if not, determining that the difference type of the difference data is a deletion type, and marking the difference type of the difference data to the data corresponding to the target data table.

[0010] In one example, after determining the difference type of the difference data through the screening field of the difference data and marking the difference type to the target data table, the method further includes: formatting the data in the temporary storage data table.

[0011] In one example, after determining the difference type of the difference data through the screening field of the difference data and marking the difference type to the target data table, the method also includes: determining the data number of the data to be recovered through the data difference retrieval table; according to the data number of the data to be recovered, querying and determining the target data with the corresponding data number from the target data table as the data to be recovered; and synchronizing the incremental data to be recovered to the source database through the data mapping.

[0012] In one example, the method further includes: storing the target data table and the temporary storage data table in a memory of the target database, and processing the target data table and the temporary storage data table by using a directed acyclic graph algorithm.

[0013] On the other hand, the present application provides a data synchronization device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions: determine a source data table of a source database, and determine a target data table and a temporary storage data table of a target database; determine the existence of the latest data based on a filter field in the source data table; synchronize the latest data to the target data table and the temporary storage data table based on a pre-established data mapping between the source database and the target database; determine the existence of difference data based on the target data table and the temporary storage data table, and mark the difference type of the difference data to the target data table, wherein the difference type includes at least: deletion type and modification type.

[0014] On the other hand, the present application provides a non-volatile computer storage medium for data synchronization, storing computer executable instructions, wherein the computer executable instructions are configured to: determine a source data table of a source database, and determine a target data table and a temporary storage data table of a target database; determine the existence of the latest data based on a screening field in the source data table; synchronize the latest data to the target data table and the temporary storage data table based on a pre-established data mapping between the source database and the target database; determine the existence of difference data based on the target data table and the temporary storage data table, and mark the difference type of the difference data to the target data table, wherein the difference type includes at least: deletion type and modification type.

[0015] The data synchronization method, device and medium proposed in this application can bring the following beneficial effects: after data synchronization, the difference type of difference data is added to the target database to facilitate data retrieval or data recovery. At the same time, all comparison and calculation processes are performed in the target database without affecting the performance of the source database. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a method for data synchronization in an embodiment of the present application;

[0018] Figure 2 A flowchart of another method for data synchronization in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a data synchronization device in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] First of all, it should be noted that a data synchronization method described in the embodiments of the present application is stored in a corresponding system or server, and a user can log in to the system or server through a corresponding terminal, which can be a mobile phone, tablet computer, personal computer or other hardware device with corresponding functions. The system can be pre-installed in the terminal, and the user can also log in to the system or server through APP, WEB page, etc. to achieve data synchronization.

[0022] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0023] like Figure 1 As shown, a data synchronization method provided in an embodiment of the present application includes:

[0024] S101: Determine a source data table of a source database, and determine a target data table and a temporary storage data table of a target database.

[0025] Specifically, a source database and a source data table in the source database for storing source data are determined. The source database is a database that is directly used by users or management parties. In the embodiment of the present application, based on its particularity or security factors, the data in the source database cannot be backed up or restored. Although security is greatly guaranteed, it causes data irreversibility. If a user makes a mistake, key data is difficult to retrieve.

[0026] Based on this, the present application sets up a target database corresponding to the source database, which can record and back up the source database without affecting it, and mark the modified data, which greatly facilitates data retrieval. The target database is pre-set with a target data table and a temporary storage data table.

[0027] S102: Determine whether the latest data exists according to the filter fields in the source data table.

[0028] Specifically, determine the filter fields corresponding to the source data in the source data table. The filter fields here are determined in advance and may include the primary key and secondary key of the data, and may specifically include the creation time of the data, the modification time of the data, the data number, etc.

[0029] Furthermore, the data creation time of the source data is determined according to the filter field, and according to the data creation time, it is determined that the latest data exists.

[0030] The latest data here refers to the data that has not existed in the source data table before and is the latest data added.

[0031] S103: Synchronize the latest data to the target data table and the temporary storage data table according to the pre-established data mapping between the source database and the target database.

[0032] Specifically, a data mapping between the active database and the target database is pre-established, and data transmission between the source database and the target database can be realized through data mapping. It should be noted that the data mapping here is determined and executed by capturing the changed data, and the capture trigger method includes but is not limited to: timestamp type, trigger type, snapshot type, and log type. The specific capture trigger method can achieve the corresponding function and cannot affect the protection scope of this application.

[0033] Furthermore, through data mapping, the latest data increments are synchronized to the target data table, and the full amount of source data is synchronized to the temporary storage data table. The target data table is the backup table of the source data table, which stores all the data of the source data table. When the latest data appears in the source data table, the target data table will also synchronize and back up the latest data. At the same time, the target data table is only used to record all the data in the source data table. If there is changed data in the source data table, such as deleted or modified data, the target database will not synchronize and update it.

[0034] At the same time, the present application sets up a temporary storage data table, which is set in the target database to avoid occupying the computing resources of the source database. At the same time, it can fully synchronize the source data in the source data table. That is to say, no matter whether the latest data or changed data appears in the source data table, the temporary storage data table will be fully synchronized.

[0035] S104: Determine, based on the target data table and the temporary storage data table, whether there is difference data, and mark the difference type of the difference data to the target data table, where the difference type includes at least: deletion type and modification type.

[0036] Specifically, the filter fields of all target data in the target data table and the filter fields of all temporary data in the temporary storage data table can be determined, and by comparing the filter fields of all target data and the filter fields of all temporary data, it is determined that there are difference data. Since the target data table reflects the historical status of the source data table, the temporary storage data table reflects the current status of the source data table, and both data tables synchronize the latest data, if there is change data belonging to historical data in the source data table, the data will be synchronized and recorded in the temporary storage data table. Therefore, it is possible to determine whether there are difference data by comparing the filter fields of all target data and all temporary data, such as data modification time and data number.

[0037] The difference data reflects the changed historical data in the source database.

[0038] Furthermore, a data difference retrieval table is pre-set in the target database for storing the data numbers of the difference data. After the difference data is determined, the data numbers of the difference data can be stored in the data difference retrieval table.

[0039] Further, the difference type of the difference data is determined through the screening field of the difference data, and the difference type is marked in the target data table, and the difference type at least includes: deletion type and modification type. Specifically, the data modification time and data number of the difference data are determined through the screening field of the difference data.

[0040] Taking any one of the multiple difference data as an example for explanation, first, it is necessary to determine whether the data number corresponding to the difference data exists in the temporary difference data table. Since the temporary difference data table can reflect the current status of the source data table, if the data number of the difference data does not exist in the temporary difference data table, it means that the difference data existed in the source data table before, but does not exist now. It can be inferred that the difference data has been deleted in the source data table. At this time, it can be determined that the difference type of the difference data is modified, and the difference type of the difference data is marked in the data corresponding to the target data table. The marking method can be marked in the remarks of the numbered data.

[0041] If the data number of the difference data exists in the temporary difference data table, the data number of the difference data must also exist in the target data table. Further, the corresponding data of the difference data in the target data table can be determined according to the data number.

[0042] Furthermore, by determining whether the data modification time of the difference data is consistent with the data modification time of the corresponding data in the target data table, it is possible to determine whether the difference data is accurate. If they are consistent, it means that the data number and data modification time of the difference data are completely consistent with the corresponding data, and there is no difference. At this time, it can be determined that the difference data is a misjudgment of the system, so it is necessary to delete the data number of the difference data in the data difference retrieval table.

[0043] If there is inconsistency, it means that although the difference data currently exists in both the source data table and the target data table, the data has been modified in the source data table. At this time, the difference type of the difference data can be determined as a modified type, and the difference type of the difference data can be marked in the corresponding data of the target data table. The marking method can be marked in the remarks of the numbered data.

[0044] Through the above method, the data changes in the source database can be recorded without affecting the backup function of the target database.

[0045] Furthermore, after the difference type is marked on the target data table, the data in the temporary storage data table has completed all its tasks. To ensure that it can continue to perform its tasks at the next time point, the data in the temporary storage data table needs to be formatted.

[0046] Furthermore, if data in the source database is deleted or misoperated by mistake, since the original state of the data is recorded in the target database, the data in the source database can be restored through reverse synchronization. Specifically, the data number of the data to be restored is determined through the data difference search table, and according to the data number of the data to be restored, the target data with the corresponding data number is queried from the target data table as the data to be restored. Through the above data mapping, the incremental synchronization of the data to be restored is carried out to the source database.

[0047] At this point, through all the technical solutions of this application, it is possible to back up and restore the source data without affecting the source database.

[0048] In one embodiment, the target data table and the temporary storage data table described in the present application are stored in the memory of the target database, and the target data table and the temporary storage data table are processed by a directed acyclic graph algorithm. The difference data compared in this way has a calculation efficiency hundreds of times higher than that of MySQL data.

[0049] In one embodiment, Figure 2As shown, the present application also provides a data synchronization device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions:

[0050] Determine the source data table of the source database, and determine the target data table and temporary storage data table of the target database;

[0051] Determining whether the latest data exists according to the screening fields in the source data table;

[0052] According to the pre-established data mapping between the source database and the target database, the latest data is synchronized to the target data table and the temporary storage data table;

[0053] According to the target data table and the temporary storage data table, it is determined that there is difference data, and the difference type of the difference data is marked in the target data table, and the difference type includes at least: deletion type and modification type.

[0054] In one embodiment, the present application further provides a data synchronization non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0055] Determine the source data table of the source database, and determine the target data table and temporary storage data table of the target database;

[0056] Determining whether the latest data exists according to the screening fields in the source data table;

[0057] According to the pre-established data mapping between the source database and the target database, the latest data is synchronized to the target data table and the temporary storage data table;

[0058] According to the target data table and the temporary storage data table, it is determined that there is difference data, and the difference type of the difference data is marked in the target data table, and the difference type includes at least: deletion type and modification type.

[0059] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0060] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0061] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0062] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0067] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0069] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for data synchronization, characterized in that: include: Determine the source data table of the source database, and determine the target data table and temporary storage data table of the target database; Determining whether the latest data exists according to the screening fields in the source data table; Synchronizing the latest data to the target data table and the temporary storage data table according to the pre-established data mapping between the source database and the target database, specifically including: determining the pre-established data mapping between the source database and the target database; synchronizing the latest data increment to the target data table, and synchronizing the full amount of the source data to the temporary storage data table; According to the target data table and the temporary storage data table, it is determined that there is difference data, and the difference type of the difference data is marked in the target data table, and the difference type at least includes: deletion type and modification type, specifically including: determining the screening fields of all target data in the target data table, and determining the screening fields of all temporary data in the temporary storage data table; determining the existence of difference data by comparing the screening fields of all target data with the screening fields of all temporary data; determining a data difference retrieval table pre-set in the target database, and storing the data number of the difference data in the data difference retrieval table; , determine the data modification time and data number of the difference data; for any one of the multiple difference data, determine whether the data number corresponding to the difference data exists in the temporary storage data table; if so, determine whether the data modification time of the difference data is consistent with the corresponding data in the target data table, if so, delete the data number of the difference data in the data difference retrieval table, if not, determine that the difference type of the difference data is a modification type, and mark the difference type of the difference data in the data corresponding to the target data table; if not, determine that the difference type of the difference data is a deletion type, and mark the difference type of the difference data in the data corresponding to the target data table.

2. A data synchronization method according to claim 1, characterized in that: According to the filter fields in the source data table, it is determined whether the latest data exists, including: Determine a filter field corresponding to the source data in the source data table, and determine a data creation time of the source data according to the filter field; According to the data creation time, it is determined that the latest data exists.

3. A data synchronization method according to claim 1, characterized in that: After determining the difference type of the difference data through the screening field of the difference data and marking the difference type to the target data table, the method further includes: The data in the temporary storage data table is formatted.

4. A data synchronization method according to claim 1, characterized in that: After determining the difference type of the difference data through the screening field of the difference data and marking the difference type to the target data table, the method further includes: Determine the data number of the data to be restored through the data difference retrieval table; According to the data number of the data to be restored, query and determine the target data with the corresponding data number from the target data table as the data to be restored; The data to be recovered is incrementally synchronized to the source database through the data mapping.

5. A data synchronization method according to claim 1, characterized in that: The method further comprises: The target data table and the temporary storage data table are stored in the memory of the target database, and the target data table and the temporary storage data table are processed by a directed acyclic graph algorithm.

6. A data synchronization device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions: a data synchronization method as described in claim 1.

7. A data synchronization non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: a data synchronization method as described in claim 1.

Images