A multi-type database data comparison synchronization method, device, system and medium

Abstract

The application provides a multi-type database data comparison and synchronization method, device, system and medium, which can be applied to the field of big data or the field of finance. The data dictionary of each type of database is parsed to obtain structured data; the structured data of one database is divided into multiple parts according to a preset standard to obtain each part of structured data; a database generation rule is generated according to the attributes of each part of structured data; the structured data of different time nodes is compared according to the database generation rule to obtain a comparison result; and the structured data of each type of database is synchronized according to the database generation rule and the comparison result. By establishing a unified database generation rule, the data of different time nodes can be compared according to a unified standard, and different types of database data can be quickly unified to a specific time node according to the comparison result and the database generation rule, without the need for a specific database comparison and synchronization tool, thereby improving the efficiency of multi-type database data comparison and synchronization.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, system and medium for data comparison and synchronization of multiple types of databases. Background Technology

[0002] With the increasing variety of database types, a single enterprise may use multiple types of databases simultaneously. With the adoption of microservice architecture, there is a possibility that multiple types of databases may be used in a single project.

[0003] When it's necessary to uniformly upgrade all types of databases involved in the entire project to a specific point in time, it's essential to compare and synchronize the table structures of these databases to ensure they maintain the same table structure. Currently, the comparison and synchronization of snapshots of the databases at multiple points in time are performed using proprietary tools provided by their respective database service providers. This lack of a unified approach and the need to use multiple comparison and synchronization tools makes the entire process increasingly cumbersome.

[0004] Therefore, how to improve the efficiency of data comparison and synchronization among multiple types of databases is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] In view of this, a summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] The purpose of this application is to provide a method, apparatus, system and medium for comparing and synchronizing data from multiple types of databases, which can improve the efficiency of comparing and synchronizing data from multiple types of databases.

[0007] To achieve the above objectives, this application provides the following technical solution:

[0008] In a first aspect, embodiments of this application provide a method for comparing and synchronizing data from multiple types of databases, including:

[0009] The data dictionary of each type of database is parsed according to preset rules to obtain structured data;

[0010] The structured data in a database is divided into multiple parts according to a preset standard to obtain the structured data of each part;

[0011] Based on the attributes of the structured data in each part, database generation rules are generated;

[0012] The structured data at different time points are compared according to the database generation rules to obtain the comparison results;

[0013] The structured data of each type of database is synchronized according to the database generation rules and the comparison results.

[0014] In one possible implementation, synchronizing the structured data of the various types of databases according to the database generation rules and the comparison results includes:

[0015] The database generation rules and the comparison results are matched to obtain the database synchronization statement;

[0016] The structured data of each type of database is synchronized according to the database synchronization statement.

[0017] In one possible implementation, after obtaining the structured data and before dividing the structured data of the database into the multiple parts according to the preset standard, the method further includes:

[0018] The structured data is stored locally from memory.

[0019] In one possible implementation, the comparison of structured data from different time points according to database generation rules includes:

[0020] The structured data at different time points are segmented according to the multiple parts to obtain the segmented data at different time points;

[0021] The segmented data at different time points are compared separately within the same part.

[0022] Secondly, embodiments of this application provide a multi-type database data comparison and synchronization device, comprising:

[0023] The parsing unit is used to parse the data dictionary of various types of databases according to preset rules to obtain structured data;

[0024] A segmentation unit is used to divide the structured data of a database into multiple parts according to a preset standard to obtain the structured data of each part.

[0025] The generation unit is used to generate database generation rules based on the attributes of the structured data in each part.

[0026] The comparison unit is used to compare the structured data at different time points according to the database generation rules to obtain the comparison results;

[0027] The synchronization unit is used to synchronize the structured data of the various types of databases according to the database generation rules and the comparison results.

[0028] In one possible implementation, the synchronization unit is specifically used for:

[0029] The database generation rules and the comparison results are matched to obtain the database synchronization statement;

[0030] The structured data of each type of database is synchronized according to the database synchronization statement.

[0031] In one possible implementation, the device further includes:

[0032] A storage unit for storing the structured data from memory to local storage.

[0033] In one possible implementation, the comparison unit is specifically used for:

[0034] The structured data at different time points are segmented according to the multiple parts to obtain the segmented data at different time points;

[0035] The segmented data at different time points are compared separately within the same part.

[0036] Thirdly, embodiments of this application provide a multi-type database data comparison and synchronization system, including:

[0037] Memory, used to store computer programs;

[0038] A processor is used to implement the steps of the multi-type database data comparison and synchronization method described above when executing the computer program.

[0039] Fourthly, embodiments of this application provide a computer-readable medium storing a computer program, which, when processed and executed, implements the steps of the multi-type database data comparison and synchronization method described above.

[0040] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0041] This application provides a method, apparatus, system, and medium for comparing and synchronizing data from multiple database types, applicable to the big data or financial fields. The method includes: parsing the data dictionaries of various database types according to preset rules to obtain structured data; dividing the structured data of a database into multiple parts according to preset standards to obtain structured data for each part; generating database generation rules based on the attributes of each part of the structured data; comparing structured data at different time points according to the database generation rules to obtain comparison results; and synchronizing the structured data of various database types based on the database generation rules and the comparison results. Thus, by establishing unified database generation rules, this application can compare data at different time points according to a unified standard. Based on the comparison results and database generation rules, it can quickly unify data from different types of databases to a specific time point, without being dependent on specific database comparison and synchronization tools, thereby improving the efficiency of comparing and synchronizing data from multiple database types. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 The flowchart of a multi-type database data comparison and synchronization method provided in an embodiment of this application is shown;

[0045] Figure 2 A schematic diagram of a multi-type database data comparison and synchronization device provided in an embodiment of this application is shown. Detailed Implementation

[0046] It should be noted that the multi-type database data comparison and synchronization method, apparatus, system, and medium provided by this invention can be used in the fields of big data or finance. The above are merely examples and do not limit the application areas of the multi-type database data comparison and synchronization method, apparatus, system, and medium provided by this invention.

[0047] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0048] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0049] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0050] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0051] As described in the background section, with the increasing variety of database types, a single enterprise may use multiple types of databases simultaneously. With the adoption of microservice architecture, there is a possibility that multiple types of databases may be used in a single project.

[0052] When it's necessary to uniformly upgrade all types of databases involved in the entire project to a specific point in time, it's essential to compare and synchronize the table structures of these databases to ensure they maintain the same table structure. Currently, the comparison and synchronization of snapshots of the databases at multiple points in time are performed using proprietary tools provided by their respective database service providers. This lack of a unified approach and the need to use multiple comparison and synchronization tools makes the entire process increasingly cumbersome.

[0053] Database comparison tools in the industry are usually provided by various database providers / third parties and are only for comparison tools for specific types of databases. They have different functions and cannot well adapt to the comparison and synchronization of specific database snapshot versions when multiple databases are used in a project.

[0054] Specifically, various specialized tools have different functions and are cumbersome to use. When multiple types of databases are used in a project, it is impossible to quickly and uniformly compare and unify them to a specific batch of snapshot versions; it also cannot support access to the continuous integration tools widely used in the industry.

[0055] Therefore, how to improve the efficiency of data comparison and synchronization among multiple types of databases is a technical problem that needs to be solved in this field.

[0056] To address the aforementioned technical issues, this application provides a method, apparatus, system, and medium for comparing and synchronizing data from multiple database types, applicable to the big data or financial fields. The method includes: parsing the data dictionaries of various database types according to preset rules to obtain structured data; dividing the structured data of a database into multiple parts according to preset standards to obtain structured data for each part; generating database generation rules based on the attributes of each part of the structured data; comparing structured data from different time points according to the database generation rules to obtain comparison results; and synchronizing the structured data of various database types based on the database generation rules and the comparison results. Thus, by establishing unified database generation rules, this application can compare data from different time points according to a unified standard. Based on the comparison results and database generation rules, it can quickly unify data from different database types to a specific time point, without being dependent on specific database comparison and synchronization tools, thereby improving the efficiency of comparing and synchronizing data from multiple database types.

[0057] Exemplary methods

[0058] See Figure 1 The flowchart shown is a method for comparing and synchronizing data from multiple database types according to an embodiment of this application, including:

[0059] S101: Parse the data dictionary of each type of database according to preset rules to obtain structured data.

[0060] In this embodiment of the application, to compare and synchronize data from multiple types of databases, it is first necessary to parse the data dictionary of each type of database according to preset rules to obtain structured data.

[0061] Specifically, the data in different types of databases are in different formats, such as HTML (Hyper Text Markup Language) and XML (Extensible Markup Language).

[0062] Therefore, data dictionaries from various database types can be parsed to obtain structured data, facilitating subsequent data comparison and synchronization. Optionally, only data from the time points requiring comparison can be parsed, thereby reducing system resources and energy consumption and saving costs.

[0063] Optionally, in this embodiment, after parsing the data dictionary of each type of database according to preset rules, i.e., after one parsing, unless it is necessary to compare data at a new time point, supplementary parsing is not required. Otherwise, comparisons can be made arbitrarily multiple times between the data snapshots of the imported time points. This allows for flexible comparison while avoiding repeated parsing that wastes time and resources, thus saving costs.

[0064] S102: Divide the structured data of a database into multiple parts according to a preset standard to obtain the structured data of each part.

[0065] In this embodiment of the application, the structured data of a database can be divided into multiple parts according to a preset standard to obtain the structured data of each part.

[0066] Specifically, to facilitate subsequent comparison and synchronization, long data can be divided into shorter data segments to improve the efficiency of subsequent comparison and synchronization.

[0067] For example, the structured data of the Oracle time nodes to be compared can be divided into multiple parts according to a preset standard to obtain the structured data of each part at different time nodes. See Table 1 for the structured data of each part after splitting at time node 1; see Table 2 for the structured data of each part after splitting at time node 2.

[0068] object name Attribute Name Attribute type Attribute length Student Information Name String 10 people

[0069] Table 1

[0070] object name Attribute Name Attribute type Attribute length Student Information Name String 20 people

[0071] Table 2

[0072] Tables 1 and 2 are divided into four parts: object name, attribute name, attribute type, and attribute length. They are similar in that the object name is student information, the attribute name is name, the attribute type is string, and the attribute length is 20 characters. The difference is that Table 2 has modified the attribute length from 10 characters to 20 characters compared to Table 1.

[0073] S103: Generate database generation rules based on the attributes of the structured data in each part.

[0074] In this embodiment of the application, in order to standardize the subsequent comparison and synchronization, it is first necessary to unify the standards so as to compare and synchronize data from different types of databases.

[0075] Specifically, database generation rules can be generated based on the attributes of the structured data in each part.

[0076] For example, taking Table 1 and Table 2 above as examples, the attributes of the four parts of structured data are object name, attribute name, attribute type and attribute length, respectively. Database generation rules are generated based on these four attributes.

[0077] Referring to Table 3, the database generation rules generated in this application embodiment are as follows. The database type can be an Oracle database, the rule type can be a field modification, and the generated database statement style, i.e., the database generation rule, can be Altertable ${object name} modify(${attribute name}${attribute type}${attribute length}).

[0078]

[0079] Table 3

[0080] S104: The structured data at different time points are compared according to the database generation rules to obtain the comparison results.

[0081] S105: Synchronize the structured data of each type of database according to the database generation rules and the comparison results.

[0082] In this embodiment of the application, structured data at different time points can be compared according to the database generation rules to obtain the comparison results; and the structured data of various types of databases can be synchronized according to the database generation rules and the comparison results.

[0083] Specifically, the comparison is performed according to the database generation rules, which can be based on the comparison of the predefined parts, as shown in Table 4.

[0084] object name Attribute Name Attribute type Attribute length Operation type Student Information Name String 20 people Revise

[0085] Table 4

[0086] In this embodiment of the application, after comparing the data at different time points, for example, after comparing Table 1 and Table 2, it was found that the attribute length of time point 2 was modified relative to time point 1, and the comparison result is that the attribute length was modified to 20 digits.

[0087] At this point, based on the comparison results, the attribute lengths in the structured data of other types of databases can be synchronously modified using the database generation rules.

[0088] For example, the database synchronization statement `Alter table student_information_modify(name_string_20)` can be used to synchronize and modify other types of databases.

[0089] Therefore, by establishing unified database generation rules, this application can compare data at different time points according to unified standards. Based on the comparison results and database generation rules, data from different types of databases can be quickly unified to a specific time point without being subject to specific database comparison and synchronization tools, thus improving the efficiency of data comparison and synchronization of multiple types of databases.

[0090] In one possible implementation, the synchronization of structured data from various types of databases based on database generation rules and comparison results, as provided in this embodiment, may specifically include:

[0091] The database generation rules and comparison results are matched to obtain the database synchronization statement; the structured data of various types of databases are synchronized according to the database synchronization statement.

[0092] In this embodiment, database synchronization statements can be obtained by matching database generation rules and comparison results. Therefore, the database synchronization statements are generated according to a unified standard and correspond to the comparison results, improving the targeting of synchronization. Targeted and efficient synchronization can be achieved without being dependent on specific database comparison and synchronization tools. Parsing and generation are decoupled, and subsequent integration with other database types only requires personalized configuration without secondary development or modification.

[0093] In one possible implementation, after obtaining the structured data, and before dividing the structured data of a database into multiple parts according to a preset standard, the embodiments of this application further include:

[0094] Store structured data locally from memory.

[0095] In this embodiment of the application, the structured data that has just been parsed is generally stored in memory, and the data in memory is easily lost when there is a power outage or a fault light.

[0096] Therefore, in order to achieve the persistence of structured data and avoid loss that could cause user losses, the embodiments of this application can store the structured data from memory to local storage.

[0097] In one possible implementation, the comparison of structured data at different time points based on database generation rules provided in this application embodiment may specifically include:

[0098] The structured data at different time points is divided into multiple parts to obtain segmented data at different time points; the segmented data at different time points are compared between the same parts.

[0099] In this embodiment of the application, as shown in Tables 1 and 2, when comparing time node 1 and time node 2, the entire data is not compared.

[0100] Instead, it is divided into four parts and compared one by one: the object names in Table 1 and Table 2 belong to the same part, so they can be compared; the attribute names in Table 1 and Table 2 belong to the same part, so they can be compared; the attribute lengths in Table 1 and Table 2 belong to the same part, so they can be compared; and the attribute types in Table 1 and Table 2 belong to the same part, so they can be compared.

[0101] This allows long, resource-intensive datasets to be divided into shorter datasets for comparison, reducing simultaneous consumption of system resources, avoiding potential lag during the comparison process, and improving comparison efficiency.

[0102] This application provides a method for comparing and synchronizing data from multiple database types, applicable to big data or financial fields. The method includes: parsing the data dictionaries of various database types according to preset rules to obtain structured data; dividing the structured data of a database into multiple parts according to preset standards to obtain structured data for each part; generating database generation rules based on the attributes of each part of the structured data; comparing structured data at different time points according to the database generation rules to obtain comparison results; and synchronizing the structured data of various database types based on the database generation rules and the comparison results. Thus, by establishing unified database generation rules, this application can compare data at different time points according to a unified standard. Based on the comparison results and database generation rules, it can quickly unify data from different types of databases to a specific time point, without being dependent on specific database comparison and synchronization tools, thereby improving the efficiency of comparing and synchronizing data from multiple database types.

[0103] Exemplary device

[0104] See Figure 2 The diagram shown is a schematic of a multi-type database data comparison and synchronization device provided in an embodiment of this application, including:

[0105] Parsing unit 201 is used to parse the data dictionary of various types of databases according to preset rules to obtain structured data;

[0106] The segmentation unit 202 is used to segment the structured data of a database into multiple parts according to a preset standard to obtain the structured data of each part.

[0107] The generation unit 203 is used to generate database generation rules based on the attributes of the structured data in each part;

[0108] The comparison unit 204 is used to compare the structured data at different time points according to the database generation rules to obtain the comparison results;

[0109] Synchronization unit 205 is used to synchronize the structured data of each type of database according to the database generation rules and the comparison results.

[0110] In one possible implementation, the synchronization unit is specifically used for:

[0111] The database generation rules and the comparison results are matched to obtain the database synchronization statement;

[0112] The structured data of each type of database is synchronized according to the database synchronization statement.

[0113] In one possible implementation, the device further includes:

[0114] A storage unit for storing the structured data from memory to local storage.

[0115] In one possible implementation, the comparison unit is specifically used for:

[0116] The structured data at different time points are segmented according to the multiple parts to obtain the segmented data at different time points;

[0117] The segmented data at different time points are compared separately within the same part.

[0118] This application provides a multi-type database data comparison and synchronization device, applicable to the big data or financial fields. The method applied to this device includes: parsing the data dictionaries of various types of databases according to preset rules to obtain structured data; dividing the structured data of a database into multiple parts according to preset standards to obtain structured data for each part; generating database generation rules based on the attributes of each part of the structured data; comparing structured data at different time points according to the database generation rules to obtain comparison results; and synchronizing the structured data of various types of databases according to the database generation rules and the comparison results. Thus, by establishing unified database generation rules, this application can compare data at different time points according to a unified standard. Based on the comparison results and database generation rules, it can quickly unify data from different types of databases to a specific time point, without being restricted by specific database comparison and synchronization tools, thereby improving the efficiency of multi-type database data comparison and synchronization.

[0119] Based on the above embodiments, this application provides a multi-type database data comparison and synchronization system, including:

[0120] Memory, used to store computer programs;

[0121] A processor is used to implement the steps of the multi-type database data comparison and synchronization method described above when executing the computer program.

[0122] Based on the above embodiments, this application also provides a computer-readable medium storing a computer program, which, when processed and executed, implements the steps of the multi-type database data comparison and synchronization method described above.

[0123] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0124] The aforementioned computer-readable medium may be included in the aforementioned system, or it may exist independently and not assembled into the system.

[0125] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] The above description is merely a preferred embodiment of this application. Although this application has disclosed preferred embodiments above, it is not intended to limit this application. Any person skilled in the art can make many possible variations and modifications to the technical solutions of this application using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of this application. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solutions of this application shall still fall within the protection scope of the technical solutions of this application.

Claims

1. A method for comparing and synchronizing data from multiple database types, characterized in that, include: The data dictionary of each type of database is parsed according to preset rules to obtain structured data; The structured data in a database is divided into multiple parts according to a preset standard to obtain the structured data of each part; Based on the attributes of the structured data in each part, database generation rules are generated; The structured data at different time points are compared according to the database generation rules to obtain the comparison results; Based on the database generation rules and the comparison results, the structured data of each type of database is synchronized. The step of synchronizing the structured data of each type of database according to the database generation rules and the comparison results includes: The database generation rules and the comparison results are matched to obtain the database synchronization statement; According to the database synchronization statement, the structured data of each type of database is synchronized; The comparison of structured data at different time points based on database generation rules includes: The structured data at different time points are segmented according to the multiple parts to obtain the segmented data at different time points; The segmented data at different time points are compared separately within the same part.

2. The method according to claim 1, characterized in that, After obtaining the structured data, but before dividing the structured data of the database into the multiple parts according to the preset standard, the method further includes: The structured data is stored locally from memory.

3. A multi-type database data comparison and synchronization device, characterized in that, include: The parsing unit is used to parse the data dictionary of various types of databases according to preset rules to obtain structured data; A segmentation unit is used to divide the structured data of a database into multiple parts according to a preset standard to obtain the structured data of each part. The generation unit is used to generate database generation rules based on the attributes of the structured data in each part. The comparison unit is used to compare the structured data at different time points according to the database generation rules to obtain the comparison results; The synchronization unit is used to synchronize the structured data of the various types of databases according to the database generation rules and the comparison results; The synchronization unit is specifically used for: The database generation rules and the comparison results are matched to obtain the database synchronization statement; According to the database synchronization statement, the structured data of each type of database is synchronized; The comparison unit is specifically used for: The structured data at different time points are segmented according to the multiple parts to obtain the segmented data at different time points; The segmented data at different time points are compared separately within the same part.

4. The apparatus according to claim 3, characterized in that, The device further includes: A storage unit for storing the structured data from memory to local storage.

5. A multi-type database data comparison and synchronization system, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multi-type database data comparison and synchronization method as described in any one of claims 1-2 when executing the computer program.

6. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program that, when processed and executed, implements the steps of the multi-type database data comparison and synchronization method as described in any one of claims 1-2.

Images