Data synchronization system and storage medium
By using the acquisition and conversion module in the data synchronization system, the data synchronization problem caused by the syntax differences between distributed databases and commercial databases is solved, and efficient heterogeneous database data synchronization is achieved.
Patent Information
- Application Number
- CN202411204307.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
The syntax of distributed databases and commercial databases is different, making it difficult to synchronize data entered into distributed databases to commercial databases.
A data synchronization system is designed, including a collection module and a conversion module. The collection module obtains the data to be synchronized. The conversion module converts the data by obtaining syntax features and syntax correspondence to ensure that the target data is suitable for commercial databases.
Data synchronization between distributed databases and commercial databases is realized, data synchronization efficiency is improved, and deployment difficulty is reduced.
Smart Images

Figure CN119066127A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of big data technology, and in particular, relates to a data synchronization system and a storage medium. Background Art
[0002] Currently, enterprises and institutions that have data processing businesses or generate a large amount of data in the process of processing businesses usually choose to import data into databases for unified storage and management. Due to the difficulty of database development and insufficient technical accumulation, in the past, when using databases, they had to purchase commercial databases (Commercial Database, such as Oracle developed by Oracle) developed by foreign companies at high prices. With the increasing investment and attention paid to database research and development, the functions of self-developed databases developed by domestic companies are also gradually improving, especially the development of distributed databases (Distributed Database, such as TiDB), which can store data in a distributed manner on different computers compared to traditional commercial databases, and has the advantages of low data storage cost, high reliability, and strong scalability.
[0003] As more and more users abandon traditional commercial databases and start using distributed databases, in the process of switching between the two different types of databases, the data stored in the commercial database needs to be migrated to the distributed database, and the data entered into the distributed database needs to be synchronized to the commercial database, thereby achieving two-way synchronization between the two different types of databases.
[0004] However, the syntax of the two different types of databases is different, which makes it difficult to synchronize the data entered into the distributed database to the commercial database. Therefore, how to synchronize the data entered into the distributed database to the commercial database has become an urgent problem that needs to be solved. Summary of the invention
[0005] In view of this, an embodiment of the present application provides a data synchronization system and a storage medium to solve the problem that the syntax of the distributed database and the commercial database is different, resulting in the data entered into the distributed database being difficult to synchronize to the commercial database.
[0006] A first aspect of an embodiment of the present application provides a data synchronization system, including a collection module and a conversion module, wherein the collection module is connected to a first database, and the conversion module is connected to a second database;
[0007] The acquisition module is used to obtain the data to be synchronized from the first database;
[0008] The conversion module is used for:
[0009] Acquire grammatical features of the data to be synchronized, where the grammatical features include grammatical attributes of each minimum unit data, and the grammatical features are used to indicate a logical relationship between the minimum unit data;
[0010] According to the grammatical features of the data to be synchronized and the grammatical correspondence between the first database and the second database, the data to be synchronized is converted to obtain target data;
[0011] sending the target data to the second database;
[0012] The code type of the data to be synchronized is applicable to the first database, and the code type of the target data is applicable to the second database.
[0013] In one embodiment, the collection module includes a log collection unit and a log conversion unit;
[0014] The log collection unit is used to obtain log data of the first database;
[0015] The log conversion unit is used to convert the log data to obtain the data to be synchronized.
[0016] In one embodiment, the acquisition module is further used to acquire the transaction sequence of the first database, and the transaction sequence is used to reflect the transaction number, transaction information, logical relationship and transaction status of each data in the log data.
[0017] In one embodiment, the conversion module is used to convert the data to be synchronized according to the transaction timing, the grammatical features of the data to be synchronized, and the grammatical correspondence between the first database and the second database to obtain target data.
[0018] In one embodiment, the conversion module further includes an identification unit;
[0019] The identification unit is used to read the identity information of the data to be synchronized, where the identity information includes the type of the first database and the table name of the data to be synchronized;
[0020] The identification unit is further used to read rule information of the data to be synchronized, where the rule information includes index configuration, constraint configuration and directory configuration.
[0021] In one embodiment, the conversion module is further used to: convert the identity information of the data to be synchronized to obtain the identity information of the target data, wherein the code type of the identity information of the target data is applicable to the second database;
[0022] Sending the identity information of the target data to the second database;
[0023] Converting the rule information of the data to be synchronized to obtain the rule information of the target data, wherein the code type of the rule information of the target data is applicable to the second database;
[0024] The rule information of the target data is sent to the second database.
[0025] In one embodiment, it further comprises an accuracy analysis module, wherein the accuracy analysis module is connected to the first database, the acquisition module and the conversion module respectively;
[0026] The accuracy analysis module is used to read the source data, log data and the data to be synchronized of the first database, and analyze the data acquisition accuracy of the acquisition module;
[0027] The accuracy analysis module is further used to read the source data of the first database, the data to be synchronized and the target data, and analyze the data conversion accuracy of the conversion module.
[0028] In one embodiment, an optimization module is further included, and the optimization module is connected to the accuracy analysis module, the acquisition module and the conversion module respectively;
[0029] The optimization module is used to optimize the acquisition module according to the data acquisition accuracy;
[0030] The optimization module is also used to optimize the conversion module according to the data conversion accuracy.
[0031] A first aspect of an embodiment of the present application provides a data synchronization system, which collects data to be synchronized in a first database through an acquisition module, and obtains grammatical features of the data to be synchronized through a conversion module. According to the grammatical features and the grammatical correspondence between the first database and the second database, the data to be synchronized is converted to obtain target data. After sending the target data to the second database, data synchronization with the first database is achieved. Based on the lightweight acquisition module and the conversion module, the data synchronization system can be quickly deployed and data synchronization of heterogeneous databases can be achieved, thereby improving the data synchronization efficiency between distributed databases and commercial databases.
[0032] A second aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the various steps of the synchronization program in the data synchronization system provided by the first aspect of the embodiment of the present application.
[0033] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 This is a first structural diagram of a data synchronization system provided in an embodiment of the present application;
[0036] Figure 2 This is a first interaction diagram of the data synchronization system provided in an embodiment of the present application;
[0037] Figure 3 This is a second structural diagram of the data synchronization system provided in an embodiment of the present application;
[0038] Figure 4 This is a second interaction diagram of the data synchronization system provided in an embodiment of the present application;
[0039] Figure 5 This is a third structural diagram of the data synchronization system provided in an embodiment of the present application;
[0040] Figure 6 This is a fourth structural diagram of the data synchronization system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0042] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0043] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0044] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0045] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0046] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0047] In applications, more and more users are abandoning traditional commercial databases and starting to use distributed databases. In the process of switching between two different types of databases, the data stored in the commercial database needs to be migrated to the distributed database, and the data entered into the distributed database needs to be synchronized to the commercial database, so as to achieve two-way synchronization between the two different types of databases. However, the syntax of the distributed database and the commercial database is different, which makes it difficult to synchronize the data entered into the distributed database to the commercial database.
[0048] In response to the above technical problems, an embodiment of the present application provides a data synchronization system, which collects the data to be synchronized in the first database through an acquisition module, and obtains the grammatical features of the data to be synchronized through a conversion module. According to the grammatical features and the grammatical correspondence between the first database and the second database, the data to be synchronized is converted to obtain target data. After sending the target data to the second database, data synchronization with the first database is achieved. Based on the lightweight acquisition module and conversion module, the data synchronization system can be quickly deployed and data synchronization of heterogeneous databases can be achieved, thereby improving the data synchronization efficiency between distributed databases and commercial databases.
[0049] like Figure 1As shown, the data synchronization system 100 provided in the embodiment of the present application includes a collection module 110 and a conversion module 120, wherein the collection module 110 is connected to a first database 200, and the conversion module 120 is connected to a second database 300;
[0050] The acquisition module 110 is used to obtain the data to be synchronized from the first database 200;
[0051] The conversion module 120 is used to:
[0052] Acquire grammatical features of the data to be synchronized, where the grammatical features include grammatical attributes of each minimum unit of data, and the grammatical features are used to indicate the logical relationship between the minimum unit of data;
[0053] According to the grammatical features of the data to be synchronized and the grammatical correspondence between the first database 200 and the second database 300, the data to be synchronized is converted to obtain target data;
[0054] Sending target data to the second database 300;
[0055] The code type of the data to be synchronized is applicable to the first database 200 , and the code type of the target data is applicable to the second database 300 .
[0056] Figure 2 The interactive schematic diagram of the data synchronization system 100 is shown exemplarily.
[0057] The working principle of the acquisition module 110 is described below:
[0058] In the application, the first database 200 can be a distributed database, and the second database 300 can be a commercial database (non-distributed database). The first database 200 can specifically be a TiDB database, and the second database 300 can specifically be an Oracle database. The embodiment of the present application does not impose any restrictions on the specific types of the first database 200 and the second database 300.
[0059] In the application, the data to be synchronized of the first database 200 can be obtained through the acquisition module 110. Specifically, the data acquisition instruction is sent to the first database 200 through the acquisition module 110, and the first database 200 responds to the data acquisition instruction and feeds back the data to be synchronized to the acquisition module 110; the acquisition module 110 can obtain the data to be synchronized of the first database 200 according to a preset period. At this time, the data to be synchronized is the data updated by the first database 200 within the preset period, thereby realizing incremental acquisition of the data to be synchronized; the acquisition module 110 can also detect whether the first database 200 has updated the data. When the first database 200 has updated the data, it traverses and obtains each updated data to obtain the data to be synchronized, thereby realizing real-time acquisition of the data to be synchronized.
[0060] The working principle of the conversion module 120 is described below:
[0061] In the application, the conversion module 120 can be a conversion component inside the first database 200, specifically a Drainer (filter) component, or an external interface plugged into the first database 200. The embodiment of the present application does not limit the specific type of the conversion module 120.
[0062] In the application, the conversion module 120 may include a grammar analysis component (Lexer) built based on the compilation principle. Before the data to be synchronized is segmented, it is a code string composed of characters. After the grammar analysis component of the conversion module 120 performs word segmentation on the data to be synchronized, multiple minimum unit data can be obtained, and each minimum unit data has corresponding grammar attributes. Among them, the grammar attributes may include encapsulation (Encapsulation), object (Class, also known as class), function (Function), variable (Variable), value (Value) and operator (Operator), etc.
[0063] For example, if one of the code character strings of the data to be synchronized is "int x=10", then after word segmentation by the conversion module 120, four minimum unit data can be obtained. Specifically, the first minimum unit data is "int", and the grammatical attribute is the integer type definition in the function; the second minimum unit data is "x", and the grammatical attribute is a variable; the third minimum unit data is "=", and the grammatical attribute is an operator; the fourth minimum unit data is "10", and the grammatical attribute is a numerical value.
[0064] In the application, the conversion module 120 can perform syntax analysis on each minimum unit data obtained by the syntax analysis. Specifically, each minimum unit data can be parsed based on the computer language syntax specified by Backus Naur Form (BNF), and an abstract syntax tree (Abstract Syntax Tree, AST) composed of each minimum unit data can be formed. The abstract syntax tree includes multiple layers of minimum unit data. The syntax attributes of the minimum unit data in the same layer are the same, and the syntax attributes of the minimum unit data in adjacent layers are different. According to the abstract syntax tree, the logical relationship between multiple minimum unit data can be determined, thereby obtaining the syntax features of the data to be synchronized.
[0065] In the application, the conversion module 120 can further parse the data to be synchronized according to the grammatical features of the data to be synchronized. Specifically, a preset grammatical attribute can be set, and the core data in the data to be synchronized can be extracted from the multiple minimum unit data according to the preset grammatical attribute, and the logical relationship between the core data can be obtained according to the grammatical features; or the core data can be not extracted and all the data to be synchronized can be converted.
[0066] In the application, the data synchronization system 100 can store a grammatical correspondence table of the second database 300 and the first database 200, and can convert the core data or all the data to be synchronized based on the grammatical correspondence table to obtain target data, and the code type of the converted target data is suitable for the second database 300.
[0067] In one embodiment, the collection module 110 is further used to collect the transaction sequence of the first database 200, and the transaction sequence is used to reflect the transaction number, transaction information, logical relationship and transaction status of each data in the log data;
[0068] The conversion module 120 is used to convert the data to be synchronized according to the transaction sequence, the grammatical features of the data to be synchronized, and the grammatical correspondence between the first database and the second database to obtain target data.
[0069] In the application, when the conversion module 120 converts the data to be synchronized, the transaction sequence can also be substituted into the conversion process as a parameter. Since the data to be synchronized is generated according to the data changes of the first database 200, each data change can generate a piece of data, and the transaction sequence can reflect the transaction number, transaction information, transaction logic and transaction status of each data change. Therefore, the transaction number can reflect the order of data changes, that is, the generation order of each piece of data, the transaction information can reflect the reason for the data change, that is, the reason for the generation of each piece of data, the transaction logic can reflect the causal relationship between different data changes, that is, the causal relationship generated between multiple pieces of data, and the transaction status can reflect whether the data has changed, that is, whether each piece of data has been modified. By substituting the transaction sequence as a parameter into the conversion process, the accuracy of converting the data to be synchronized can be improved.
[0070] In the application, the conversion module 120 can establish a connection with the second database 300 and send the target data to the second database 300. The code type of the target data is the code type of the second database 300. Therefore, the second database 300 can obtain the same data as the source data of the first database 200 by executing the code included in the target data, thereby realizing data synchronization between the distributed database and the commercial database.
[0071] The following is an example of the specific application of this solution: In the financial field such as banking, securities, funds, asset management and trust, commercial databases are usually purchased in the past to store important data such as customer information and product information. Since commercial databases do not have distributed storage capabilities, data cannot be shared, migrated and stored together within a safe range, resulting in high storage and management costs and low sharing efficiency. By synchronizing the data of distributed databases to commercial databases through this solution, not only can the data storage cost, high reliability and strong scalability be achieved, but also the data of all distributed computers can be synchronized to the commercial database, ensuring that all data can be summarized, improving the security and integrity of data storage. Figure 3 As shown, in one embodiment, based on Figure 1 In the corresponding embodiment, the collection module 110 includes a log collection unit 111 and a log conversion unit 112:
[0072] The log collection unit 111 is used to obtain log data of the first database 200;
[0073] The log conversion unit 112 is used to convert the log data to obtain the data to be synchronized.
[0074] In the application, the log collection unit 111 can be a log recording component inside the first database 200, specifically a Pump component of the TiDB database; it can also be an external interface plugged into the first database 200. The embodiment of the present application does not limit the specific type of the log collection unit 111.
[0075] In the application, the first database 200 records a binlog (binary log) file, which is used to record data changes when the first database 200 is working. Therefore, by collecting the log data of the first database 200, the data changes of the first database 200 can be obtained, and then the data currently recorded in the first database 200 can be obtained.
[0076] In the application, the binlog file is stored in binary form, and the log data can be converted by the log conversion unit 112 to obtain the data to be synchronized, so that the code type of the data to be synchronized is suitable for the first database 200 .
[0077] In the application, log collection and log conversion are implemented through the log collection unit 111 and the log conversion unit 112 to obtain the data to be synchronized. Compared with directly obtaining the data to be synchronized from the first database 200, the resource occupancy of the first database 200 can be reduced, and the acquisition speed of the data to be synchronized can be increased, thereby improving the working efficiency of the data synchronization system 100.
[0078] Figure 4 The example shows Figure 3 The corresponding interaction diagram of the data synchronization system 100 is shown.
[0079] like Figure 5 As shown, in one embodiment, based on Figure 3 In the corresponding embodiment, the conversion module 120 further includes an identification unit 121;
[0080] The identification unit 121 is used to read the identity information of the data to be synchronized, where the identity information includes the type of the first database 200 and the table name of the data to be synchronized;
[0081] The identification unit 121 is further used to read rule information of the data to be synchronized, where the rule information includes index configuration, constraint configuration and directory configuration.
[0082] In the application, the conversion module 120 also includes an identification unit 121, which can read the identity information of the data to be synchronized. The identity information may include the type of the first database 200 and the table name of the data to be synchronized. Specifically, the type of the first database 200 can be determined by reading the code type of the data to be synchronized; and the corresponding table name definition function can be determined according to the type of the first database 200, and the table name of the data to be synchronized can be retrieved based on the above corresponding table name function, so as to obtain the table name of the data to be synchronized.
[0083] In the application, the identification unit 121 can also read the rule information of the data to be synchronized, and the rule information may include the index configuration, constraint configuration and directory configuration of the data to be synchronized. The method of obtaining the rule information is consistent with the method of obtaining the table name. Specifically, the corresponding index definition function, constraint definition function and directory definition function can be determined according to the type of the first database 200, and the index retrieval of the data to be synchronized is performed based on the corresponding index definition function, so as to obtain the index configuration of the data to be synchronized, and the constraint retrieval of the data to be synchronized is performed based on the corresponding constraint definition function, so as to obtain the constraint configuration of the data to be synchronized, and the directory retrieval of the data to be synchronized is performed based on the corresponding directory definition function, so as to obtain the directory configuration of the data to be synchronized.
[0084] In one embodiment, the conversion module 120 is further configured to:
[0085] The identity information of the data to be synchronized is converted to obtain the identity information of the target data, wherein the code type of the identity information of the target data is applicable to the second database 300;
[0086] Sending the identity information of the target data to the second database 300;
[0087] Convert the rule information of the data to be synchronized to obtain the rule information of the target data, where the code type of the rule information of the target data is applicable to the second database 300;
[0088] The rule information of the target data is sent to the second database 300 .
[0089] In the application, the conversion module 120 can convert the identity information and rule information of the data to be synchronized. The conversion method is consistent with the conversion method of converting the data to be synchronized into the target data, which will not be repeated here. After the conversion, the identity information and rule information of the target data can be obtained, so that the target data can inherit the identity information and rule information of the data to be synchronized, thereby improving the synchronization integrity of the data to be synchronized and the target data.
[0090] like Figure 6 As shown, in one embodiment, based on Figure 1 In the corresponding embodiment, the data synchronization system 100 further includes an accuracy analysis module 130 and an optimization module 140;
[0091] The accuracy analysis module 130 is connected to the first database 200, the acquisition module 110 and the conversion module 120 respectively, and the optimization module 140 is connected to the accuracy analysis module 130, the acquisition module 110 and the conversion module 120 respectively;
[0092] The accuracy analysis module 130 is used to read the source data, log data and data to be synchronized of the first database 200, and analyze the data acquisition accuracy of the acquisition module 110;
[0093] The accuracy analysis module 130 is also used to read the source data, the data to be synchronized and the target data of the first database 200, and analyze the data conversion accuracy of the conversion module 120;
[0094] The optimization module 140 is used to optimize the acquisition module 110 according to the accuracy of data acquisition;
[0095] The optimization module 140 is further used to optimize the conversion module 120 according to the data conversion accuracy.
[0096] In the application, the specific connection relationship between the accuracy analysis module 130 and the collection module 110 can be connected to the log conversion unit 112, or connected to the log collection unit 111, or connected to the log conversion unit 112 and the log collection unit 111 respectively; the specific connection relationship between the optimization module 140 and the collection module 110 is consistent with the specific connection relationship between the above-mentioned accuracy analysis module 130 and the collection module 110, and will not be repeated here.
[0097] In the application, the accuracy analysis module 130 can determine whether the source data and the data to be synchronized match by comparing the source data of the first database 200 and the data to be synchronized, analyze the data acquisition accuracy of the acquisition module 110, and obtain the loss function of the acquisition module; it can also determine whether the log data and the data to be synchronized match by comparing the log data of the first database 200 and the data to be synchronized, analyze the data acquisition accuracy of the acquisition module 110, and obtain the loss function of the acquisition module.
[0098] In the application, the accuracy analysis module 130 can compare the source data and the target data of the first database 200 to determine whether the source data and the target data match, so as to analyze the data conversion accuracy of the conversion module 120 and obtain the loss function of the conversion module; it can also compare the log data and the target data of the first database 200 to determine whether the log data and the target data match, analyze the data conversion accuracy of the conversion module 120, and obtain the loss function of the conversion module. The embodiment of the present application does not impose any limitation on the specific analysis method of the accuracy analysis module 130.
[0099] In the application, the loss functions of the above-mentioned acquisition module and conversion module can be different types of loss functions such as 0-1 loss function, L1 norm loss function (Mean Absolute Error Loss Function), logarithmic loss function (Logistic Loss Function), square loss function (Quadratic Loss Function), exponential loss function (Exponential LossFunction), cross entropy loss function (Cross Entropy Loss Function), etc. The type of loss function can be selected according to actual needs.
[0100] In the application, the optimization module 140 can adjust the configuration parameters of the acquisition module 110 according to the data acquisition accuracy and the loss function of the acquisition module 110 to optimize the acquisition module 110; and adjust the configuration parameters of the conversion module 120 according to the data conversion accuracy and the loss function of the conversion module 120 to optimize the conversion module 120. The optimization module 140 can adjust the configuration parameters based on a neural network algorithm. The neural network algorithm can be a different type of training algorithm such as Gradient Descent, Newton's Method, Conjugate Gradient, Quasi-Newton Method or Least Squares Method. The specific neural network algorithm selected can be set according to actual needs.
[0101] In application, the accuracy analysis module 130 can be used to determine the data acquisition accuracy of the acquisition module 110 and the data conversion accuracy of the conversion module 120; in conjunction with the optimization module 140, the automatic adjustment of the configuration parameters of the acquisition module 110 and the automatic adjustment of the configuration parameters of the conversion module 120 can be achieved to improve the data synchronization accuracy of the data synchronization system 100.
[0102] In one embodiment, the data synchronization system 100 further includes a distribution module, which is connected to the acquisition module 110 and the conversion module 120 respectively;
[0103] The distribution module is used to obtain the data to be synchronized and the transaction timing, and send the data to be synchronized and the transaction timing to the conversion module 120 according to the load condition of the conversion module 120 .
[0104] In the application, the data synchronization system 100 may also include a distribution module, which is arranged between the acquisition module and the conversion module; the distribution module is used to collect the data to be synchronized and the transaction sequence, and send the data to be synchronized and the transaction sequence to the conversion module according to the load of the conversion module, so as to balance the processing load of the conversion module, avoid overload operation of the conversion module, and improve the operation stability of the conversion module. Specifically, the distribution module may be a Kafka server.
[0105] The data synchronization system provided in the embodiment of the present application collects the data to be synchronized of the first database through the collection module, and obtains the grammatical features of the data to be synchronized through the conversion module, converts the data to be synchronized according to the grammatical features and the grammatical correspondence between the first database and the second database to obtain the target data, and realizes data synchronization with the first database after sending the target data to the second database. Based on the lightweight collection module and the conversion module, the data synchronization system can be quickly deployed and the data synchronization of heterogeneous databases can be realized, thereby improving the data synchronization efficiency between distributed databases and commercial databases.
[0106] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various data synchronization system embodiments can be implemented.
[0107] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.
[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0109] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0110] In the embodiments provided in the present application, it should be understood that the disclosed terminal device and method can be implemented in other ways. For example, the terminal device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0111] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data synchronization system, characterized in that: It includes a collection module and a conversion module, wherein the collection module is connected to a first database, and the conversion module is connected to a second database; The acquisition module is used to obtain the data to be synchronized and the transaction sequence of the first database; The conversion module is used for: Acquire grammatical features of the data to be synchronized, where the grammatical features include grammatical attributes of each minimum unit data, and the grammatical features are used to indicate a logical relationship between the minimum unit data; According to the transaction sequence and the grammatical features of the data to be synchronized, and the grammatical correspondence between the first database and the second database, converting the data to be synchronized to obtain target data; sending the target data to the second database; Among them, the code type of the data to be synchronized is applicable to the first database, and the code type of the target data is applicable to the second database; the transaction sequence is used to reflect the transaction number, transaction information, logical relationship and transaction status of each data in the data to be synchronized.
2. The data synchronization system according to claim 1, characterized in that: The collection module includes a log collection unit and a log conversion unit; The log collection unit is used to obtain log data of the first database; The log conversion unit is used to convert the log data to obtain the data to be synchronized.
3. The data synchronization system according to claim 1, characterized in that: The conversion module also includes an identification unit; The identification unit is used to read the identity information of the data to be synchronized, where the identity information includes the type of the first database and the table name of the data to be synchronized.
4. The data synchronization system according to claim 3, characterized in that: The conversion module is further used to: convert the identity information of the data to be synchronized to obtain the identity information of the target data, wherein the code type of the identity information of the target data is applicable to the second database; The identity information of the target data is sent to the second database.
5. The data synchronization system according to claim 1, characterized in that: The conversion module also includes an identification unit; The identification unit is used to read the rule information of the data to be synchronized, and the rule information includes index configuration, constraint configuration and directory configuration.
6. The data synchronization system according to claim 5, characterized in that: The conversion module is also used for: Converting the rule information of the data to be synchronized to obtain the rule information of the target data, wherein the code type of the rule information of the target data is applicable to the second database; The rule information of the target data is sent to the second database.
7. The data synchronization system according to any one of claims 1 to 6, characterized in that: It also includes an accuracy analysis module, which is connected to the first database, the acquisition module and the conversion module respectively; The accuracy analysis module is used to read the source data, log data and the data to be synchronized of the first database, and analyze the data acquisition accuracy of the acquisition module; The accuracy analysis module is further used to read the source data of the first database, the data to be synchronized and the target data, and analyze the data conversion accuracy of the conversion module.
8. The data synchronization system according to claim 7, characterized in that: It also includes an optimization module, which is connected to the accuracy analysis module, the acquisition module and the conversion module respectively; The optimization module is used to optimize the acquisition module according to the data acquisition accuracy; The optimization module is also used to optimize the conversion module according to the data conversion accuracy.
9. The data synchronization system according to any one of claims 1 to 6, characterized in that: It also includes a distribution module, which is connected to the acquisition module and the conversion module respectively; The distribution module is used to obtain the data to be synchronized and the transaction timing, and send the data to be synchronized and the transaction timing to the conversion module according to the load condition of the conversion module.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, each step of the synchronization program in the data synchronization system according to any one of claims 1 to 9 is implemented.
