Data synchronization method and system

Abstract

The present invention provides a data synchronization method and system, comprising: determining a preset data publishing interface of a microservice platform module; a data subscription module obtains the data to be published through the preset data publishing interface, and selects a data synchronization mode and a target database for data synchronization. The present invention can realize incremental synchronization and full synchronization functions, and can realize synchronization of user interface configuration, thereby effectively ensuring the consistency of data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method and system. Background Art

[0002] At present, most network management systems need to carry a large amount of data operations. More and more real-time data query operations make the database load heavier and heavier, affecting the system's inability to display the results of direct queries. Therefore, in order to improve the system's processing efficiency for data operations, the system can distribute a large number of data operations to multiple different devices for execution. And because the data carried by large network management systems comes from multiple devices, or multiple systems need the same data at the same time. Therefore, data synchronization is required between devices and between systems, so that the data displayed on the system is more comprehensive and the data information stored on each device is completely consistent, so as to avoid data inconsistency and cause the system to fail to operate normally.

[0003] The data synchronization operation not only requires the application system to perform add, delete, modify and query operations on the database, but also the data warehouse needs to obtain different transaction data from numerous databases to improve its own data set. As for data synchronization, there are two methods: incremental synchronization and full synchronization. These two methods have their own advantages and problems. The processing logic of incremental synchronization is not very simple, and it is easy to cause data inconsistency over time. Full synchronization generally does not cause data inconsistency, but as the amount of data increases, the full amount of data will often be very large in the end. Summary of the invention

[0004] The present invention provides a data synchronization method and system to solve the defects in the prior art.

[0005] In a first aspect, the present invention provides a data synchronization method, comprising:

[0006] Determine the preset data publishing interface of the microservice platform module;

[0007] The step of determining a preset data publishing interface of the microservice platform module includes:

[0008] The microservice platform module publishes a RESTFUL interface in JSON format;

[0009] The data subscription module obtains the data to be published by calling the RESTFUL interface, and selects the data synchronization method and the target database for data synchronization, including:

[0010] Query the data to be published based on SQL format and configure database information on the microservice platform module;

[0011] Complete synchronization configuration in the corresponding column of the data subscription module;

[0012] Submit a subscription application, and after approval, configure the synchronous execution configuration on the data subscription module;

[0013] Start the synchronization task and connect to the destination database according to the synchronization execution configuration;

[0014] The data synchronization method includes incremental synchronization and full synchronization.

[0015] In one embodiment, it further includes:

[0016] Save the execution log of each data synchronization task and mark the execution result of the data synchronization task on the data subscription module.

[0017] In one embodiment, the database information includes database IP, database port, database type, database service name, database user name, database password and SQL query statement.

[0018] In one embodiment, the step of completing synchronization configuration in a corresponding column of the data subscription module includes:

[0019] Configure the interface address in the service product column of the data subscription module, add the service in the user management column, and configure the destination database in the data source column.

[0020] In a second aspect, the present invention further provides a data synchronization system, comprising:

[0021] The first processing module is used to determine the preset data publishing interface of the microservice platform module;

[0022] The step of determining a preset data publishing interface of the microservice platform module includes:

[0023] The microservice platform module publishes a RESTFUL interface in JSON format;

[0024] The second processing module is used for the data subscription module to obtain the data to be published by calling the RESTFUL interface, and select the data synchronization method and the target database for data synchronization, including:

[0025] Query the data to be published based on SQL format and configure database information on the microservice platform module;

[0026] Complete synchronization configuration in the corresponding column of the data subscription module;

[0027] Submit a subscription application, and after approval, configure the synchronous execution configuration on the data subscription module;

[0028] Start the synchronization task and connect to the destination database according to the synchronization execution configuration;

[0029] The data synchronization method includes incremental synchronization and full synchronization.

[0030] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned data synchronization methods are implemented.

[0031] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the data synchronization methods described above.

[0032] The data synchronization method and system provided by the present invention can realize incremental synchronization and full synchronization functions by applying a microservice-based data synchronization method, and can realize synchronization of user interface configuration, thereby effectively ensuring the consistency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flowchart of the data synchronization method provided by the present invention;

[0035] Figure 2 It is a structural schematic diagram of the data synchronization system provided by the present invention;

[0036] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In view of the data synchronization problem existing in the prior art, the present invention proposes a data synchronization method based on microservices. Figure 1is a flow chart of the data synchronization method provided by the present invention, such as Figure 1 As shown, including:

[0039] S1, determine the preset data publishing interface of the microservice platform module;

[0040] S2, the data subscription module obtains the data to be published through the preset data publishing interface, and selects the data synchronization method and the target database for data synchronization.

[0041] Specifically, the microservices applied by the present invention are a cloud-native architecture approach in which a single application is composed of many smaller components or services that are loosely coupled and independently deployable, with the advantage that the code can be updated more easily, teams can use different stacks for different components, and components can be scaled independently of each other, thereby reducing the waste and cost caused by having to scale the entire application because a single function may face excessive load.

[0042] The core of the microservice architecture is to solve the service governance problem after the application is microserviced. It is necessary to use one of the most important components in the microservice architecture: the service registry. All services are registered with the service registry, and the current list of available services can also be obtained from the service registry. After solving the service discovery problem, it is necessary to solve the second problem brought about by the distributed deployment of microservices: the problem of service configuration management. When the number of services exceeds a certain level, if it is necessary to maintain the configuration files of each service separately in each service, the second important component in the microservice architecture is needed: the configuration center. In addition, the service governance of microservices also involves many contents, such as: ensuring high availability through mechanisms such as fuses and current limiting; load balancing of calls between microservices; distributed transactions (2PC, 3PC, TCC, LCN, etc.); service call chain tracking, etc.

[0043] Correspondingly, the data synchronization method proposed in the present invention has the main functions of service publishing, data subscription and data storage, and the corresponding system modules include a microservice platform module and a data subscription module. The microservice platform publishes a RESTFUL interface in JSON format, and the data subscription module obtains data by calling the RESTFUL interface of the microservice, and then selects the data synchronization mode and the target database through the data subscription module to perform the data synchronization task.

[0044] The present invention can realize incremental synchronization and full synchronization functions by applying a microservice-based data synchronization method, and can realize synchronization of user interface configuration, thereby effectively ensuring the consistency of data.

[0045] Based on the above embodiment, it also includes:

[0046] Save the execution log of each data synchronization task and mark the execution result of the data synchronization task on the data subscription module.

[0047] Specifically, after the data synchronization task is executed, the execution log of each synchronization task is saved, and the result of the data synchronization task execution is marked on the data subscription module as a success state or a synchronization abnormal state.

[0048] Usually, according to the configured synchronization task, the execution log is saved for each execution, and the system can query the log. If the execution is abnormal, there will be an abnormal mark in the task list.

[0049] The present invention can completely retain the synchronization execution process by saving the log of executing the synchronization task and classifying and marking the execution results, which is convenient for subsequent work backtracking.

[0050] Based on any of the above embodiments, step S1 in the method includes:

[0051] The microservice platform module publishes a RESTFUL interface in JSON format.

[0052] Specifically, among the three mainstream Web service interaction schemes, REST is simpler and clearer than SOAP (Simple Object Access protocol) and XML-RPC. Whether it is URL processing or Payload encoding, REST tends to be designed and implemented with a simpler and lighter method.

[0053] Therefore, the present invention adopts the microservice platform module to publish a RESTFUL interface in JSON format.

[0054] Based on any of the above embodiments, step S2 in the method includes:

[0055] Query the data to be published based on SQL format and configure database information on the microservice platform module;

[0056] Complete synchronization configuration in the corresponding column of the data subscription module;

[0057] Submit a subscription application, and after approval, configure the synchronous execution configuration on the data subscription module;

[0058] The synchronization task is started, and the destination database is connected according to the synchronization execution configuration.

[0059] The database information includes database IP, database port, database type, database service name, database user name, database password and SQL query statement.

[0060] Wherein, completing the synchronization configuration in the corresponding column of the data subscription module includes:

[0061] Configure the interface address in the service product column of the data subscription module, add the service in the user management column, and configure the destination database in the data source column.

[0062] The data synchronization method includes incremental synchronization and full synchronization.

[0063] Specifically, the data to be published is first queried in the form of SQL, and the database information of the data is configured on the microservice platform module, including database IP, database port, database type, database service name, database user name, database password and SQL query statement. After the interface is configured, the publishing is completed. The client accesses the interface to obtain data from the database and returns it in JSON data format.

[0064] Secondly, configure the interface address published by the microservice platform in the service product column on the data subscription module, add services in the user management column, and configure the destination database of the synchronization process in the data source column.

[0065] Next, submit the subscription application. After approval, you will have the authority to call this service.

[0066] Then, configure the data source interface address, scheduling mode, scheduling frequency, synchronization mode, receiving library, receiving table, key fields, deletion conditions and the correspondence between fields on the data subscription module.

[0067] After the synchronization task is started, connect to the database of the receiving library according to the configuration. The corresponding data synchronization methods include incremental synchronization and full synchronization. In incremental synchronization, delete the data in the target table that is not in the temporary table according to the key field, insert the data that does not exist in the target table, and update the data that already exists in the target table. If the execution is abnormal, perform a rollback operation, close the database connection, and delete the temporary table; in full synchronization, clear the target table data and insert data into the target table in batches.

[0068] The present invention realizes cross-product or cross-system data synchronization, that is, realizes incremental synchronization and full synchronization functions, and can configure synchronization through an interface, thereby simplifying the synchronization process and synchronization workload.

[0069] The data synchronization system provided by the present invention is described below. The data synchronization system described below and the data synchronization method described above can be referenced to each other.

[0070] Figure 2 is a schematic diagram of the structure of the data synchronization system provided by the present invention, such as Figure 2As shown, it includes: a first processing module 21 and a second processing module 22, wherein:

[0071] The first processing module 21 is used to determine the preset data publishing interface of the microservice platform module; the second processing module 22 is used for the data subscription module to obtain the data to be published through the preset data publishing interface, and select the data synchronization method and the target database for data synchronization.

[0072] The present invention can realize incremental synchronization and full synchronization functions by applying a microservice-based data synchronization system, and can realize synchronization of user interface configuration, thereby effectively ensuring the consistency of data.

[0073] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the data synchronization method, which includes: determining the preset data publishing interface of the microservice platform module; the data subscription module obtains the data to be published through the preset data publishing interface, and selects the data synchronization mode and the target database for data synchronization.

[0074] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data synchronization method provided by the above methods, and the method includes: determining a preset data publishing interface of the microservice platform module; the data subscription module obtains the data to be published through the preset data publishing interface, and selects the data synchronization method and the destination database for data synchronization.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned data synchronization methods, the method comprising: determining a preset data publishing interface of the microservice platform module; the data subscription module obtains the data to be published through the preset data publishing interface, and selects a data synchronization method and a destination database for data synchronization.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Determine the preset data publishing interface of the microservice platform module; The step of determining a preset data publishing interface of the microservice platform module includes: The microservice platform module publishes a RESTFUL interface in JSON format; The data subscription module obtains the data to be published by calling the RESTFUL interface, and selects the data synchronization method and the target database for data synchronization, including: Query the data to be published based on SQL format and configure database information on the microservice platform module; Complete synchronization configuration in the corresponding column of the data subscription module; Submit a subscription application, and after approval, configure the synchronous execution configuration on the data subscription module; Start the synchronization task and connect to the destination database according to the synchronization execution configuration; The data synchronization method includes incremental synchronization and full synchronization.

2. The data synchronization method according to claim 1, characterized in that: Also includes: Save the execution log of each data synchronization task and mark the execution result of the data synchronization task on the data subscription module.

3. The data synchronization method according to claim 1, characterized in that: The database information includes database IP, database port, database type, database service name, database user name, database password and SQL query statement.

4. The data synchronization method according to claim 1, characterized in that: The synchronization configuration is completed in the corresponding column of the data subscription module, including: Configure the interface address in the service product column of the data subscription module, add the service in the user management column, and configure the destination database in the data source column.

5. A data synchronization system, characterized in that: include: The first processing module is used to determine the preset data publishing interface of the microservice platform module; The step of determining a preset data publishing interface of the microservice platform module includes: The microservice platform module publishes a RESTFUL interface in JSON format; The second processing module is used for the data subscription module to obtain the data to be published by calling the RESTFUL interface, and select the data synchronization method and the target database for data synchronization, including: Query the data to be published based on SQL format and configure database information on the microservice platform module; Complete synchronization configuration in the corresponding column of the data subscription module; Submit a subscription application, and after approval, configure the synchronous execution configuration on the data subscription module; Start the synchronization task and connect to the destination database according to the synchronization execution configuration; The data synchronization method includes incremental synchronization and full synchronization.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data synchronization method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 4 are implemented.

Images