A redis database-based data real-time synchronization method and system

Abstract

The application discloses a kind of based on redis database's data real-time synchronization method and system, comprising the following steps: A: sender carries out business operation;B: data acquisition is carried out;C: data filtering is carried out;D: whether the judgment of transaction is concerned;E: transaction monitoring is carried out;F: after transaction submission, data is sent;G: whether transaction is successfully executed is judged;H: if sending is not successful, then stop data synchronization;I: the data of transaction is sent to redis database;G: data persistence operation is carried out;K: redis database carries out data synchronization operation;L: subscription party carries out synchronous data monitoring;M: subscription party carries out business operation, then carries out data real-time synchronization.The application can efficiently realize the real-time synchronization of data, effectively meet the demand of shared data in different business systems under micro-service architecture environment.

Description

TECHNICAL FIELD

[0001] The application relates to a data synchronization method under a micro-service architecture, in particular to a data real-time synchronization method and system based on a redis database. BACKGROUND

[0002] At present, in the micro-service architecture environment, in order to improve the running efficiency and search accuracy, NoSQL databases such as redis and mongoDB are used in large quantities, and full-text retrieval services such as Solr and Elasticsearch are also used in large quantities. Due to the demand for shared data in different business systems, the importance of data synchronization is increasingly evident.

[0003] In the prior art, the commonly used data synchronization methods include the following:

[0004] 1. Data synchronization is realized by using application code;

[0005] This method has high coupling degree with business code, the execution efficiency of the business code is low, and transaction consistency cannot be guaranteed;

[0006] 2. Data synchronization is realized by a timing task (spring task, Quartz, XXLJOB);

[0007] The real-time performance of data under this method is not high, and the use demand cannot be well met;

[0008] 3. Data synchronization is realized by MQ;

[0009] This method needs to add code for sending messages to MQ in the business code, the data calling interface is coupled, the system availability is reduced, the cost of maintaining MQ is additionally increased, and the problem of data consistency is also faced;

[0010] 4. Data synchronization is realized by technical middleware, such as Canal, Databus, MySQL-Streamer and Debezium;

[0011] When data synchronization is realized by Canal, it is a MySQL binlog-based incremental data synchronization method, which is limited by technology and only supports MySQL;

[0012] When data synchronization is realized by Databus, although it supports oracle and mysql, the technology is not mature, and the use and maintenance cost is too high by using self-developed relay as a change distribution platform;

[0013] When data synchronization is realized by MySQL-Streamer, it is a python-based data pipeline, which is limited by technology and only supports MySQL;

[0014] When data synchronization is implemented by Debezium, although it supports change capture of three data sources of MySQL, MongoDB and PostgreSQL, it does not support Oracle and domestic databases, and needs to rely on kafka;

[0015] In summary, the existing data synchronization method under the microservice architecture cannot effectively meet the demand of sharing data in different business systems under the microservice architecture. SUMMARY

[0016] The purpose of the present application is to provide a redis database-based data real-time synchronization method and system, which can efficiently realize real-time synchronization of data and effectively meet the demand of sharing data in different business systems under the microservice architecture.

[0017] The present application adopts the following technical solutions:

[0018] A redis database-based data real-time synchronization method comprises the following steps:

[0019] A: the sender performs business operation; then enter step B;

[0020] B: data collection is performed by using a data proxy module; then enter step C;

[0021] C: data filtering is performed by using a data processing module; then enter step D;

[0022] D: the data sending module judges whether to care about the transaction according to the strategy defined by the sender; if yes, enter step E; if no, enter step I;

[0023] E: the data sending module performs transaction listening, and then enter step F;

[0024] F: transaction submission is performed, and data is sent after transaction submission, and then enter step G;

[0025] G: the data sending module judges whether the transaction is successfully executed; if yes, enter step I; if no, enter step H;

[0026] H: the sender database transaction is not successfully executed, and the data synchronization is stopped;

[0027] I: the data of the transaction is sent to the redis database; then enter step G;

[0028] G: the redis database performs data persistence operation; then enter step K;

[0029] K: the redis database performs a data synchronization operation; then step L is entered;

[0030] L: the subscriber performs a synchronization data listening operation; then step M is entered;

[0031] M: the subscriber performs a business operation, and then performs real-time data synchronization.

[0032] In step B, the data proxy module is used to cut into the business operation process of the sender, and can use a Spring AOP module, through an ASM bytecode framework or byteBuddy, and use dynamic bytecode technology to realize data collection.

[0033] In step C, the data processing module uses a Google Gson serialization tool.

[0034] In step D, the data sending module realizes real-time transmission of data through spring-data-redis.

[0035] In step D, the strategy defined by the sender includes data generated by the database operation of the sender and whether the database operation is concerned about transactions.

[0036] In step E, the data sending module realizes transaction listening through a TransactionSynchronizationManager transaction manager.

[0037] In step G, the data sending module judges whether the transaction is successfully executed through a callback action of the TransactionSynchronizationManager.

[0038] Using the real-time data synchronization method based on the redis database, one sender can synchronize transactions to one subscriber, one sender can synchronize transactions to multiple subscribers, one sender can synchronize transactions to another subscriber through an intermediate sender or an intermediate subscriber, or multiple senders can synchronize transactions to one subscriber.

[0039] The synchronization system realized by the real-time synchronization method includes a sender, a data proxy module, a data processing module, a data sending module, a redis database, and a subscriber.

[0040] The data proxy module is used to cut into the business operation process of the sender.

[0041] The data processing module is used to filter the transmitted data according to the filter field configured by the sender.

[0042] The data sending module is used for real-time transmission of data.

[0043] The redis database is used for storing data.

[0044] The sender, the data proxy module, the data processing module, the data sending module, the redis database and the subscriber perform data interaction according to the redis database-based data real-time synchronization method in any one of claims 1 to 9.

[0045] The system comprises one or more senders and one or more subscribers.

[0046] The application can decouple the database operation and the sending of the synchronization data after the business data transaction is submitted, effectively overcomes the common drawbacks of the conventional scheme of using the MQ message middleware for data synchronization, such as the data consistency problem. The application overcomes the problem in the prior art that the sending message and the database operation are in one transaction, and the long-time sending of the message will occupy the database connection, thereby causing the problem of long transaction of the database.

[0047] The application has the characteristics of lightweight and easy integration, supports multiple integrations in a product system to meet a complex synchronization topology system, and can realize one-to-one unidirectional synchronization, one-to-many unidirectional synchronization, cascaded unidirectional synchronization and many-to-one unidirectional synchronization. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The figure is a flowchart of the application;

[0049] Figure 2 The figure is a synchronization system diagram of one-to-one unidirectional synchronization in the application;

[0050] Figure 3 The figure is a synchronization system diagram of one-to-many unidirectional synchronization in the application;

[0051] Figure 4 The figure is a synchronization system diagram of cascaded unidirectional synchronization in the application;

[0052] Figure 5 The figure is a synchronization system diagram of many-to-one unidirectional synchronization in the application. DETAILED DESCRIPTION

[0053] The application will be described in detail below in combination with the drawings and embodiments:

[0054] As shown in the figure, the redis database-based data real-time synchronization method comprises the following steps: Figure 1 A: The sender performs business operation; then enters step B;

[0055] The business operation refers to the action of adding, modifying and deleting data in the sender.

[0056] The business operation refers to the action of adding, modifying and deleting data in the sender.

[0057] B: data collection is performed by using the data agent module; then, step C is entered;

[0058] In the present application, the data agent module is used to cut into the service operation process of the sender, and can adopt a Spring AOP module, and the data collection is realized by using a dynamic bytecode technology through an ASM bytecode framework or byteBuddy;

[0059] In the present embodiment, in the process of data collection, the data agent module adopts an AOP Around notification, and the unified interception of the program function is performed through a pre-compiled mode and a dynamic agent during running, so as to isolate each part of the business logic, thereby reducing the coupling degree between each part of the business logic, improving the reusability of the program, and improving the development efficiency.

[0060] In the present embodiment, the data agent module combines the ASM bytecode framework, and the disadvantage that the AOP Around notification can only obtain the method parameters is overcome; the method parameter name in the bytecode can be accurately and quickly obtained, and a MAP is used as a storage collection to match the data in the program and find the time complexity O(1), so that the sending of the synchronous data is quickly and efficiently completed. Meanwhile, the sender does not need to write a large amount of code, and does not need to invest a large amount of research and development cost and maintain the cost of service resources like other synchronization middleware. Only the synchronization channel annotation needs to be introduced on the business method, and the related properties need to be configured.

[0061] In the present embodiment, the data agent module adopts a synchronization strategy based on the byte-buddy bytecode enhancement technology, and the byte-buddy bytecode enhancement technology has certain advantages in performance compared with the JDK dynamic agent, the cglib and the javassist; during the startup of the sender application program, a customized synchronization executor is created, the sender can customize the data form, and the synchronization executor is explicitly called; the program is completely dependent on the interface, and the development cost is greatly reduced. The present application has very low invasiveness to the original business logic code, and does not change the original business logic, but only extends the original program, thereby satisfying the customization of the business and ensuring the real-time performance and the efficiency of the synchronous data.

[0062] C: data filtering is performed by using the data processing module; then, step D is entered;

[0063] In the present application, the data processing module is used to convert the format of the transmission data, and the transmission data is filtered in the conversion process according to the filtering field configured by the sender, so that the data transmission amount can be effectively reduced, and the data synchronization efficiency can be improved. In the present embodiment, the data processing module can adopt a Google Gson serialization tool;

[0064] D: the data sending module judges whether to care about the transaction according to the strategy defined by the sender; if yes, it goes to step E; if no, it goes to step I;

[0065] In the application, the data sending module is used for real-time transmission of data, and can be implemented by spring-data-redis. The strategy defined by the sender includes data generated by database operation of the sender and whether the database operation cares about the transaction. The transaction refers to a series of database operations defined by the sender, and can be regarded as a complete logical processing work unit.

[0066] E: the data sending module performs transaction listening, and then goes to step F;

[0067] In the application, the transaction listening refers to that the data sending module listens whether the database operation of the sender succeeds or not, and is used for ensuring consistency of data synchronization and the database operation of the sender, so as to add corresponding specified operations before and after the transaction. In the embodiment, the data sending module can realize transaction listening on the TransactionSynchronizationManager transaction manager.

[0068] F: transaction submission is performed, and data is sent after the transaction submission, and then goes to step G;

[0069] In the application, the transaction submission refers to that the database operation of the sender succeeds, and is used for sending the synchronization data by the data sending module after the transaction submission succeeds.

[0070] G: the data sending module judges whether the transaction is executed successfully; if yes, it goes to step I; if no, it goes to step H;

[0071] In the application, the data sending module can judge whether the transaction is executed successfully through the callback action of the TransactionSynchronizationManager.

[0072] H: the transaction of the sender database is not executed successfully, and the data synchronization is stopped;

[0073] I: the data of the transaction is sent to the redis database; and then goes to step G;

[0074] G: the redis database performs data persistence operation; and then goes to step K;

[0075] Because the data will be cleared from the memory by the operating system due to non-normal reason downtime, redis process closing or computer rear closing. In order to quickly recover the data after the above-mentioned situations occur, in the application, the redis database is used for data persistence operation, so as to save important data.

[0076] K: redis database carries out data synchronization operation; then enter step L;

[0077] L: the subscription party carries out synchronization data listening; then enter step M;

[0078] M: the subscription party carries out business operation, and then carries out real-time data synchronization.

[0079] In the prior art, since the message sending and database operation are in a transaction, the database connection will be occupied when the message sending time is too long, thereby causing the problem of long transaction of the database.

[0080] The real-time data synchronization method based on the redis database has the characteristics of lightweight and easy integration, supports multiple integrations in a product system to meet a complex synchronization topology system, and can realize one-to-one unidirectional synchronization, one-to-many unidirectional synchronization, cascaded unidirectional synchronization and many-to-one unidirectional synchronization, as shown in Figures 2 to 5 .

[0081] Figure 2 In the present application, the default synchronization system is one-to-one unidirectional synchronization, that is, a sender synchronizes transactions to a subscription party by using the real-time data synchronization method. Figure 3 As shown in the figure, the real-time data synchronization method can also be used to realize one-to-many unidirectional synchronization by deploying multiple subscribers, that is, a sender synchronizes transactions to multiple subscription parties. Figure 4 As shown in the figure, the real-time data synchronization method can also be used to realize cascaded unidirectional synchronization, that is, a sender synchronizes transactions to another subscription party through an intermediate sender or an intermediate subscription party. Figure 5 As shown in the figure, the real-time data synchronization method can also be used to realize many-to-one unidirectional synchronization, that is, multiple senders synchronize transactions to a subscription party.

[0082] In the present application, the synchronization system realized by using the real-time data synchronization method based on the redis database includes one or more senders, a data agent module, a data processing module, a data sending module, a redis database and one or more subscription parties; the sender, the data agent module, the data processing module, the data sending module, the redis database and the subscription party carry out data interaction according to the real-time data synchronization method based on the redis database in any one of claims 1 to 9.

[0083] Among them, the data agent module is used for cutting into the sender business operation process; the data processing module is used for filtering the transmitted data according to the filter field configured by the sender; the data sending module is used for real-time transmission of data; and the redis database is used for data storage.

Claims

1. A redis database-based data real-time synchronization method, characterized in that, It comprises the following steps: A: the sender carries out business operation; then enters step B; B: data collection is carried out by using a data agent module; then enters step C; In step B, the data agent module is used to cut into the business operation process of the sender, and a Spring AOP module is adopted to realize data collection by means of an ASM bytecode framework or byteBuddy and a dynamic bytecode technology; The data agent module adopts AOP Around notification, and carries out unified interception of program functions by means of pre-compilation and dynamic proxy during running, so as to isolate each part of the business logic; C: data filtering is carried out by using a data processing module; then enters step D; D: the data sending module judges whether a transaction is concerned according to a strategy defined by the sender; if yes, enters step E; if no, enters step I; E: the data sending module carries out transaction listening, and then enters step F; F: transaction submission is carried out, and data is sent after the transaction is submitted, and then enters step G; G: the data sending module judges whether the transaction is successfully executed; if yes, enters step I; if no, enters step H; H: the database transaction of the sender is not successfully executed, and data synchronization is stopped; I: the data of the transaction is sent to a redis database; then enters step G; G: the redis database carries out data persistence operation; then enters step K; K: the redis database carries out data synchronization operation; then enters step L; L: the subscriber carries out synchronization data listening; then enters step M; M: the subscriber carries out business operation, and then carries out real-time data synchronization.

2. The redis database-based data real-time synchronization method according to claim 1, characterized in that: In step C, the data processing module adopts a Google Gson serialization tool. 3.The redis database-based data real-time synchronization method according to claim 1, characterized in that: In step D, the data sending module realizes real-time transmission of data by means of spring-data-redis.

4. The redis database-based data real-time synchronization method according to claim 1, characterized in that: In step D, the strategy defined by the sender comprises data generated by database operation of the sender and whether the database operation concerns a transaction.

5. The redis database-based data real-time synchronization method according to claim 1, characterized in that: In step E, the data sending module realizes transaction listening by means of a TransactionSynchronizationManager transaction manager.

6. The redis database-based data real-time synchronization method according to claim 1, characterized in that: In step G, the data sending module judges whether the transaction is successfully executed by means of callback action of the TransactionSynchronizationManager.

7. The redis database-based data real-time synchronization method according to claim 1, characterized in that: By using the real-time data synchronization method based on the redis database, one sender can synchronize a transaction to one subscriber, one sender can synchronize a transaction to multiple subscribers, one sender can synchronize a transaction to another subscriber through an intermediate sender or an intermediate subscriber, or multiple senders can synchronize a transaction to one subscriber.

8. A synchronization system implemented using the real-time synchronization method of any one of claims 1 to 7, characterized in that: It comprises a sender, a data agent module, a data processing module, a data sending module, a redis database and a subscriber; The data agent module is used to cut into the business operation process of the sender; The data processing module is used to filter the transmitted data according to the filtering field configured by the sender; The data sending module is used for real-time transmission of data; The redis database is used for data storage. The sender, the data agent module, the data processing module, the data sending module, the redis database and the subscriber interact with each other according to the data real-time synchronization method based on the redis database in any one of claims 1 to 7.

9. The synchronization system of claim 8, wherein: The system comprises one or more senders and one or more subscribers.

Images