Data synchronization method, system and device of big data platform and medium

Abstract

The invention provides a data synchronization method, system and device for a big data platform and a medium, and the method comprises the steps: after the big data platform completes data modification or newly-added operation, the modified or newly-added data is inserted into a MySQL main library, and when the MySQL main library executes a modification or newly-added command, a binlog record operation command is automatically generated; if it is detected that the binlog changes through the monitoring thread, reading a binlog file from the last offset, analyzing the changed part of the binlog, pushing the change of MySQL to a message queue kafka, monitoring unconsumed messages in the kafka by using a kafka monitor, taking out the messages, and pushing the messages to a data storage and processing component; and reminding the user to perform data synchronization. According to the invention, efficient and real-time synchronization of the data is realized, and the consistency of the data and the expandability of the system are ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically to a data synchronization method, system, device and medium for a big data platform. Background Art

[0002] In recent years, with the rapid advancement of Internet technology, the computer service system has undergone significant changes, transforming and upgrading from the initial monolithic service model to a complex architecture with multiple clusters and microservices in parallel. In these microservice systems, data does not only exist in a single database, but various middleware such as ElasticSearch and Redis also need data support. Different platforms and systems may need to use the same data for calculation and display. When the data in the database changes, the changes need to be synchronized to other databases and other required components, which requires maintaining data consistency. In the big data platform, data is mainly stored in the MySQL database, and user login information is usually stored in redis to speed up data loading. Usually, a system will have multiple modules, and different modules may be implemented through different services. Data consistency must also be maintained between multiple modules.

[0003] Currently, in order to ensure the consistency of data in the data platform, the method of configuring triggers in the database is usually adopted. This method requires creating triggers in the database table to be synchronized in advance and configuring the trigger rules. When the data information in the table changes, the trigger can mobilize the external program to perform synchronization operations.

[0004] However, the configuration of triggers in the database is very complicated. If too many triggers are configured, the performance of the database will be reduced and the operation efficiency of the database will be reduced. In addition, if multiple databases are deployed in different servers or systems, triggers need to be configured in each database, which is complicated and not conducive to later maintenance. When the database table is modified or a new table is added, the triggers of each database also need to be modified accordingly. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a data synchronization method, system, device and medium for a big data platform, which realizes efficient and real-time synchronization of data by real-time monitoring of MySQL binlog logs and automatically pushing data changes to the message queue, which is then distributed to the data storage and processing components by the message queue, thereby ensuring data consistency and system scalability.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, the present invention discloses a data synchronization method for a big data platform, comprising:

[0008] After the big data platform completes the modification or addition of data, the modified or added data is inserted into the MySQL main database. The MySQL main database automatically generates a binlog log when executing the data modification or addition command to record the specific command of the operation;

[0009] Set up a monitoring thread to detect changes in the binlog logs of the MySQL master database in real time. When changes are detected in the binlog logs, read the binlog log information from the preset offset, and parse the data changes by performing structured processing on the binlog log information.

[0010] Encapsulate the parsed data changes into a message entity, execute the message sending operation, and push the changed data to the message queue;

[0011] The listener of the message queue is used to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the data storage and processing component for synchronization processing.

[0012] Further, the structured processing of binlog log information includes:

[0013] The irregular strings in the binlog log are structured according to the schema information of the database table to convert the operation commands in the binlog log into a structured data format;

[0014] The schema information of the database table includes: table name, field name, and field type.

[0015] Further, encapsulating the parsed data change content into a message entity, executing a message sending operation, and pushing the changed data to a message queue includes:

[0016] The parsed data changes are encapsulated into message entities, and the message sending operation is performed through the API of the message queue middleware Kafka to push the changed data to the Kafka queue;

[0017] When a message sending operation fails, a retry strategy is implemented.

[0018] Further, the retry strategy includes:

[0019] When a message fails to be sent, it will be automatically resent according to the preset retry times and retry interval until the message is successfully sent or the maximum retry times are reached.

[0020] Further, the data change content is pushed to the data storage and processing component for synchronization processing, including:

[0021] Push data changes to the Redis cache system, BI analysis system, and MySQL slave database for synchronization.

[0022] Furthermore, the Redis cache system uses an adapted Redis data structure to store data according to business requirements, and the adapted Redis data structure includes a string, a hash table, and a list.

[0023] Furthermore, the method further comprises:

[0024] Real-time monitoring of MySQL master database 、 Message Queues 、 The data stores and processes the running status of components. If any abnormality is found, an alarm message will be issued and automatic recovery will be attempted.

[0025] In a second aspect, the present invention further discloses a data synchronization system for a big data platform, comprising:

[0026] The data operation recording module is used to insert the modified or added data into the MySQL main database after the big data platform completes the modification or addition of data. The MySQL main database automatically generates a binlog log when executing data modification or addition commands to record the specific commands of the operation;

[0027] The log monitoring and parsing module is used to set up a monitoring thread to detect the changes in the binlog logs of the MySQL main database in real time. When a change in the binlog log is detected, the binlog log information is read from the preset offset, and the data change content is parsed by structurally processing the binlog log information;

[0028] The message push module is used to encapsulate the parsed data change content into a message entity, execute the message sending operation, and push the changed data to the message queue;

[0029] The synchronization module is used to use the listener of the message queue to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the data storage and processing component for synchronization processing.

[0030] In a third aspect, the present invention further discloses a data synchronization device for a big data platform, comprising:

[0031] Storage, used to store data synchronization programs of the big data platform;

[0032] A processor is used to implement the steps of the data synchronization method of the big data platform as described in any of the above items when executing the data synchronization program of the big data platform.

[0033] In a fourth aspect, the present invention further discloses a readable storage medium, on which a data synchronization program for a big data platform is stored. When the data synchronization program for the big data platform is executed by a processor, the steps of the data synchronization method for the big data platform as described in any of the above items are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention uses a synchronization mechanism based on binlog logs to enable each data storage and processing component (such as Redis, Kafka, BI) to run and expand independently without directly relying on the query performance or structural changes of the MySQL database. This loosely coupled design makes the system more flexible and easy to maintain and upgrade.

[0036] 2. The present invention performs data synchronization based on the actual change operations (such as INSERT, UPDATE, and DELETE) of the MySQL database, which can ensure the accuracy and integrity of the data. Compared with regular full data synchronization or incremental synchronization based on query results, the synchronization method based on binlog logs can capture data changes more accurately.

[0037] 3. The present invention can support complex data processing processes, such as data cleaning, conversion, aggregation, etc., by pushing data changes to the Kafka message queue in real time. These processing processes can be performed asynchronously without blocking the operation of the original database, thereby improving the throughput and response speed of the entire system.

[0038] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 It is a method flow chart of a specific implementation mode of the present invention.

[0041] Figure 2 It is a system structure diagram of a specific implementation mode of the present invention.

[0042] In the figure, 1. Data operation recording module; 2. Log monitoring and analysis module; 3. Message push module; 4. Synchronization module. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The present invention discloses a data synchronization method for a big data platform. The method is applied to the big data platform. The system architecture of the big data platform can be divided into three modules: management background, data processing middle platform and BI report. The management background mainly manages the system and sets the basic configuration of the system, including personnel, roles, menus, permissions and other settings. The data processing middle platform is mainly used for data introduction, data calculation, data processing and other operations. The BI report is used for data display. According to the needs, the large screen can be customized and edited to query and synchronize the data to the components of the large screen.

[0045] In the big data platform system, data is mainly stored in MySQL database, and some data is stored in Redis, Hive, and HDFS. Redis mainly stores data in memory, and uses persistence technology to refresh data to disk for storage regularly. Since Redis operates in memory, it reads and writes data very quickly, and can perform tens of thousands of operations per unit time. For example, after a user logs in, the user's credentials can be stored in redis. Unprocessed data can be stored in HDFS, and after data processing, it will be stored in MySQL. BI reports and large screens need to retrieve data from MySQL. In this system, in order to ensure the stability of the system, two databases are set up. One database is mainly connected to the management background and the data processing middle platform to complete the data modification operation, that is, the write operation. The other database is mainly connected to the BI report. BI queries the database data through SQL statements, so this database mainly performs read operations. This read-write separation database design can greatly improve the operating efficiency of the database. In addition, it can also improve the robustness of the system. When a database has a problem, it can temporarily switch to another database to work without system downtime. In order to maintain data consistency, data synchronization is also required between the two databases. When the write database operates on the data, the data changes need to be synchronized to the read database.

[0046] See also Figure 1As shown, based on the above big data platform system, this embodiment provides a data synchronization method for a big data platform, including the following steps:

[0047] S1: After the big data platform completes the modification or addition of data, the modified or added data is inserted into the MySQL main database. The MySQL main database automatically generates a binlog log when executing data modification or addition commands to record the specific commands of the operation.

[0048] As an example, after the data modification and data calculation are completed on the big data platform, the big data platform inserts the modified or newly added data into the MySQL main database.

[0049] S2: Set up a monitoring thread to detect changes in the binlog log of the MySQL main database in real time. When a change in the binlog log is detected, read the binlog log information from the preset offset, and parse the data change content by performing structured processing on the binlog log information.

[0050] In a specific implementation, when the MySQL main database executes a modification command, it will automatically generate a command for recording the operation in the binlog log. Therefore, a monitoring thread is set to detect the changes in the binlog log of the MySQL main database in real time. When a change in the binlog log is detected, the binlog file will be read from the last offset, and the changed part of the binlog log will be read and parsed.

[0051] During parsing, the irregular strings in the binlog log are structured according to the schema information of the database table to convert the operation commands in the binlog log into a structured data format, and then parse the data change content. Among them, the schema information of the database table includes: table name, field name, and field type.

[0052] It should be noted that binlog is a binary log file that records all DML operations of MySQL, including add, modify and delete operations, and saves them on disk. Through binlog logs, data recovery, incremental backup, master-slave replication and other operations can be performed, and data synchronization can also be achieved.

[0053] S3: Encapsulate the parsed data changes into a message entity, execute the message sending operation, and push the changed data to the message queue.

[0054] In a specific implementation, the parsed data change content is first encapsulated into a message entity, and a message sending operation is performed through the API of the message queue middleware Kafka to push the changed data to the Kafka queue. In addition, when the message sending operation fails, a retry strategy is executed.

[0055] The retry strategy specifically includes: when a message fails to be sent, the message is automatically resent according to the preset retry times and retry interval until the message is successfully sent or the maximum retry times are reached.

[0056] From the above steps, we can see that for incremental synchronization production, we need to first loop and monitor the Binlog log of the database. The original Binlog log is an irregular string, so we need to structure the log according to the schema information of the database table. After the data is assembled, the message transmission method is executed through the entity class of the message queue middleware, and the retry mechanism is used as a backup.

[0057] S4: Use the listener of the message queue to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, take out the message and parse the data change content, and push the data change content to the data storage and processing component for synchronization processing.

[0058] In a specific implementation, a listener of the Kafka queue is used to detect in real time whether there are unconsumed messages in the Kafka queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the Redis cache system, BI analysis system and MySQL slave library to remind them to synchronize data.

[0059] Among them, the Redis cache system uses an adapted Redis data structure to store data according to business requirements, and the adapted Redis data structure includes a string, a hash table, and a list.

[0060] In a specific implementation, the method may also include: real-time monitoring of the MySQL master database 、 Message Queues 、 The data stores and processes the running status of components. If any abnormality is found, an alarm message will be issued and automatic recovery will be attempted.

[0061] The present invention provides a data synchronization method for a big data platform, which realizes efficient and real-time synchronization of data by real-time monitoring of MySQL binlog logs and automatically pushing data changes to Kafka, which is then distributed by Kafka to Redis, BI and MySQL slave libraries, thereby ensuring data consistency and system scalability.

[0062] See also Figure 2 As shown, the present invention also discloses a data synchronization system for a big data platform, including: a data operation recording module 1, a log monitoring and parsing module 2, a message pushing module 3 and a synchronization module 4.

[0063] The data operation recording module 1 is used to insert the modified or added data into the MySQL main database after the modification or addition operation of the data is completed on the big data platform. The MySQL main database automatically generates a binlog log when executing the data modification or addition command to record the specific commands of the operation.

[0064] The log monitoring and parsing module 2 is used to set the monitoring thread to detect the binlog log changes of the MySQL main library in real time. When the binlog log is detected to have changed, the binlog log information is read from the preset offset, and the data change content is parsed by structurally processing the binlog log information.

[0065] The message push module 3 is used to encapsulate the parsed data change content into a message entity, execute the message sending operation, and push the changed data to the message queue.

[0066] The synchronization module 4 is used to use the listener of the message queue to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the data storage and processing component for synchronization processing.

[0067] The specific implementation of the data synchronization system of the big data platform of this embodiment is basically the same as the specific implementation of the data synchronization method of the big data platform mentioned above, and will not be repeated here.

[0068] The present invention also discloses a data synchronization device for a big data platform, comprising a processor and a memory; wherein, when the processor executes the data synchronization program of the big data platform stored in the memory, the steps of the data synchronization method for the big data platform as described in any of the above items are implemented.

[0069] Furthermore, the data synchronization device of the big data platform in this embodiment may also include:

[0070] The input interface is used to obtain the data synchronization program of the big data platform imported from the outside, and save the obtained data synchronization program of the big data platform to the memory. It can also be used to obtain various instructions and parameters transmitted by external terminal devices, and transmit them to the processor so that the processor can use the above various instructions and parameters to carry out corresponding processing. In this embodiment, the input interface can specifically include but is not limited to a USB interface, a serial interface, a voice input interface, a fingerprint input interface, a hard disk reading interface, etc.

[0071] The output interface is used to output various data generated by the processor to the terminal device connected thereto, so that other terminal devices connected to the output interface can obtain various data generated by the processor. In this embodiment, the output interface may specifically include but is not limited to a USB interface, a serial interface, etc.

[0072] The communication unit is used to establish a remote communication connection between the data synchronization device of the big data platform and the external server, so that the data synchronization device of the big data platform can mount the image file to the external server. In this embodiment, the communication unit may specifically include but is not limited to a remote communication unit based on wireless communication technology or wired communication technology.

[0073] The keyboard is used to obtain various parameter data or instructions input by the user by tapping the keycaps in real time.

[0074] The display is used to display the relevant information of the data synchronization process of the big data platform in real time.

[0075] The mouse can be used to assist users in inputting data and simplify user operations.

[0076] The present invention also discloses a readable storage medium, wherein the readable storage medium includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium known in the technical field. The readable storage medium stores a data synchronization program of a big data platform, and when the data synchronization program of the big data platform is executed by a processor, the steps of the data synchronization method of the big data platform described in any one of the above items are implemented.

[0077] In summary, the present invention realizes efficient and real-time synchronization of data by real-time monitoring of MySQL binlog logs and automatically pushing data changes to the message queue, which is then distributed to the data storage and processing components by the message queue, thereby ensuring data consistency and system scalability.

[0078] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. As for the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0079] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit.

[0083] Similarly, each processing unit in each embodiment of the present invention may be integrated into one functional module, or each processing unit may exist physically, or two or more processing units may be integrated into one functional module.

[0084] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0085] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0086] The data synchronization method, system, device and readable storage medium of the big data platform provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A data synchronization method for a big data platform, characterized in that: include: After the big data platform completes the modification or addition of data, the modified or added data is inserted into the MySQL main database. The MySQL main database automatically generates a binlog log when executing the data modification or addition command to record the specific command of the operation; Set up a monitoring thread to detect changes in the binlog logs of the MySQL master database in real time. When changes are detected in the binlog logs, read the binlog log information from the preset offset, and parse the data changes by performing structured processing on the binlog log information. Encapsulate the parsed data changes into a message entity, execute the message sending operation, and push the changed data to the message queue; The listener of the message queue is used to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the data storage and processing component for synchronization processing.

2. The data synchronization method of the big data platform according to claim 1, characterized in that: The structured processing of binlog log information includes: The irregular strings in the binlog log are structured according to the schema information of the database table to convert the operation commands in the binlog log into a structured data format; The schema information of the database table includes: table name, field name, and field type.

3. The data synchronization method of the big data platform according to claim 1, characterized in that: The step of encapsulating the parsed data change content into a message entity, executing a message sending operation, and pushing the changed data to a message queue includes: The parsed data changes are encapsulated into message entities, and the message sending operation is performed through the API of the message queue middleware Kafka to push the changed data to the Kafka queue; When a message sending operation fails, a retry strategy is implemented.

4. The data synchronization method of the big data platform according to claim 3, characterized in that: The retry strategies include: When a message fails to be sent, it will be automatically resent according to the preset retry times and retry interval until the message is successfully sent or the maximum retry times are reached.

5. The data synchronization method of the big data platform according to claim 1, characterized in that: The pushing of the data change content to the data storage and processing component for synchronization processing includes: Push data changes to the Redis cache system, BI analysis system, and MySQL slave database for synchronization.

6. The data synchronization method of the big data platform according to claim 5, characterized in that: The Redis cache system uses an adapted Redis data structure to store data according to business requirements, and the adapted Redis data structure includes a string, a hash table, and a list.

7. The data synchronization method of a big data platform according to claim 1, characterized in that: The method further comprises: Real-time monitoring of MySQL master database 、 Message Queues 、 The data stores and processes the running status of components. If any abnormality is found, an alarm message will be issued and automatic recovery will be attempted.

8. A data synchronization system for a big data platform, characterized in that: include: The data operation recording module is used to insert the modified or added data into the MySQL main database after the big data platform completes the modification or addition of data. The MySQL main database automatically generates a binlog log when executing data modification or addition commands to record the specific commands of the operation; The log monitoring and parsing module is used to set up a monitoring thread to detect the changes in the binlog logs of the MySQL main database in real time. When a change in the binlog log is detected, the binlog log information is read from the preset offset, and the data change content is parsed by structurally processing the binlog log information; The message push module is used to encapsulate the parsed data change content into a message entity, execute the message sending operation, and push the changed data to the message queue; The synchronization module is used to use the listener of the message queue to detect in real time whether there are unconsumed messages in the message queue. If an unconsumed message is detected, the message is taken out and the data change content is parsed, and the data change content is pushed to the data storage and processing component for synchronization processing.

9. A data synchronization device for a big data platform, characterized in that: include: Storage, used to store data synchronization programs of the big data platform; A processor, used to implement the steps of the data synchronization method of the big data platform as described in any one of claims 1 to 7 when executing the data synchronization program of the big data platform.

10. A readable storage medium, characterized in that: The readable storage medium stores a data synchronization program for a big data platform, and when the data synchronization program for the big data platform is executed by a processor, the steps of the data synchronization method for a big data platform as described in any one of claims 1 to 7 are implemented.

Images