Data synchronization method, device and system based on Uliweb

Abstract

The application discloses a kind of based on Uliweb's data synchronization method, device and system, it is related to distributed field, wherein the method comprises: obtaining the data change record of the binlog log file of MySQL;The change record is the record generated when MySQL database occurs data change due to the DDL operation of current microservice;Parsing the record obtains the data identification of the event change record to be synchronized;The identification is sent to different theme in message queue;Different theme is pre-subscribed by each other microservice according to business requirement respectively;When each other microservice receives the data change notification of the subscribed theme, complete change data corresponding to the identification is obtained by callback interface provided by web application, and is updated to own database.The application can realize the data quasi real-time synchronization between different microservices without invading business code, and without affecting the performance of database.

Description

Technical Field

[0001] This invention relates to the field of distributed technology, and in particular to a data synchronization method, apparatus and system based on Uliweb. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Currently, in microservice architectures developed using the Uliweb framework, the data dependencies between various services are quite severe, the data in the database tables changes frequently and in a variety of ways, making real-time data synchronization between different microservices (or business systems) quite difficult.

[0004] In a microservice system, when data changes in an application's database due to user interface operations, API calls, or other data modification operations, some of this changed data needs to be synchronized to other applications in real time for use by other business systems. Generally, there are two ways to synchronize data modifications to other business systems:

[0005] The first approach is to write code at every possible moment when data changes are triggered, and then use HTTP to call the interfaces of other applications to notify other business systems of the changed data.

[0006] The second approach does not require intruding into the business code. It uses the database's built-in triggers to record data before and after changes, and then uses asynchronous tasks to call the interfaces provided by other applications one by one to synchronize the data with other business systems.

[0007] The disadvantages of the above two data synchronization methods are:

[0008] The first approach requires developers to write code at every possible moment that could cause data changes. This is labor-intensive, prone to omissions, and necessitates extensive code modifications for each new piece of data or application that needs synchronization. It is highly intrusive to the code and makes subsequent maintenance difficult.

[0009] The second approach requires writing many triggers to monitor database changes and also requires additional database tables to record data changes. With frequent changes, the growth rate of change records will be very fast, and too many triggers will also have a certain impact on database performance. Summary of the Invention

[0010] This invention provides a data synchronization method based on Uliweb. This method is applied to a data synchronization device to achieve near real-time data synchronization between different microservices without intruding on business code or affecting database performance. The Uliweb-based data synchronization method is an asynchronous task initiated by Uliweb, and includes:

[0011] Get the current microservice's binary binlog file and its current offset;

[0012] Read the data change records of the MySQL binlog log file from the offset of the binlog log file; the change records are the records generated in the binlog log file to record database changes when the MySQL database undergoes data changes due to the DDL operation of the current microservice.

[0013] Parse the data change records in the MySQL binlog file to obtain the data identifiers of the event change records to be synchronized;

[0014] The data identifier of the event change record to be synchronized is sent to different topics in the message queue; each other microservice can pre-subscribe to the different topics according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0015] In one embodiment, parsing the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized includes: parsing the data change records of the MySQL binlog file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0016] Sending the data identifier of the event change record to be synchronized to different topics in the message queue, including: sending the data identifier and record type of the event change record to be synchronized to different topics in the message queue;

[0017] When each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application. This includes: when each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0018] In one embodiment, the above-described Uliweb-based data synchronization method may further include: updating the offset information of the binlog file for use in the next data synchronization.

[0019] In one embodiment, the above-described Uliweb-based data synchronization method may further include: modifying the business requirements of each microservice according to the form of modifying subscription configuration information.

[0020] In one embodiment, reading data change records from the offset of the MySQL binlog file includes: reading data change records from the offset of the MySQL binlog file by instantiating a stream object.

[0021] In one embodiment, the change record is specifically a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice under a preset data change scenario.

[0022] In one embodiment, the data change scenario includes: scenarios where data changes occur due to file synchronization, API interfaces, or DDL operations in web applications.

[0023] In one embodiment, the message queue is a Kafka queue.

[0024] In one embodiment, the event type is an event type of addition, deletion, or modification.

[0025] This invention provides a data synchronization method based on Uliweb. This method is applied to a data synchronization system to achieve near real-time data synchronization between different microservices without intruding on business code or affecting database performance. The method includes:

[0026] When a microservice detects that a user's DDL operation in the current microservice causes data changes, it generates a record in the binlog file of the MySQL database to record the database changes and the offset of the current file.

[0027] The Uliweb service obtains the binary binlog file of the current microservice and the offset of the current file; reads the data change records of the MySQL binlog file from the offset of the binlog file; parses the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized; and sends the data identifier of the event change record to be synchronized to different topics in the message queue.

[0028] Each other microservice pre-subscribes to different topics based on its own business needs for different data; when it receives a data change notification for a subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0029] In one embodiment, parsing the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized may include: parsing the data change records of the MySQL binlog file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0030] Sending the data identifier of the event change record to be synchronized to different topics in the message queue can include: sending the data identifier and record type of the event change record to different topics in the message queue;

[0031] When a data change notification is received from a subscribed topic, the complete change data corresponding to the data identifier of the change record is obtained through the callback interface provided by the web application. This may include: when each other microservice receives a data change notification from a subscribed topic, it obtains the data identifier of the change record and the complete change data corresponding to the record type through the callback interface provided by the web application.

[0032] In one embodiment, the above-described Uliweb-based data synchronization method may further include: updating the offset information of the binlog file for use in the next data synchronization.

[0033] In one embodiment, the above-described Uliweb-based data synchronization method may further include: modifying the business requirements of each microservice according to the form of modifying subscription configuration information.

[0034] In one embodiment, reading data change records from the offset of the MySQL binlog file may include: reading data change records from the offset of the MySQL binlog file by instantiating a stream object.

[0035] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0036] In one embodiment, the data change scenario may include: a scenario where data changes occur due to file synchronization, API interface, or web application DDL operations.

[0037] In one embodiment, the message queue can be a Kafka queue.

[0038] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0039] This invention also provides a data synchronization device based on Uliweb, which enables near real-time data synchronization between different microservices without intruding on business code or affecting database performance. The device includes:

[0040] The acquisition unit is used to obtain the binary binlog file of the current microservice and the offset of the current file.

[0041] The extraction unit is used to read data change records from the offset of the binlog log file of MySQL; the change records are records generated in the binlog log file to record database changes when the MySQL database undergoes data changes due to the DDL operation of the current microservice.

[0042] The parsing unit is used to parse the data change records in the MySQL binlog file to obtain the data identifier of the event change record to be synchronized;

[0043] The sending unit is used to send the data identifier of the event change record to be synchronized to different topics in the message queue; the different topics are pre-subscribed to by each other microservice according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0044] In one embodiment, the parsing unit can be specifically used to: parse the data change records of the MySQL binlog log file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0045] Specifically, the sending unit can be used to send the data identifier and record type of the event change record to be synchronized to different topics in the message queue;

[0046] When each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application. This may include: when each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0047] In one embodiment, the above-mentioned Uliweb-based data synchronization device may further include: an update unit for updating the offset information of the binlog log file for use in the next data synchronization.

[0048] In one embodiment, the above-mentioned Uliweb-based data synchronization device may further include: a modification unit, used to modify the business requirements of each microservice according to the form of modifying subscription configuration information.

[0049] In one embodiment, the extraction unit can be specifically used to: read data change records from the offset of the MySQL binlog log file by instantiating a stream object.

[0050] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0051] In one embodiment, the data change scenario may include: a scenario where data changes occur due to file synchronization, API interface, or web application DDL operations.

[0052] In one embodiment, the message queue can be a Kafka queue.

[0053] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0054] This invention also provides a data synchronization system based on Uliweb, which enables near real-time data synchronization between different microservices without intruding on business code or affecting database performance. The system includes:

[0055] The microservice is used to generate a record in the binlog file of the MySQL database to record the database change and the offset of the current file when it detects that the user's DDL operation in the current microservice causes data changes; it pre-subscribes to different topics according to its own business needs for different data; when it receives a data change notification from the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0056] The Uliweb service is used to obtain the binary binlog file of the current microservice and the offset of the current file; read the data change records of the MySQL binlog file from the offset of the binlog file; parse the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized; and send the data identifier of the event change record to be synchronized to different topics in the message queue.

[0057] In one embodiment, the Uliweb service can be used to: parse the data change records of the MySQL binlog log file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete or modify event; send the data identifier and record type of the event change record to be synchronized to different topics in the message queue; when receiving the data change notification of the subscribed topic, obtain the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0058] In one embodiment, the Uliweb service can also be used to update the offset information of the binlog log file for use during the next data synchronization.

[0059] In one embodiment, the microservice can also be used to modify the business requirements of each microservice according to the form of modifying subscription configuration information.

[0060] In one embodiment, the Uliweb service is specifically used to: read data change records from the offset of the MySQL binlog log file by instantiating a stream object.

[0061] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0062] In one embodiment, the data change scenario may include: a scenario where data changes occur due to file synchronization, API interface, or web application DDL operations.

[0063] In one embodiment, the message queue can be a Kafka queue.

[0064] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0065] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described data synchronization method based on Uliweb.

[0066] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described Uliweb-based data synchronization method.

[0067] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described Uliweb-based data synchronization method.

[0068] In this embodiment of the invention, an asynchronous task service is quickly deployed based on the Uliweb data synchronization scheme. During operation: the asynchronous task obtains the binary binlog file of the current microservice and its offset; it reads data change records from the MySQL binlog file at the offset; these change records are generated in the binlog file to record database changes when data changes occur in the MySQL database due to DDL operations of the current microservice; the data change records in the MySQL binlog file are parsed to obtain the data identifier of the event change record to be synchronized; the data identifier of the event change record to be synchronized is sent to different topics in the message queue; these different topics are pre-subscribed to by each other microservice according to its own business needs for different data; when each other microservice receives a data change notification from a subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0069] Compared with existing technologies where data dependencies between services are significant, database tables experience diverse and frequent changes, and real-time data synchronization between different microservices is difficult, the beneficial technical effects of the Uliweb-based data synchronization solution provided in this invention are:

[0070] First, in a large system, data synchronization is an essential business requirement. Existing technologies use HTTP interfaces for real-time synchronization, but this requires developers to write business code in all places where data changes may occur. This is labor-intensive, prone to omissions, highly intrusive to the code, and difficult to maintain and modify later. Compared with this existing technology, the embodiments of the present invention do not require developers to write business code in all places where data changes may occur, and there is no difficulty in subsequent maintenance and modification. It directly reads and parses the binary file of the database through the Uliweb data synchronization service, which is independent of microservices, without requiring modification to the original project code, and the intrusion to the code is basically zero.

[0071] Secondly, to synchronize data, existing technologies also use triggers to monitor database changes. However, triggers require the creation of additional tablespaces in the database to record database changes. The creation of a large number of triggers will seriously affect database performance and the normal operation of business systems. Moreover, the development and maintenance of triggers also require certain costs. Compared with this existing technology, the embodiments of the present invention rely on database binary files. These binary files have little impact on the database, do not require additional database space, and do not affect database performance.

[0072] In summary, the embodiments of the present invention can achieve near real-time data synchronization between different microservices without intruding on business code or affecting database performance. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0074] Figure 1 This is a flowchart illustrating the Uliweb-based data synchronization method applied to the device in an embodiment of the present invention;

[0075] Figure 2 This is a flowchart illustrating the Uliweb-based data synchronization method applied to the system in this embodiment of the invention.

[0076] Figure 3 This is a schematic diagram of the structure of the data synchronization device based on Uliweb in an embodiment of the present invention;

[0077] Figure 4This is a schematic diagram of the structure of the Uliweb-based data synchronization system in an embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram of the structure of a data synchronization system based on Uliweb in another embodiment of the present invention;

[0079] Figure 6 This is a schematic diagram of the computer device structure according to an embodiment of the present invention. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0081] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0082] Before introducing the embodiments of the present invention, the names involved in the embodiments of the present invention will be described in detail first.

[0083] 1. Uliweb: is an open-source web development framework written in Python.

[0084] 2. Binlog: The binary log of the MySQL database.

[0085] 3. python-mysql-replication: This is a MySQL replication protocol tool implemented in Python. It can be used to parse binlog and obtain insert, update, delete, and other events from the logs.

[0086] 4. Microservices: A software architecture and organization method in which software consists of small, independent services that communicate with each other through well-defined APIs.

[0087] 5. Kafka: A high-throughput distributed publish-subscribe messaging system.

[0088] 6. Topic: A message category in Kafka, used to distinguish and isolate different message data.

[0089] 7. Producer: The message producer, a client that sends messages to Kafka.

[0090] 8. Consumer: A message consumer, a client that retrieves messages from Kafka.

[0091] Considering the technical problems existing in current data synchronization methods, this invention proposes a data synchronization solution based on Uliweb. This solution is an asynchronous data synchronization task based on the Uliweb framework, which does not require intrusion into business code. By analyzing the binary log of the data, it parses out the data changes and publishes the IDs of the changed data to a Topic in a message queue (Kafka) for other applications to subscribe to. It also provides a callback interface for other business systems to pull the latest data. The following is a detailed description of this Uliweb-based data synchronization solution.

[0092] Figure 1 This is a flowchart illustrating the Uliweb-based data synchronization method applied to the device in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0093] Step 1021: Obtain the binary binlog file of the current microservice and the offset of the current file;

[0094] Step 1022: Read the data change records of the MySQL binlog log file from the offset of the binlog log file; the change records are the records generated in the binlog log file to record database changes when the MySQL database undergoes data changes due to the DDL operation of the current microservice;

[0095] Step 1023: Parse the data change records in the MySQL binlog file to obtain the data identifiers of the event change records to be synchronized;

[0096] Step 1024: Send the data identifier of the event change record to be synchronized to different topics in the message queue; each other microservice can pre-subscribe to the different topics according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0097] The data synchronization method based on Uliweb provided in this embodiment of the invention (applied to) Figure 5The method in the "Uliweb Service" data synchronization device quickly deploys an asynchronous task service. During operation: this asynchronous task obtains the current microservice's binary binlog file and its offset; it reads data change records from the MySQL binlog file at the offset; these change records are generated in the binlog file to record database changes when data changes occur in the MySQL database due to DDL operations of the current microservice; it parses the data change records in the MySQL binlog file to obtain the data identifier of the event change record to be synchronized; it sends the data identifier of the event change record to different topics in the message queue; these different topics are pre-subscribed to by each other microservice according to its own business needs for different data; when each other microservice receives a data change notification from a subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0098] Compared with existing technologies where data dependencies between services are significant, database tables experience diverse and frequent changes, and real-time data synchronization between different microservices is difficult, the beneficial technical effects of the Uliweb-based data synchronization method provided in this invention are:

[0099] First, in a large system, data synchronization is an essential business requirement. Existing technologies use HTTP interfaces for real-time synchronization, but this requires developers to write business code in all places where data changes may occur. This is labor-intensive, prone to omissions, highly intrusive to the code, and difficult to maintain and modify later. Compared with this existing technology, the embodiments of the present invention do not require developers to write business code in all places where data changes may occur, and there is no difficulty in subsequent maintenance and modification. It directly reads and parses the binary file of the database through the Uliweb data synchronization service, which is independent of microservices, without requiring modification to the original project code, and the intrusion to the code is basically zero.

[0100] Secondly, to synchronize data, existing technologies also use triggers to monitor database changes. However, triggers require the creation of additional tablespaces in the database to record database changes. The creation of a large number of triggers will seriously affect database performance and the normal operation of business systems. Moreover, the development and maintenance of triggers also require certain costs. Compared with this existing technology, the embodiments of the present invention rely on database binary files. These binary files have little impact on the database, do not require additional database space, and do not affect database performance.

[0101] In summary, the embodiments of the present invention can achieve near real-time data synchronization between different microservices without intruding on business code or affecting database performance. A detailed description follows.

[0102] This invention proposes a data synchronization scheme based on Uliweb parsing of MySQL database Binlog. For example... Figure 5 As shown, the core principle of this invention is a producer-consumer model, where the producer is an asynchronous task initiated by Uliweb (e.g., ...). Figure 5 The Uliweb service (located within the Uliweb service) primarily parses MySQL's binlog file. When MySQL data undergoes DDL (Data Definition Language) operations due to web applications or API interfaces, a record is generated in the binlog file to record the database change. The binlog is a binary file, and the Uliweb service parses it (identifying the changed table structure, whether the data changes need to be synchronized, the event type, and whether synchronization is required; for update events, it also needs to determine the updated fields based on the data changes before and after the change, thus determining whether synchronization is necessary—that is, the parsing of the log file also considers the parsed record type, as described below). This is the detailed content of step 1023.

[0103] As can be seen from the above, in one embodiment, the inventors propose the following scheme to further improve the accuracy of data synchronization by considering the type of records obtained from parsing log files:

[0104] Parsing the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized may include: parsing the data change records of the MySQL binlog file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0105] Sending the data identifier of the event change record to be synchronized to different topics in the message queue can include: sending the data identifier and record type of the event change record to different topics in the message queue;

[0106] When each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application. This may include: when each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0107] When the Uliweb service starts, it retrieves data from Redis based on the configuration information (which includes the IP addresses, ports, database names, usernames, and passwords for Redis and MySQL, typically stored in a pre-configured file). Figure 5 The system retrieves the Binlog log file and its current data offset from the database (the offset is the position information of the MySQL log file, recording where the log file is currently being parsed; it can be obtained through MySQL commands or recorded in Redis). Using the BinLogStreamReader (a class object that serves as the entry point for reading binlog logs) from the pymysqlreplication module, a stream object is instantiated. This allows the system to read the data change records at the current position from the offset in the binary file. In one embodiment, this means reading the MySQL binlog log from the offset in the binlog file. The data change records of the binlog file can include: reading the data change records of the MySQL binlog file from the offset of the binlog file by instantiating a stream object, filtering the data to be synchronized (filtering the event types in the log and sending the record types and record numbers to the message queue), publishing the data numbers to different topics in the message queue (Kafka), and updating the offset information in Redis for use in the next production message. That is, in one embodiment, the above data synchronization method based on Uliweb can also include: updating the offset information of the binlog file for use in the next data synchronization.

[0108] like Figure 5As shown, other business systems (microservice 1, microservice 2, microservice 3, microservice...) act as consumers. Different services subscribe to different topics based on their specific data needs to obtain data IDs (e.g., user data and order data both need to be synchronized to other services, so data change messages are sent to user_topic and order_topic respectively; application A needs user data and only needs to subscribe to user_topic, while application B needs both user data and order data and therefore needs to subscribe to both user_topic and order_topic). They then obtain the complete data through the callback interface provided by the web application (from...). Figure 5 The data is obtained from a web application developed using the ULIWeb framework. The web application provides API interfaces for other services to query complete data information by record type and number, and then update it to their own database.

[0109] In one embodiment, the above-described Uliweb-based data synchronization method may further include: modifying the business requirements of each microservice according to the form of modifying subscription configuration information.

[0110] In practical implementation, embodiments of the present invention can modify business requirements by changing the configuration (for example, a user table contains user ID, username, mobile phone number, address, etc. Service A initially requires username and mobile phone number, so this information can be configured as user table: [username, mobile phone number]. A change message will only be sent when the username and mobile phone number change; if only the address changes, the change record will be filtered. Later, if Service A also needs address information, the configuration information only needs to be changed to user table: [username, mobile phone number, address], and a change message will be sent when the address changes), greatly reducing the difficulty of maintenance and updates.

[0111] In one embodiment, the change record is specifically a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice under a preset data change scenario.

[0112] In one embodiment, the data change scenario includes: scenarios where data changes occur due to file synchronization, API interfaces, or DDL operations in web applications.

[0113] In specific implementation, such as Figure 5 As shown, data change scenarios can include: file synchronization, API interfaces, or web applications. Figure 5In this context, file synchronization, API interfaces, and web applications are data update scenarios, and MySQL is the corresponding data storage. Together, they form a complete microservice. Alternatively, each scenario can have its own dedicated microservice with MySQL, in which case MySQL is shared by multiple microservices. For example... Figure 5 As shown, the ULIWeb framework is a Python framework that combines the advantages of multiple frameworks, making it as simple and easy to use as possible. It can be used to develop web applications, support backend development for websites, provide API interfaces, and easily start asynchronous services via commands.

[0114] In one embodiment, the message queue can be a Kafka queue.

[0115] In specific implementations, the message queue in this embodiment of the invention can also be other message queues such as RabbitMQ and Redis.

[0116] In one embodiment, the event type is an event type of addition, deletion, or modification.

[0117] To facilitate understanding of how this invention is implemented, an example is given below.

[0118] In practical applications within the cloud desktop ecosystem, data changes in various scenarios. Some changes originate from changes in source data outside the cloud desktop; due to network isolation between the cloud desktop and the user, synchronization can only be achieved via files or messages. Other changes occur during web application operations, and still others are updated by other services through API calls. The scenarios and timing of data changes are extremely complex. To avoid coupling between data synchronization and the application layer, this method is used for data synchronization at the data layer. When data changes in a specific business scenario, the record in the MySQL table is first updated, generating a binlog record. The uliweb framework can quickly deploy an asynchronous task service using the `call` command. This service parses the MySQL binlog events, filters the events requiring synchronization, and sends the data record type and unique ID to a message queue. Other microservice systems, upon receiving the message, call back the interface provided by the uliweb application service to retrieve the latest data and update their own databases.

[0119] In summary, the data synchronization method based on Uliweb proposed in this embodiment of the invention achieves the following:

[0120] 1. This invention proposes a solution for near real-time data synchronization between different services and applications.

[0121] 2. This invention proposes a solution that does not require intrusion into business code and will not affect database performance.

[0122] 3. This invention proposes a solution that allows for flexible configuration of data and applications to be synchronized according to business needs, while maintaining relatively low maintenance costs.

[0123] In summary, the beneficial technical effects of the data synchronization method based on Uliweb proposed in this invention are:

[0124] 1. In a large system, data synchronization is an essential business requirement. While real-time synchronization via HTTP interfaces can be used, this requires developers to write business logic in every possible place where data changes might occur. This is labor-intensive, prone to omissions, highly intrusive to the code, and difficult to maintain and modify later. This solution directly reads the database's binary file through the Uliweb service, requiring no modification to the existing project code and exhibiting virtually zero intrusion.

[0125] 2. The embodiments of the present invention can modify business requirements by modifying the configuration, which greatly reduces the difficulty of maintenance and updates.

[0126] 3. Data synchronization can also be achieved using triggers to monitor database changes. However, triggers require additional tablespaces in the database to record these changes. Creating a large number of triggers can severely impact database performance and disrupt the normal operation of the business system. Furthermore, trigger development and maintenance also incur costs. This invention relies on a database binary file, which has minimal impact on the database, does not require additional database space, and does not affect database performance.

[0127] This invention also provides an application to a system (such as...) Figure 5 The data synchronization method based on Uliweb (for the entire system) is described in the following embodiments. Since the principle of the Uliweb-based data synchronization method applied to the system is similar to that applied to the device, the implementation of this method can be found in the implementation of the Uliweb-based data synchronization method applied to the device, and the repeated parts will not be described again.

[0128] Figure 2 This is a flowchart illustrating the Uliweb-based data synchronization method applied to the system in this embodiment of the invention. Figure 2 As shown, the method includes the following steps:

[0129] Step 101: When the current microservice detects that a user's DDL operation in the current microservice causes data changes, it generates a record in the binlog file of the MySQL database to record the database changes and the offset of the current file.

[0130] Step 102: The Uliweb service obtains the binary binlog file of the current microservice and the offset of the current file; reads the data change records of the MySQL binlog file from the offset of the binlog file; parses the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized; and sends the data identifier of the event change record to be synchronized to different topics in the message queue.

[0131] Step 103: Each other microservice pre-subscribes to the different topics according to its own business needs for different data; when it receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0132] In one embodiment, parsing the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized may include: parsing the data change records of the MySQL binlog file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0133] Sending the data identifier of the event change record to be synchronized to different topics in the message queue can include: sending the data identifier and record type of the event change record to different topics in the message queue;

[0134] When a data change notification is received from a subscribed topic, the complete change data corresponding to the data identifier of the change record is obtained through the callback interface provided by the web application. This may include: when each other microservice receives a data change notification from a subscribed topic, it obtains the data identifier of the change record and the complete change data corresponding to the record type through the callback interface provided by the web application.

[0135] In one embodiment, the above-described Uliweb-based data synchronization method may further include: updating the offset information of the binlog file for use in the next data synchronization.

[0136] In one embodiment, the above-described Uliweb-based data synchronization method may further include: modifying the business requirements of each microservice according to the form of modifying subscription configuration information.

[0137] In one embodiment, reading data change records from the offset of the MySQL binlog file may include: reading the data change records from the offset of the MySQL binlog file by instantiating a stream object.

[0138] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0139] In one embodiment, the data change scenario may include: a scenario where data changes occur due to file synchronization, API interface, or web application DDL operations.

[0140] In one embodiment, the message queue can be a Kafka queue.

[0141] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0142] This invention also provides a data synchronization device based on Uliweb (e.g., Figure 5 The "Uliweb service" mentioned below is an example of this device. Since the principle by which this device solves the problem is similar to the Uliweb-based data synchronization method applied to the device, the implementation of this device can refer to the implementation of the Uliweb-based data synchronization method applied to the device; repeated details will not be elaborated further.

[0143] Figure 3 This is a schematic diagram of the data synchronization device based on Uliweb in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes:

[0144] Unit 021 is used to obtain the binary binlog file of the current microservice and the offset of the current file;

[0145] Extraction unit 022 is used to read data change records from the offset of the binlog log file of MySQL; the change records are records generated in the binlog log file to record database changes when the MySQL database undergoes data changes due to the DDL operation of the current microservice;

[0146] Parsing unit 023 is used to parse the data change records of the MySQL binlog log file to obtain the data identifier of the event change record to be synchronized;

[0147] Sending unit 024 is used to send the data identifier of the event change record to be synchronized to different topics in the message queue; the different topics are pre-subscribed to by each other microservice according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0148] In one embodiment, the parsing unit can be specifically used to: parse the data change records of the MySQL binlog log file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event;

[0149] Specifically, the sending unit can be used to send the data identifier and record type of the event change record to be synchronized to different topics in the message queue;

[0150] When each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application. This may include: when each other microservice receives a data change notification for the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0151] In one embodiment, the above-mentioned Uliweb-based data synchronization device may further include: an update unit for updating the offset information of the binlog log file for use in the next data synchronization.

[0152] In one embodiment, the above-mentioned Uliweb-based data synchronization device may further include: a modification unit, used to modify the business requirements of each microservice according to the form of modifying subscription configuration information.

[0153] In one embodiment, the extraction unit can be specifically used to: read data change records from the offset of the MySQL binlog log file by instantiating a stream object.

[0154] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0155] In one embodiment, the data change scenario may include: a scenario where data changes occur due to file synchronization, API interface, or web application DDL operations.

[0156] In one embodiment, the message queue can be a Kafka queue.

[0157] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0158] This invention also provides a data synchronization system based on Uliweb (e.g., Figure 5 The entire system shown (also referred to as the framework) is as described in the following embodiments. Since the principle by which this system solves the problem is similar to the Uliweb-based data synchronization method applied to devices, the implementation of this system can be found in the implementation of the Uliweb-based data synchronization method applied to devices; details that are repeated will not be repeated here.

[0159] Figure 4 This is a schematic diagram of the structure of the data synchronization system based on Uliweb in an embodiment of the present invention, as shown below. Figure 4 As shown, the system includes:

[0160] Microservice 01 is used to generate a record in the binlog file of the MySQL database to record the database change and the offset of the current file when it detects that the user's DDL operation in the current microservice causes data changes; it pre-subscribes to different topics according to its own business needs for different data; when it receives a data change notification from the subscribed topic, it obtains the complete change data corresponding to the data identifier of the change record through the callback interface provided by the web application, and updates the complete change data to its own database.

[0161] Uliweb service 02 is used to obtain the binary binlog file of the current microservice and the offset of the current file; read the data change records of the MySQL binlog file from the offset of the binlog file; parse the data change records of the MySQL binlog file to obtain the data identifier of the event change record to be synchronized; and send the data identifier of the event change record to be synchronized to different topics in the message queue.

[0162] In one embodiment, the Uliweb service can be used to: parse the data change records of the MySQL binlog log file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete or modify event; send the data identifier and record type of the event change record to be synchronized to different topics in the message queue; when receiving the data change notification of the subscribed topic, obtain the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application.

[0163] In one embodiment, the Uliweb service can also be used to update the offset information of the binlog log file for use during the next data synchronization.

[0164] In one embodiment, the microservice can also be used to modify the business requirements of each microservice according to the form of modifying subscription configuration information.

[0165] In one embodiment, the Uliweb service is specifically used to: read data change records from the offset of the MySQL binlog log file by instantiating a stream object.

[0166] In one embodiment, the change record can specifically be a record generated in the binlog file to record database changes when the MySQL database undergoes data changes due to DDL operations performed by the current microservice in a preset data change scenario.

[0167] In one embodiment, such as Figure 5 As shown, the data change scenarios may include: scenarios where data changes occur due to file synchronization, API interfaces, or DDL operations in web applications.

[0168] In one embodiment, such as Figure 5 As shown, the message queue can be a Kafka queue.

[0169] In one embodiment, the event type can be an event type of addition, deletion, or modification.

[0170] Based on the aforementioned inventive concept, such as Figure 6 As shown, the present invention also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the aforementioned data synchronization method based on Uliweb.

[0171] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described Uliweb-based data synchronization method.

[0172] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described Uliweb-based data synchronization method.

[0173] Compared with existing technologies where data dependencies between services are significant, database tables experience diverse and frequent changes, and real-time data synchronization between different microservices is difficult, the beneficial technical effects of the Uliweb-based data synchronization solution provided in this invention are:

[0174] First, in a large system, data synchronization is an essential business requirement. Existing technologies use HTTP interfaces for real-time synchronization, but this requires developers to write business code in all places where data changes may occur. This is labor-intensive, prone to omissions, highly intrusive to the code, and difficult to maintain and modify later. Compared with this existing technology, the embodiments of the present invention do not require developers to write business code in all places where data changes may occur, and there is no difficulty in subsequent maintenance and modification. It directly reads and parses the binary file of the database through the Uliweb data synchronization service, which is independent of microservices, without requiring modification to the original project code, and the intrusion to the code is basically zero.

[0175] Secondly, to synchronize data, existing technologies also use triggers to monitor database changes. However, triggers require the creation of additional tablespaces in the database to record database changes. The creation of a large number of triggers will seriously affect database performance and the normal operation of business systems. Moreover, the development and maintenance of triggers also require certain costs. Compared with this existing technology, the embodiments of the present invention rely on database binary files. These binary files have little impact on the database, do not require additional database space, and do not affect database performance.

[0176] In summary, the embodiments of the present invention can achieve near real-time data synchronization between different microservices without intruding on business code or affecting database performance.

[0177] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for synchronizing data based on Uliweb, characterized in that, This Uliweb-based data synchronization method performs data synchronization at the data layer to avoid coupling between the data synchronization and application layers. This Uliweb-based data synchronization method is an asynchronous task service initiated by Uliweb. The Uliweb framework deploys this asynchronous task service via the `call` command. This service is independent of the microservice-based Uliweb data synchronization service and does not intrude on business logic code. This method is applied to data synchronization devices and includes: Get the current microservice's binary binlog file and its current offset; By instantiating a stream object, the data change records of the MySQL binlog log file are read from the offset of the binlog log file. The data change scenarios include: data changes corresponding to DDL operations of file synchronization, API interfaces, or web applications; the change records are the records generated in the binlog log file to record database changes when the MySQL database changes due to the DDL operation of the current microservice. Parse the data change records in the MySQL binlog file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete, or modify event. Filter the event types in the MySQL binlog file, where the event types are add, delete, or modify events; The data identifier and record type of the event change record to be synchronized are sent to different topics in the message queue; each other microservice can pre-subscribe to the different topics according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application, and updates the complete change data to its own database. Update the offset information of the MySQL binlog log file to Redis for use in the next production message; Modify the business requirements of each microservice based on the form of modifying the subscription configuration information.

2. The method of claim 1, wherein, The message queue is a Kafka queue.

3. A method for synchronizing data based on Uliweb, characterized by, This Uliweb-based data synchronization method performs data synchronization at the data layer to avoid coupling between the data synchronization and application layers. This Uliweb-based data synchronization method is an asynchronous task service started by Uliweb. The Uliweb framework deploys this asynchronous task service via the `call` command. This service is independent of the microservice's Uliweb data synchronization service and does not intrude on business logic. The method includes: When a microservice detects that a user's DDL operation in the current microservice causes data changes, it generates a record in the binlog file of the MySQL database to record the database changes and the offset of the current file. The Uliweb service obtains the binary binlog file and offset of the current microservice; it reads data change records from the MySQL binlog file at the offset by instantiating a stream object. Data change scenarios include file synchronization, API interface, or DDL operations in the web application. The service parses the MySQL binlog file's data change records to obtain the data identifier and record type of the event change records to be synchronized; the record type corresponds to add, delete, or modify events. The service filters the event types in the MySQL binlog file, ensuring the event types are add, delete, or modify events. The service updates the offset information of the MySQL binlog file in Redis for use in the next production message. It sends the data identifier and record type of the event change records to be synchronized to different topics in the message queue. Finally, it modifies the business requirements of each microservice according to the modified subscription configuration information. Each other microservice pre-subscribes to different topics based on its own business needs for different data; when it receives a data change notification for a subscribed topic, it obtains the data identifier of the change record and the complete change data corresponding to the record type through the callback interface provided by the web application, and updates the complete change data to its own database.

4. A data synchronization apparatus based on Uliweb, characterized by, This Uliweb-based data synchronization device performs data synchronization at the data layer to avoid coupling between the data synchronization and application layers. This Uliweb-based data synchronization device is an asynchronous task service initiated by Uliweb. The Uliweb framework deploys this asynchronous task service via the `call` command. This service is independent of the microservice-based Uliweb data synchronization service and does not intrude on business logic code. It includes: The acquisition unit is used to obtain the binary binlog file of the current microservice and the offset of the current file. The extraction unit is used to read data change records from the offset of the binlog log file of MySQL by instantiating a stream object. The data change scenarios include: data changes corresponding to DDL operations of file synchronization, API interfaces, or web applications; the change record is a record generated in the binlog log file to record database changes when the MySQL database changes due to the DDL operation of the current microservice. The parsing unit is used to parse the data change records of the MySQL binlog log file to obtain the data identifier and record type of the event change record to be synchronized; the record type is the record type corresponding to the add, delete or modify event; the event types in the MySQL binlog log file are filtered, and the event types are the add, delete or modify event types; The sending unit is used to send the data identifier and record type of the event change record to be synchronized to different topics in the message queue; the different topics are pre-subscribed to by each other microservice according to its own business needs for different data; when each other microservice receives the data change notification of the subscribed topic, it obtains the complete change data corresponding to the data identifier and record type of the change record through the callback interface provided by the web application, and updates the complete change data to its own database; The update unit is used to update the offset information of the MySQL binlog log file to Redis for use in the next production message; The modification unit is used to modify the business requirements of each microservice according to the form of modifying subscription configuration information.

5. A data synchronization system based on Uliweb, characterized in that, This Uliweb-based data synchronization system performs data synchronization at the data layer to avoid coupling between the data synchronization and application layers. This Uliweb-based data synchronization system is an asynchronous task service launched by Uliweb. The Uliweb framework deploys this asynchronous task service via the `call` command. This service is independent of the microservice-based Uliweb data synchronization service and does not intrude on business logic code. It includes: The microservice is used to generate a record in the binlog file of the MySQL database to record the database change and the offset of the current file when it detects that the user's DDL operation in the current microservice causes data changes; it pre-subscribes to different topics according to its own business needs for different data; when it receives a data change notification from the subscribed topic, it obtains the data identifier of the change record and the complete change data corresponding to the record type through the callback interface provided by the web application, and updates the complete change data to its own database. The Uliweb service is used to obtain the binary binlog file of the current microservice and its offset. It reads data change records from the MySQL binlog file at the offset by instantiating a stream object. Data change scenarios include file synchronization, API interface, or DDL operations in the web application. The service parses the MySQL binlog file's data change records to obtain the data identifier and record type of the event change records to be synchronized. The record type corresponds to add, delete, or modify events. The service filters the event types in the MySQL binlog file, ensuring they are add, delete, or modify events. It updates the offset information of the MySQL binlog file to Redis for use in the next production message. It sends the data identifier and record type of the event change records to be synchronized to different topics in the message queue. Finally, it modifies the business requirements of each microservice according to the modified subscription configuration information.

6. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 3.

8. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 3.

Images