Database synchronization system, method, device, electronic device and medium

Abstract

The present disclosure provides a database synchronization system, method, device, electronic device and medium, which relate to the field of computer technology, and in particular to the field of big data technology. The specific implementation scheme is: a producer microservice is used to obtain the operation event executed when the source system updates the data in the source database, and store the operation event in a message queue, wherein the operation event includes a business identifier. A consumer microservice is used to obtain the operation event from the message queue, determine the consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database. It is achieved that the execution order of the operation event is guaranteed on the basis of improving the real-time performance of database synchronization.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of big data technology. Background Art

[0002] With the development of big data technology, there is a need for data synchronization between databases in many scenarios. For example, when a user activates a photo album cloud storage service, every time the user terminal adds, deletes, or modifies images in the local photo album, the cloud storage system needs to synchronously modify the photo album of the user terminal stored in the cloud database. For another example, in order to eliminate data silos and improve data utilization, different data producers can share the data they produce. At this time, every time the data in the database of a data producer is updated, other data producers can synchronously update the data stored in their own databases. Summary of the invention

[0003] The present disclosure provides a database synchronization system, method, device, electronic device and medium.

[0004] A first aspect of an embodiment of the present disclosure provides a database synchronization system, including: a producer microservice, a message queue, and a consumer microservice;

[0005] The producer microservice is used to obtain an operation event performed when the source system updates the data in the source database, and store the operation event in the message queue, wherein the operation event includes a business identifier;

[0006] The consumer microservice is used to obtain an operation event from the message queue, determine a consumption thread corresponding to a business identifier of the operation event, and use the consumption thread to execute the operation event on a target database.

[0007] A second aspect of the embodiments of the present disclosure provides a database synchronization method, which is applied to a producer microservice, including:

[0008] Acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier;

[0009] The operation event is stored in the message queue, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0010] A third aspect of the embodiments of the present disclosure provides a database synchronization method, which is applied to a consumer microservice, including:

[0011] Obtaining an operation event stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier;

[0012] Determine the consumer thread corresponding to the business identifier of the operation event, and use the consumer thread to execute the operation event on the target database. A database synchronization method, applied to a message queue, includes:

[0013] Receive an operation event stored by a producer microservice, where the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier;

[0014] The operation event is sent to a consumer microservice, so that the consumer microservice determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0015] A fourth aspect of the embodiments of the present disclosure provides a database synchronization method, which is applied to a message queue, and includes:

[0016] Receive an operation event stored by a producer microservice, where the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier;

[0017] The operation event is sent to a consumer microservice, so that the consumer microservice determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0018] A fifth aspect of the embodiments of the present disclosure provides a database synchronization device, which is applied to a producer microservice, including:

[0019] An acquisition module, used to acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier;

[0020] A storage module is used to store the operation event in the message queue so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0021] A sixth aspect of the embodiments of the present disclosure provides a database synchronization device, which is applied to a consumer microservice, including:

[0022] An acquisition module, used to acquire an operation event stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier;

[0023] The consumption module is used to determine the consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database.

[0024] A seventh aspect of the embodiments of the present disclosure provides a database synchronization device, which is applied to a message queue, including:

[0025] A receiving module, used to receive an operation event stored by a producer microservice, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier;

[0026] The sending module is used to send the operation event to the consumer microservice, so that the consumer microservice determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0027] According to an eighth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0028] at least one processor; and

[0029] a memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods of the second aspect to the fourth aspect.

[0031] According to a ninth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute a method according to any one of the second to fourth aspects.

[0032] According to a tenth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method according to any one of the second to fourth aspects.

[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0035] Figure 1is a structural diagram of a database synchronization system provided by an embodiment of the present disclosure;

[0036] Figure 2 is an exemplary schematic diagram of a first database synchronization process provided by an embodiment of the present disclosure;

[0037] Figure 3 is an exemplary schematic diagram of a second database synchronization process provided by an embodiment of the present disclosure;

[0038] Figure 4 is an exemplary schematic diagram of a third database synchronization process provided by an embodiment of the present disclosure;

[0039] Figure 5 is a flow chart of a first database synchronization method provided by an embodiment of the present disclosure;

[0040] Figure 6 is a flow chart of a second database synchronization method provided by an embodiment of the present disclosure;

[0041] Figure 7 is a flowchart of a third database synchronization method provided by an embodiment of the present disclosure;

[0042] Figure 8 is a structural diagram of a first database synchronization device provided by an embodiment of the present disclosure;

[0043] Fig. 9 is a structural diagram of a second database synchronization device provided by an embodiment of the present disclosure;

[0044] Fig.10 is a structural diagram of a third database synchronization device provided by an embodiment of the present disclosure;

[0045] Fig.11 It is a block diagram of an electronic device used to implement the database synchronization method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0047] At present, when synchronizing the data of two system databases, log synchronization or real-time interface call synchronization is generally adopted. For the convenience of description, the following takes the case where the data in database 1 of system 1 has changed and the data in database 2 of system 2 needs to be synchronously updated as an example for explanation.

[0048] The log synchronization method is as follows: System 2 obtains the operation log from System 1, where the operation log records various operations performed by System 1 on Database 1 over a period of time. System 2 then parses the operation log to obtain the operation events for updating the data in the database included in the operation log, and then executes each operation event on Database 2. However, since some systems do not record the operation events of the database in real time in the operation log, that is, the update of the database will not be immediately reflected in the operation log, and the acquisition and parsing of the operation log requires a large amount of computing resources and transmission resources, the log synchronization method is generally executed once at a relatively long interval, making this method lack of real-time performance.

[0049] The real-time interface call synchronization method is: whenever system 1 executes an operation event on database 1 to update the data in database 1, the corresponding interface of system 2 is called to make system 2 execute the same operation event on database 2. However, in a high-concurrency scenario, in order to improve the efficiency of data synchronization, system 2 uses multiple threads to execute operation events on database 2 in parallel. At this time, it is difficult to ensure that the order of each operation event executed by system 2 on database 2 is the same as the order of each operation event executed by system 1 on database 1. However, the order of operation events executed on the database is very important and directly affects the consistency of the data stored in the two databases. For example, operation event 1 is to add data A to the database, and operation event 2 is to modify data A to data B in the database. If operation event 1 is executed first and then operation event 2, the data stored in the database after the execution is completed is data B. If operation event 2 is executed first and then operation event 1, operation event 2 may not be executed, so that the data stored in the database after the execution of operation event 1 is data A.

[0050] In order to improve the real-time performance of the database synchronization process while ensuring the execution order of operation events, the embodiments of the present disclosure provide a database synchronization system.

[0051] Before introducing the database synchronization system provided by the embodiment of the present disclosure, the terms involved in the embodiment of the present disclosure are first explained:

[0052] Service bus: A message queue product used to transfer data between applications and services as messages. Messages contain data and metadata, where metadata is used to describe the properties of the data, such as the size and type of the data. The data in the message can be of various types, including structured data encoded in a common format, such as JavaScript Object Notation (JSON), Extensible Markup Language (XML), Apache Avro, or plain text.

[0053] Message session function: Using the session function, a series of related messages can be processed exclusively and orderly, that is, a series of related messages can be treated as messages in the same session, and the messages in the same session can be executed serially in order to ensure the first-in-first-out principle in the message processing process. In order to be applied in a scalable and high-availability system, the session function also supports the storage of session status.

[0054] The session ID of a message is used to mark the session to which the message belongs. Multiple messages can have the same session ID, that is, the same session can include multiple messages. Session ID is often used in web browsing scenarios. In this scenario, from the time the user terminal opens the web page to the time the web page is closed, each operation performed by the user on the web page corresponds to a message sent by the user terminal to the server, and these messages include the same session ID. In this case, the session ID is used to indicate that the message originates from the same access process to the web page. The embodiment of the present disclosure uses the session ID to distinguish different businesses to which the operation events on the database belong. The specific method can be referred to the description below.

[0055] Dead letter queue: A queue used to store messages that cannot be processed normally temporarily. Messages in the message queue that cannot be processed normally temporarily can be transferred to the dead letter queue. When the messages in the dead letter queue can be processed normally, they can be transferred to the message queue.

[0056] Duplicate detection: The message queue can check whether there are duplicate messages in the messages stored in itself, and delete the duplicate messages. The message sender may encounter an error after storing the message in the message queue. In this case, after the message sender restarts, it may mistakenly believe that the previous message was not successfully stored in the message queue, so it stores the message in the message queue again, resulting in duplicate messages in the message queue. Alternatively, the message sender obtains two identical messages due to network reasons, and based on this, sends two identical messages to the message queue. Alternatively, the storage of duplicate messages in the message queue may also be caused by other reasons, which are not specifically limited in the embodiments of the present disclosure.

[0057] Case ID: used to indicate the business to which the operation event belongs.

[0058] Event ID is the unique identifier of an operation event, that is, different operation events have different event IDs. Case ID and event ID can have a one-to-many relationship, that is, different operation events generated by the same business have different event IDs, but the case ID is the same.

[0059] Source system: used to manage source databases, and can perform operations such as adding data, deleting data, modifying data, and querying data on source databases. In the disclosed embodiment, the database originally targeted by the operation event is called the source database, and the database that needs to be consistent with the data in the source database is called the target database.

[0060] The event producer, i.e., the source system, can generate operation events on the source database, where each operation event has a sequence of occurrence.

[0061] Event consumer: that is, the database synchronization system, which can obtain operation events from the source system and execute the operation events on the target database. When the event consumer executes the operation event on the target database, it needs to follow the order in which the event producer produces the operation event.

[0062] like Figure 1 As shown, the database synchronization system provided by the embodiment of the present disclosure includes: a producer microservice 101, a message queue 102 and a consumer microservice 103;

[0063] The producer microservice 101 is used to obtain the operation event executed when the source system updates the data in the source database, and store the operation event in the message queue 102. The operation event includes a business identifier.

[0064] The consumer microservice 103 is used to obtain the operation event from the message queue 102, determine the consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database. The consumer microservice 103 has multiple consumption threads, each of which can execute the operation event.

[0065] In the embodiment of the present disclosure, the operation event performed when the source system updates the data in the source database may include at least one of the following events: an addition event for adding data in the database, a deletion event for deleting data in the database, and a modification event for modifying data in the database.

[0066] Among them, the source system is a software system that can be run on the first electronic device. The database synchronization system is a software system that can be run on the second electronic device, and the second electronic device where the producer microservice, message queue and consumer microservice included in the database synchronization system are located can be physically the same electronic device, or the second electronic devices where the producer microservice, message queue and consumer microservice are located can be physically different electronic devices. The first electronic device can be an electronic device with database operation capabilities such as a server, a desktop computer or a laptop computer; similarly, the second electronic device can be an electronic device with database operation capabilities such as a server, a desktop computer or a laptop computer.

[0067] The database synchronization system of the disclosed embodiment can execute the same operation event on the target database according to the operation event executed on the source database, thereby realizing data synchronization between the source database and the target database. Moreover, the data synchronization process of the disclosed embodiment does not rely on the log of the source system, thus avoiding the problem of synchronization delay caused by the log synchronization method and improving the real-time performance of database synchronization. In addition, the consumer microservice in the disclosed embodiment can use the consumer thread corresponding to the business identifier of the operation event to execute the operation event, so that the operation event of the same business can be executed serially by the same consumer thread, thereby ensuring the execution order of the operation event, that is, ensuring the data consistency in the source database and the target database.

[0068] The following is a detailed description of the process of implementing data synchronization by the database synchronization system in the embodiment of the present disclosure:

[0069] In some embodiments of the present disclosure, the producer microservice obtains operation events in the following two ways.

[0070] In the first implementation of obtaining operation events, the producer microservice is specifically used to perform the following steps:

[0071] Step 1: Periodically call the event list application programming interface (Application Programming Interface, API) of the source system to obtain the event ID (event id) of each operation event executed when the source system updates the data in the source database in the current period.

[0072] The event list API is a software interface provided by the source system for obtaining event identifiers of operation events performed by the source system. The event list API can be pre-configured in the producer microservice.

[0073] The period of the producer microservice calling the event list can be pre-set according to the frequency of the source system executing operation events, the event processing capability of the source system and / or the event processing capability of the database synchronization system. For example, the period is 1 minute, that is, the producer microservice calls the event list API once every 1 minute to obtain the event identifiers of each operation event executed by the source system in the previous minute.

[0074] Optionally, after calling the event list API, the producer microservice can obtain not only the event identifier of the operation event, but also other information such as the business identifier of the operation event. This embodiment of the present application does not specifically limit this.

[0075] Step 2: Send an operation event acquisition request to the source system, wherein the operation event acquisition request includes each event identifier.

[0076] Step 3: Receive the operation events corresponding to the event identifiers sent by the source system.

[0077] The operation event may include: a service identifier, an event type (event type) and an event identifier. The operation event may also include other parameters, which are not specifically limited in the embodiments of the present disclosure. The event type includes: adding data, deleting data and modifying data.

[0078] For example, the following is the data structure of an operation event in XML format:

[0079] 1 <item>

[0080] 2<event type>123< / event type>

[0081] 3<case id>321< / case id>

[0082] 4<occurred at>2024-01-01 01:01:01< / occurred at>

[0083] 5 <additional> aaa< / additional>

[0084] 6<event id>456789< / event id>

[0085] 7< / item>

[0086] In this data structure, <item> "and"< / item> " represents an operation event,"<event_type> "and"< / event_type> "123" between " indicates the event type of the operation event;<case id> "321" between "" indicates the business identifier of the operation event;<occurred at> "and"< / occurred at> "2024-01-01 01:01:01" between " indicates the execution time of the operation event; "<additional> "and"< / additional> "aaa" between "aaa" and "aaa" indicates the additional parameters of the operation event. The additional parameters can be set according to business requirements. For example, the additional parameter is the approval manual number of the approval business;<event id> "and"< / event id> "456789" between " indicates the event ID of the operation event.

[0087] In actual application scenarios, the parameters included in each operation event are not limited to the above examples, and the parameters included in each operation event do not need to cover all the parameters included in the above examples.

[0088] In the disclosed embodiment, the source system and the database synchronization system can operate independently, that is, the source system does not need to consider the database synchronization system when executing an operation event. For example, the source system does not need to control the database synchronization system in real time to execute the operation event, nor does it need to execute the next operation event after the database synchronization system executes successfully. Moreover, the database synchronization system can actively obtain and execute operation events based on the event list API without being controlled by the source system. Therefore, in the disclosed embodiment, the source system only needs to provide the event list API to achieve decoupling between the source system and the database synchronization system, so that the two systems can operate independently, reduce the impact of errors in one system on other systems, and improve the stability and reliability of the operation of the source system and the database synchronization system.

[0089] That is, Figure 2 As shown, the producer of the event is the source system, and the consumer of the event is the database synchronization system. The producer microservice of the database synchronization system periodically calls the event list API of the source system to obtain the event identifier of each operation event in the operation event list executed when the source system updates the data in the source database in the current cycle. Then the producer microservice obtains the operation event list formed by each operation event based on the event identifier, and stores each operation event in the operation event list into the message queue. So that the consumer microservice obtains each operation event from the message queue in sequence, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0090] After calling the event list API, the producer microservice can obtain the operation events performed by the source system on the source database within a period of time, that is, multiple operation events. In order to ensure the execution order of these multiple operation events, the producer microservice can store these multiple operation events in the message queue in sequence.

[0091] That is, the producer microservice is specifically used to store the received operation events into the message queue in the order of execution time from earliest to latest.

[0092] The operation event includes the execution time of the operation event by the source system. For example, the execution time indicates: the start time of the operation event by the source system, or the completion time of the operation event by the source system.

[0093] Since the producer microservice can store each operation event in the message queue according to the execution time sequence of each operation event received, and the message queue has a first-in-first-out feature, each operation event can be executed in sequence by the consumer microservice to update the data in the target database.

[0094] In the second implementation of obtaining operation events, the producer microservice is specifically used to perform the following steps:

[0095] Step 1: Receive an event identifier of an operation event sent by a source system.

[0096] The event identifier is sent after the source system executes an operation event to update the data in the source database. That is, every time the source system executes an operation event for updating the data in the source database, it sends the event identifier of the operation event to the producer microservice.

[0097] Step 2: Send an operation event acquisition request to the source system, wherein the operation event acquisition request includes an event identifier.

[0098] Step 3: Receive the operation event corresponding to the event identifier sent by the source system.

[0099] Through the above method, the source system can enable the producer microservice to obtain an operation event each time an operation event is executed on the source database, thereby synchronously executing the operation event on the target database, thereby improving the real-time performance of data synchronization between the source database and the target database.

[0100] In order to enable the operation events of the same business to be executed serially in sequence, the producer microservice can store the operation events of the same business in the same session of the message queue, that is, the producer microservice is specifically used to perform the following steps:

[0101] Step (1): using a preset hash algorithm, performing a hash operation on the service identifier included in the operation event to obtain a hash value.

[0102] The business identifier is hashed using a preset hash algorithm, and the resulting hash value is an integer within a preset range, and the number of integers included in the preset range is the same as the number of sessions in the message queue. For example, the preset hash algorithm is a consistent hashing algorithm. The number of sessions in the message queue can be preset according to actual business needs. For example, since the operation events in different sessions of the message queue can be processed concurrently by the consumer microservice, that is, the number of sessions can be regarded as the number of operation events that can be processed concurrently, the number of sessions in the message queue can be set according to the real-time requirements of the business for data synchronization.

[0103] In one implementation, each business can be set with a unique business ID (case id). For example, the business ID included in the operation event generated by the financial business is "01", and the business ID included in the operation event generated by the education business is "02". In this case, the producer microservice can directly perform a hash operation on the business ID included in the operation event to obtain a hash value.

[0104] In another implementation, there is a situation where the business identifier of the operation event is composed of multiple identifiers. For example, the operation event generated by the approval business of a financial transaction includes a financial transaction identifier and an approval business identifier, and the financial transaction identifier and the approval business identifier together constitute the business identifier of the operation event. In this case, before executing step (1), the producer microservice is also used to: obtain other identifiers except the event identifier from the various identifiers included in the operation event, and if there are multiple other identifiers, splice the other identifiers as the business identifier.

[0105] For each operation event, the producer microservice can filter the data including "id" in the attribute from the data included in the operation event, and then exclude the data with the attribute "event id", and splice the remaining data as the business identifier of the operation event. For example, the operation event includes: {"service id": "11", "case id": "1", "event id": "01", "execution time": "00:00:01"}, and the operation event includes 4 data items, each separated by ",", and the parameter before ":" in each data item represents the attribute of the data, and the parameter after ":" represents a data of the operation event. When determining the business identifier, the producer microservice can obtain the data including "id" in the attribute, that is, obtain 11, 1 and 01, and then exclude 01 with the attribute "event id", and then splice 11 and 1 to obtain 111 as the business identifier of the operation event.

[0106] Through the above method, the embodiment of the present disclosure can support the business identification of the operation event represented by only one data, and also support the business identification of the operation event represented by a combination of multiple data. Therefore, the application scope is wider and database synchronization can be achieved in various application scenarios.

[0107] Step (ii): using the hash value as the session ID of the operation event, wherein the session ID may also be referred to as the group ID.

[0108] Step (iii) storing the operation event in the session corresponding to the session identifier of the message queue.

[0109] The producer microservice can set a session identifier for each operation event, so that after the operation event is stored in the cache of the message queue, the message queue can route the operation event to the session corresponding to the session identifier based on the session identifier of the operation event.

[0110] Due to the session mechanism in the message queue, it can be ensured that a series of operation events can be executed exclusively and orderly. Therefore, in the embodiment of the present disclosure, the producer microservice can set the session identifier of the operation event according to the business of the operation event, thereby instructing the message queue to store the operation events of the same business in the same session according to the session identifier, thereby ensuring that the operation events of the same business can be executed in an orderly manner.

[0111] In the disclosed embodiment, there is a situation where different service identifiers obtain the same hash value after hash operation. That is, there is a situation where the same session includes operation events of multiple services. For example, there are 4 sessions created in the message queue, namely session1, session 2, session 3 and session4, and the session identifier of session1 is 1, the session identifier of session2 is 2, the session identifier of session3 is 3, and the session identifier of session4 is 4. Assume that the preset hash operation is: calculate the remainder of the service identifier divided by 4; when the remainder is 0, use 4 as the hash value; when the remainder is not 0, use the remainder as the hash value. At this time, when the service identifier of the operation event is 1, the hash value is 1, that is, the session identifier is 1, and session1 needs to be added. When the service identifier of the operation event is 2, the hash value is 2, that is, the session identifier is 2, and session2 needs to be added. By analogy, when the service identifier of the operation event is 5, the hash value is 1, that is, the session identifier is 1, and session1 needs to be added. It can be seen from this that session1 may include an operation event with a service identifier of 1 and an operation event with a service identifier of 5.

[0112] Although the same session may include operation events of multiple services, all operation events of these multiple services are executed serially, so the execution order of the operation events of a single service can also be guaranteed in this case.

[0113] In the disclosed embodiment, the consumer microservice can also periodically poll whether it is disconnected from the subscribed session. For example, the consumer microservice can check whether it has received session exception information sent by the message queue, where the session exception information is used to indicate that the consumer microservice is disconnected from the subscribed session. If so, it requests the message queue to create a connection for the session again. If the message queue detects that the session has not been subscribed by other consumer microservices, it establishes a connection with the consumer microservice for the session; if the message queue detects that the session has been subscribed by other consumer microservices, it sends a connection rejection message to the consumer microservice.

[0114] In some embodiments of the present disclosure, there are multiple consumer microservices in the database synchronization system, and each consumer microservice processes the same operation event. Each consumer microservice can also be called a consumer instance of the operation event.

[0115] For example, Figure 3 As shown, the database synchronization system includes N consumer microservices. At this time, the operation events provided by the source system can be executed by N consumer microservices, and each consumer microservice can determine the business processing to be performed on the target database through the event type included in the operation event. For example, the event type of event 1 is 1, indicating that business processing 1 needs to be performed, that is, deleting the data in the database; the event type of event 2 is 2, indicating that business processing 2 needs to be performed, that is, adding data in the database; the event type of event 3 is 3, indicating that business processing 3 needs to be performed, that is, modifying the data in the database.

[0116] The consumer microservice is specifically used to obtain the operation event when it monitors the existence of an operation event in the session it subscribes to in the message queue, determine the consumer thread used to execute the subscribed session, and use the consumer thread to execute the operation event on the target database. Each consumer microservice subscribes to a different session.

[0117] Each consumer microservice in the embodiment of the present disclosure can be configured with a preset number of consumer threads, where the preset number is the number of sessions included in the message queue, and each consumer thread corresponds to a session in the message queue. Since each consumer microservice can subscribe to at least one session, and the sessions subscribed by the consumer microservice may vary, the consumer microservice can create multiple consumer threads and set each consumer thread to correspond to a session in the message queue.

[0118] Each consumer microservice can receive a manually assigned session identifier of at least one session, and create a session connection with the message queue based on the session identifier to subscribe to the session. After the session connection is successfully established, the consumer microservice can monitor and obtain operation events from the session.

[0119] Alternatively, the message queue can allocate at least one session in the message queue to each consumer microservice based on the load balancing principle, and establish a session connection with the consumer microservice based on the session identifier of the session allocated to the consumer microservice, so that the consumer microservice can monitor and obtain operation events from the session.

[0120] Alternatively, the session subscribed to by each consumer microservice may be allocated in other ways, which is not specifically limited in the embodiments of the present disclosure.

[0121] In the disclosed embodiment, since each consumer microservice subscribes to a different session and a single consumer thread executes only one operation event at a time, the operation events of the same business can be executed by the same thread of the same consumer microservice, thereby ensuring the execution sequence of the operation events, that is, ensuring that the order of the operation events executed by the source system on the source database is consistent with the order of the operation events executed by the database synchronization system on the target database, thereby improving the accuracy of data synchronization between databases.

[0122] Moreover, since each consumer microservice subscribes to a different session, and each consumer microservice can process operation events concurrently, that is, the embodiment of the present disclosure can process operation events of different businesses in parallel, and the execution order of operation events of different businesses does not affect the accuracy of data stored in the target database. Compared with the method of serially executing each operation event in the message queue, the embodiment of the present disclosure can process operation events of different businesses concurrently, improve the processing efficiency of operation events, make the database synchronization system have high availability, and can be applied to high concurrency scenarios.

[0123] In the disclosed embodiment, in order to ensure the idempotence of the data in the target database, the consumer microservice can also obtain the business identifier of the operation event before processing the operation event, and check whether the data corresponding to the business identifier is stored in the target database. If so, the operation event is executed on the data corresponding to the business identifier; if not, the business identifier is created in the target database, and the operation event is executed on the data corresponding to the business identifier.

[0124] In the embodiment of the present disclosure, the message queue may be a service bus.

[0125] When the message queue is a service bus, the message queue can also perform duplicate detection on the operation events stored in itself, that is, check whether there are duplicate operation events stored, and delete the duplicate operation events to prevent the operation events from being consumed repeatedly, resulting in the data stored in the target database being inconsistent with the source database.

[0126] The disclosed embodiment uses service bus as a message queue. Since service bus is a first-in-first-out queue, it can ensure that the operation events stored in the message queue can be first-in-first-out, that is, the order in which the operation events are stored in the message queue is the same as the order in which the operation events are removed from the message queue and processed. Moreover, service bus can also ensure that each stored operation event is processed, thereby ensuring the consistency of the data stored in the target database and the source database.

[0127] In order to ensure the execution order of operation events, the message queue is used to perform the following steps: when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time, the target session is set to a locked state, which can also be called locking the session, and the lock is occupied by the consumer microservice that subscribes to the session.

[0128] The target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation events in the target session are only obtained by the consumer microservice that subscribes to the target session.

[0129] Furthermore, the message queue is also used to: after the target consumer microservice obtains any operation event in the target session, receive an execution completion message sent by the target consumer microservice, and delete the operation event from the target session.

[0130] Although the message queue can store operation events on a first-in-first-out basis, when there are multiple consumer microservices, since each consumer microservice can process operation events in parallel, operation events of the same business may be executed in parallel, and even the processing order may be wrong. In the embodiment of the present disclosure, the message queue can lock the session so that the operation events in a session are only processed by one consumer microservice at the same time, avoiding the situation where consumer microservices concurrently process operation events of the same business, and avoiding the accuracy of the data stored in the target database due to the wrong execution order of operation events during data synchronization. Moreover, the message queue can be deleted from the session only after the operation event is executed, avoiding the execution omission of the operation event.

[0131] Since there is a situation where the consumer microservice cannot successfully execute the operation event, in order to ensure that the operation event is not lost, the message queue can also be used: if the execution completion message sent by the target consumer microservice is not received within the timeout period after the target consumer microservice obtains an operation event in the target session, the target session is set to an unlocked state, which can also be called unlocking the target session. After that, from other consumer microservices except the target consumer microservice, a consumer microservice is selected to subscribe to the target session, so that the selected consumer microservice obtains the operation event from the target session. When it is detected that the selected consumer microservice obtains the operation event from the target session for the first time, the target session is set to a locked state.

[0132] If the message queue does not receive the execution completion message sent by the target consumer microservice within the timeout period after the target consumer microservice obtains the operation event, it means that the target consumer microservice cannot currently execute the operation event, or it is difficult to ensure the execution efficiency of the operation event. Therefore, the message queue can reselect a consumer microservice from other consumer microservices, for example, according to the load balancing principle, select a consumer microservice with the least number of subscribed sessions. After that, the message queue creates a session connection with the consumer microservice to subscribe to the target session, so that the consumer microservice obtains the operation event from the target session, and the message queue sets the target session to a locked state.

[0133] Through the above method, the message queue can assign the operation events of the target session to another consumer microservice for processing when the target consumer microservice cannot process the operation events of the target session in time, for example, when the target consumer microservice is manually deleted in a scaling down scenario, or when the target consumer microservice goes down, thereby realizing the movement of sessions between consumer microservices and ensuring the execution efficiency of operation events within the session.

[0134] The disclosed embodiment can support the expansion of the database synchronization system, that is, increase the number of consumer microservices included in the database synchronization system. At this time, the message queue is also used to: when a new consumer microservice is detected in the database synchronization system, allocate the session subscribed by each consumer microservice according to the preset load balancing strategy.

[0135] The load balancing strategy may include: each consumer microservice is assigned at least one session, and the difference between the maximum and minimum number of sessions assigned to each consumer microservice does not exceed a preset threshold. The load balancing strategy may also include other strategies, which are not specifically limited in the embodiments of the present disclosure.

[0136] In the disclosed embodiment, a new consumer microservice can be manually added to the database synchronization system, and the message queue can receive the manually input relevant information of the new consumer microservice, such as the service identifier. At this time, the message queue can determine that the new consumer microservice exists in the database synchronization system and execute the session allocation process.

[0137] Through the above method, the embodiment of the present disclosure can support real-time expansion of the database synchronization system, thereby meeting the execution requirements of large-scale operation events in large-scale business scenarios and flexibly responding to changes in business requirements.

[0138] In some embodiments of the present disclosure, there may be a situation where the operation event cannot be processed temporarily, for example, the target database is in a read-only state and data in the target database cannot be deleted. For another example, the target database storage space is fully occupied and data cannot be added to the target database.

[0139] To ensure the processing order of operation events, the message queue is also used to: if it is detected that any operation event in the message queue meets the preset dead letter condition, determine the dead letter session to which the operation event that meets the dead letter condition belongs, and then store each operation event in the dead letter session in the dead letter queue in order.

[0140] Optionally, the preset dead letter condition includes: the number of retries of the operation event exceeds the preset number, and / or the storage time of the operation event in the message queue exceeds the preset storage time. Of course, the dead letter condition may also include other conditions, which are not specifically limited in the embodiments of the present disclosure.

[0141] For example, after the message queue sends an operation event of a session to a consumer microservice, if it receives a processing failure message sent by the consumer microservice, the number of retries for the operation event is increased by 1, and the operation event is sent to the consumer microservice again until the number of retries exceeds the preset number.

[0142] Through the above method, the message queue can transfer all messages in the session to which the operation event in the message queue belongs to the dead letter queue after any operation event in the message queue meets the dead letter condition, instead of only transferring the operation event that meets the dead letter condition to the dead letter queue, thereby avoiding the omission of some operation events in the same session and causing the execution order of each event operation to be wrong, and ensuring the execution order of the operation events in the same session.

[0143] Subsequently, when the error cause of the operation event in the dead letter queue can be manually checked and the fault can be manually eliminated so that the operation event in the dead letter queue can be processed, the message queue transfers the operation event in the dead letter queue to the message queue.

[0144] See also Figure 4The following describes the database synchronization process provided by the embodiment of the present disclosure in combination with actual application scenarios:

[0145] A timed task is set in the database synchronization system, which is used to regularly notify the producer microservice to call the event list API of the source system, so that the producer microservice periodically calls the event list API of the source system, obtains the event identifier of each operation event executed when the source system updates the data in the source database in the current cycle, and obtains the operation event from the source system based on the event identifier. After that, the operation event obtained from the source system is stored in the message queue. The consumer microservice can obtain the operation event from the message queue, determine the consumer thread corresponding to the business identifier of the operation event, and use the consumer thread to execute the operation event on the target database. The message queue can also determine the dead letter session to which the operation event that meets the dead letter condition belongs when it detects that any operation event in the message queue meets the preset dead letter condition, and store each operation event in the dead letter session in the dead letter queue in order. Among them, when the operation event in the dead letter queue can be processed, the message queue can transfer the operation event in the dead letter queue back to the message queue.

[0146] The database synchronization system provided by the embodiments of the present disclosure can meet the execution requirements of large-scale operation events in large-scale business scenarios, and has high performance such as orderly processing of operation events, high reliability and scalability.

[0147] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the operation events involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0148] Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization method, which is applied to producer microservices, such as Figure 5 As shown, the method includes:

[0149] S501. Acquire an operation event performed when a source system updates data in a source database, where the operation event includes a business identifier.

[0150] S502: Store the operation event in a message queue so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0151] The database synchronization method of the disclosed embodiment can execute the same operation event on the target database according to the operation event executed on the source database, thereby realizing data synchronization between the source database and the target database. Moreover, the data synchronization process of the disclosed embodiment does not rely on the log of the source system, thus avoiding the problem of synchronization delay caused by the log synchronization method and improving the real-time performance of database synchronization. In addition, the consumer microservice in the disclosed embodiment can use the consumer thread corresponding to the business identifier of the operation event to execute the operation event, so that the operation event of the same business can be executed serially by the same consumer thread, thereby ensuring the execution order of the operation event, that is, ensuring the data consistency in the source database and the target database.

[0152] In some embodiments of the present disclosure, obtaining an operation event performed when a source system updates data in a source database includes:

[0153] Periodically call the event list application programming interface API of the source system to obtain the event identifiers of various operation events executed when the source system updates the data in the source database in the current period;

[0154] Sending an operation event acquisition request to the source system, the operation event acquisition request including each event identifier;

[0155] Receive the operation events corresponding to the event identifiers sent by the source system.

[0156] In some embodiments of the present disclosure, the operation event includes the execution time of the operation event by the source system; storing the operation event in a message queue includes:

[0157] The received operation events are stored in the message queue in the order of execution time from the earliest to the latest.

[0158] In some embodiments of the present disclosure, obtaining an operation event performed when a source system updates data in a source database includes:

[0159] receiving an event identifier of an operation event sent by a source system, wherein the event identifier is sent after the source system executes the operation event to update data in a source database;

[0160] Sending an operation event acquisition request to the source system, where the operation event acquisition request includes an event identifier;

[0161] Receive the operation event corresponding to the event identifier sent by the source system.

[0162] In some embodiments of the present disclosure, storing an operation event in a message queue includes:

[0163] Using a preset hash algorithm, a hash operation is performed on the business identifier included in the operation event to obtain a hash value;

[0164] Use the hash value as the session identifier of the operation event;

[0165] The operation event is stored in the session corresponding to the session identifier of the message queue.

[0166] In some embodiments of the present disclosure, before using a preset hash algorithm to perform a hash operation on the service identifier included in the operation event to obtain a hash value, the following steps are further included:

[0167] From each identifier included in the operation event, obtain other identifiers except the event identifier;

[0168] If there are multiple other identifiers, concatenate them to form the business identifier.

[0169] Based on the same inventive concept, the disclosed embodiment also provides a database synchronization method, which is applied to consumer microservices, such as Figure 6 , the method comprising:

[0170] S601. Obtain an operation event stored by the producer microservice from a message queue. The operation event is an event executed when the source system updates data in the source database. The operation event includes a business identifier.

[0171] S602: Determine a consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database.

[0172] The database synchronization method of the disclosed embodiment can execute the same operation event on the target database according to the operation event executed on the source database, thereby realizing data synchronization between the source database and the target database. Moreover, the data synchronization process of the disclosed embodiment does not rely on the log of the source system, thus avoiding the problem of synchronization delay caused by the log synchronization method and improving the real-time performance of database synchronization. In addition, the consumer microservice in the disclosed embodiment can use the consumer thread corresponding to the business identifier of the operation event to execute the operation event, so that the operation event of the same business can be executed serially by the same consumer thread, thereby ensuring the execution order of the operation event, that is, ensuring the data consistency in the source database and the target database.

[0173] In some embodiments of the present disclosure, there are multiple consumer microservices;

[0174] Get the operation events stored by the producer microservice from the message queue, including:

[0175] When an operation event is detected in the session to which it has subscribed in the message queue, the operation event is obtained, where each consumer microservice subscribes to a different session;

[0176] Determine the consumer thread corresponding to the business identifier of the operation event, including:

[0177] Determines the consumer thread of the session used to perform the subscription.

[0178] Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization method, which is applied to a message queue, such as Figure 7 As shown, the method includes:

[0179] S701. Receive an operation event stored by the producer microservice. The operation event is an event executed when the source system updates the data in the source database. The operation event includes a business identifier, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0180] The database synchronization method of the disclosed embodiment can execute the same operation event on the target database according to the operation event executed on the source database, thereby realizing data synchronization between the source database and the target database. Moreover, the data synchronization process of the disclosed embodiment does not rely on the log of the source system, thus avoiding the problem of synchronization delay caused by the log synchronization method and improving the real-time performance of database synchronization. In addition, the consumer microservice in the disclosed embodiment can use the consumer thread corresponding to the business identifier of the operation event to execute the operation event, so that the operation event of the same business can be executed serially by the same consumer thread, thereby ensuring the execution order of the operation event, that is, ensuring the data consistency in the source database and the target database.

[0181] In some embodiments of the present disclosure, there are multiple consumer microservices; the method further includes:

[0182] When it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time, the target session is set to a locked state; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session;

[0183] The method further includes:

[0184] After the target consumer microservice obtains any operation event in the target session, receive the execution completion message sent by the target consumer microservice;

[0185] Delete the action event from the target session.

[0186] In some embodiments of the present disclosure, the method further includes:

[0187] If the target consumer microservice does not receive an execution completion message sent by the target consumer microservice within the timeout period after the target consumer microservice obtains an operation event in the target session, the target session is set to an unlocked state;

[0188] From other consumer microservices except the target consumer microservice, select a consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains operation events from the target session;

[0189] When it is detected that the selected consumer microservice obtains an operation event from the target session for the first time, the target session is set to a locked state.

[0190] In some embodiments of the present disclosure, the method further includes:

[0191] When a new consumer microservice is detected in the database synchronization system, the session subscribed by each consumer microservice is allocated according to the preset load balancing strategy.

[0192] In some embodiments of the present disclosure, the method further includes:

[0193] If it is detected that any operation event in the message queue meets the preset dead letter condition, the dead letter session to which the operation event meeting the dead letter condition belongs is determined;

[0194] Each operation event in the dead-letter session is stored in the dead-letter queue in sequence.

[0195] In some embodiments of the present disclosure, the message queue is a service bus.

[0196] Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization device, which is applied to producer microservices, such as Figure 8 As shown, the device includes: an acquisition module 801 and a storage module 802;

[0197] An acquisition module 801 is used to acquire an operation event performed when a source system updates data in a source database, where the operation event includes a business identifier;

[0198] The storage module 802 is used to store the operation event in the message queue so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0199] In some embodiments of the present disclosure, the acquisition module 801 is specifically used to:

[0200] Periodically call the event list application programming interface API of the source system to obtain the event identifiers of various operation events executed when the source system updates the data in the source database in the current period;

[0201] Sending an operation event acquisition request to the source system, the operation event acquisition request including each event identifier;

[0202] Receive the operation events corresponding to the event identifiers sent by the source system.

[0203] In some embodiments of the present disclosure, the operation event includes the execution time of the operation event by the source system; the storage module 802 is specifically used to:

[0204] The received operation events are stored in the message queue in the order of execution time from the earliest to the latest.

[0205] In some embodiments of the present disclosure, the acquisition module 801 is specifically used to:

[0206] receiving an event identifier of an operation event sent by a source system, wherein the event identifier is sent after the source system executes the operation event to update data in a source database;

[0207] Sending an operation event acquisition request to the source system, where the operation event acquisition request includes an event identifier;

[0208] Receive the operation event corresponding to the event identifier sent by the source system.

[0209] In some embodiments of the present disclosure, the storage module 802 is specifically configured to:

[0210] Using a preset hash algorithm, a hash operation is performed on the business identifier included in the operation event to obtain a hash value;

[0211] Use the hash value as the session identifier of the operation event;

[0212] The operation event is stored in the session corresponding to the session identifier of the message queue.

[0213] In some embodiments of the present disclosure, the device further comprises:

[0214] The acquisition module 801 is further used to perform a hash operation on the service identifier included in the operation event using a preset hash algorithm to obtain other identifiers except the event identifier from the identifiers included in the operation event before obtaining the hash value;

[0215] The splicing module is used to splice the other identifiers as a business identifier if there are multiple other identifiers.

[0216] Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization device, which is applied to consumer microservices, such as Fig. 9 As shown, the device includes: an acquisition module 901 and a consumption module 902;

[0217] The acquisition module 901 is used to obtain the operation event stored by the producer microservice from the message queue. The operation event is an event executed when the source system updates the data in the source database. The operation event includes a business identifier.

[0218] The consumption module 902 is used to determine the consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database.

[0219] In some embodiments of the present disclosure, there are multiple consumer microservices; the acquisition module 901 is specifically used to:

[0220] When an operation event is detected in the session to which it has subscribed in the message queue, the operation event is obtained, where each consumer microservice subscribes to a different session;

[0221] The consumption module 902 is specifically used for:

[0222] Determines the consumer thread of the session used to perform the subscription.

[0223] Based on the same inventive concept, the embodiment of the present disclosure also provides a database synchronization device, which is applied to a message queue, such as Fig.10 As shown, the device includes: a receiving module 1001;

[0224] The receiving module 1001 is used to receive the operation events stored by the producer microservice. The operation event is an event executed when the source system updates the data in the source database. The operation event includes a business identifier, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database.

[0225] In some embodiments of the present disclosure, there are multiple consumer microservices; the device further includes:

[0226] A setting module is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session;

[0227] The device also includes:

[0228] The receiving module 1001 is further configured to receive an execution completion message sent by the target consumer microservice after the target consumer microservice obtains any operation event in the target session;

[0229] The delete module is used to delete the operation event from the target session.

[0230] In some embodiments of the present disclosure, the device further comprises:

[0231] The setting module is further used to set the target session to an unlocked state if the execution completion message sent by the target consumer microservice is not received within a timeout period after the target consumer microservice obtains an operation event in the target session;

[0232] A selection module is used to select a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains the operation event from the target session;

[0233] The setting module is also used to set the target session to a locked state when it is detected that the selected consumer microservice obtains an operation event from the target session for the first time.

[0234] In some embodiments of the present disclosure, the device further comprises:

[0235] The allocation module is used to allocate the sessions subscribed by each consumer microservice according to a preset load balancing strategy when a new consumer microservice is detected in the database synchronization system.

[0236] In some embodiments of the present disclosure, the device further comprises:

[0237] A determination module, configured to determine the dead letter session to which the operation event meeting the dead letter condition belongs if it is detected that any operation event in the message queue meets the preset dead letter condition;

[0238] The transfer module is used to store each operation event in the dead letter session in the dead letter queue in sequence.

[0239] In some embodiments of the present disclosure, the message queue is a service bus.

[0240] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0241] Fig.11A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0242] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0243] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0244] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as a database synchronization method. For example, in some embodiments, the database synchronization method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the database synchronization method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the database synchronization method in any other appropriate manner (e.g., by means of firmware).

[0245] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0246] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0247] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0248] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0249] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0250] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0251] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0252] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A database synchronization system, comprising: Producer microservices, message queues, and consumer microservices; The producer microservice is used to obtain an operation event performed when the source system updates the data in the source database, and store the operation event in the message queue, wherein the operation event includes a business identifier; The consumer microservice is used to obtain an operation event from the message queue, determine a consumption thread corresponding to a business identifier of the operation event, and use the consumption thread to execute the operation event on a target database. There are multiple consumer microservices. The message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

2. The system according to claim 1, wherein: The producer microservice is specifically used for: Periodically calling the event list application programming interface API of the source system to obtain event identifiers of various operation events executed when the source system updates the data in the source database in the current period; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes each event identifier; Receive operation events corresponding to the event identifiers sent by the source system.

3. The system according to claim 2, wherein: The operation event includes the execution time of the operation event by the source system; the producer microservice is specifically used to: The received operation events are stored in the message queue in the order of execution time from earliest to latest.

4. The system according to claim 1, wherein: The producer microservice is specifically used for: receiving an event identifier of an operation event sent by the source system, wherein the event identifier is sent after the source system executes the operation event to update data in the source database; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes the event identifier; An operation event corresponding to the event identifier sent by the source system is received.

5. The system according to claim 1, wherein: The producer microservice is specifically used for: Using a preset hash algorithm, a hash operation is performed on the service identifier included in the operation event to obtain a hash value; Using the hash value as a session identifier of the operation event; The operation event is stored in the session corresponding to the session identifier of the message queue.

6. The system according to claim 5, wherein: The producer microservice is also used to: Before using a preset hash algorithm to perform a hash operation on the service identifier included in the operation event to obtain a hash value, obtaining other identifiers except the event identifier from each identifier included in the operation event; If there are multiple other identifiers, the other identifiers are concatenated to serve as the service identifier.

7. The system according to claim 5 or 6, wherein: There are multiple consumer microservices; the consumer microservice is specifically used for: When monitoring that an operation event exists in the session subscribed by itself in the message queue, the operation event is obtained, wherein each consumer microservice subscribes to a different session; A consumption thread for executing the subscribed session is determined, and the operation event is executed on the target database using the consumption thread.

8. The system according to claim 5 or 6, wherein: The message queue is also used for: After the target consumer microservice obtains any operation event in the target session, receiving an execution completion message sent by the target consumer microservice; Delete the operation event from the target session.

9. The system according to claim 8, wherein: The message queue is also used for: If the target consumer microservice does not receive an execution completion message sent by the target consumer microservice within the timeout period after the target consumer microservice obtains an operation event in the target session, setting the target session to an unlocked state; Selecting a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains operation events from the target session; When it is detected that the selected consumer microservice obtains an operation event from the target session for the first time, the target session is set to a locked state.

10. The system according to claim 8, wherein: The message queue is also used for: When it is detected that there is a newly added consumer microservice in the database synchronization system, the session subscribed by each consumer microservice is allocated according to a preset load balancing strategy.

11. The system according to claim 5 or 6, wherein: The message queue is also used for: If it is detected that any operation event in the message queue meets a preset dead letter condition, determining the dead letter session to which the operation event meeting the dead letter condition belongs; Each operation event in the dead letter session is stored in the dead letter queue in sequence.

12. The system according to any one of claims 1 to 11, wherein: The message queue is a service bus servicebus.

13. A database synchronization method, applied to a producer microservice, comprising: Acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier; The operation event is stored in a message queue, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database; There are multiple consumer microservices; the message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; The target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

14. The method according to claim 13, wherein: The operation events performed when the source system updates the data in the source database include: Periodically calling the event list application programming interface API of the source system to obtain event identifiers of various operation events executed when the source system updates the data in the source database in the current period; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes each event identifier; Receive operation events corresponding to the event identifiers sent by the source system.

15. The method according to claim 14, wherein: The operation event includes the execution time of the operation event by the source system; The storing the operation event into the message queue comprises: The received operation events are stored in the message queue in the order of execution time from earliest to latest.

16. The method according to claim 13, wherein: The operation events performed when the source system updates the data in the source database include: receiving an event identifier of an operation event sent by the source system, wherein the event identifier is sent after the source system executes the operation event to update data in the source database; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes the event identifier; An operation event corresponding to the event identifier sent by the source system is received.

17. The method according to claim 13, wherein: The storing the operation event into the message queue comprises: Using a preset hash algorithm, a hash operation is performed on the service identifier included in the operation event to obtain a hash value; Using the hash value as a session identifier of the operation event; The operation event is stored in the session corresponding to the session identifier of the message queue.

18. The method according to claim 17, before using a preset hash algorithm to perform a hash operation on the service identifier included in the operation event to obtain a hash value, further comprising: From the various identifiers included in the operation event, obtain other identifiers except the event identifier; If there are multiple other identifiers, the other identifiers are concatenated to serve as the service identifier.

19. A database synchronization method, applied to a consumer microservice, comprising: Obtaining an operation event stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier; Determine a consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database; There are multiple consumer microservices; the message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; The target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

20. The method according to claim 19, wherein: There are multiple consumer microservices; The obtaining the operation event stored by the producer microservice from the message queue includes: When monitoring that an operation event exists in the session subscribed by itself in the message queue, the operation event is obtained, wherein each consumer microservice subscribes to a different session; The determining of the consumption thread corresponding to the service identifier of the operation event includes: A consumer thread for the session that is to execute the subscription is determined.

21. A database synchronization method, applied to a message queue, comprising: Receive an operation event stored by a producer microservice, where the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier, so that a consumer microservice obtains the operation event from the message queue, determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on a target database; There are multiple consumer microservices, and the method further includes: When it is detected that the target consumer microservice obtains an operation event from a target session to which it has subscribed for the first time, the target session is set to a locked state; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

22. The method according to claim 21, further comprising: After the target consumer microservice obtains any operation event in the target session, receiving an execution completion message sent by the target consumer microservice; Delete the operation event from the target session.

23. The method according to claim 22, further comprising: If the target consumer microservice does not receive an execution completion message sent by the target consumer microservice within the timeout period after the target consumer microservice obtains an operation event in the target session, setting the target session to an unlocked state; Selecting a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains operation events from the target session; When it is detected that the selected consumer microservice obtains an operation event from the target session for the first time, the target session is set to a locked state.

24. The method of claim 22, further comprising: When it is detected that there is a newly added consumer microservice in the database synchronization system, the session subscribed by each consumer microservice is allocated according to a preset load balancing strategy.

25. The method of claim 21, further comprising: If it is detected that any operation event in the message queue meets a preset dead letter condition, determining the dead letter session to which the operation event meeting the dead letter condition belongs; Each operation event in the dead letter session is stored in the dead letter queue in sequence.

26. The method according to any one of claims 21 to 25, wherein: The message queue is a service bus servicebus.

27. A database synchronization device, applied to a producer microservice, comprising: An acquisition module, used to acquire an operation event performed when the source system updates data in the source database, wherein the operation event includes a business identifier; A storage module, used to store the operation event in a message queue, so that the consumer microservice obtains the operation event from the message queue, determines the consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on the target database; There are multiple consumer microservices; the message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; The target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

28. The device according to claim 27, wherein The acquisition module is specifically used for: Periodically calling the event list application programming interface API of the source system to obtain event identifiers of various operation events executed when the source system updates the data in the source database in the current period; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes each event identifier; Receive operation events corresponding to the event identifiers sent by the source system.

29. The device according to claim 28, wherein The operation event includes the execution time of the operation event by the source system; the storage module is specifically used to: The received operation events are stored in the message queue in the order of execution time from earliest to latest.

30. The device according to claim 27, wherein: The acquisition module is specifically used for: receiving an event identifier of an operation event sent by the source system, wherein the event identifier is sent after the source system executes the operation event to update data in the source database; Sending an operation event acquisition request to the source system, wherein the operation event acquisition request includes the event identifier; An operation event corresponding to the event identifier sent by the source system is received.

31. The apparatus of claim 27, wherein: The storage module is specifically used for: Using a preset hash algorithm, a hash operation is performed on the service identifier included in the operation event to obtain a hash value; Using the hash value as a session identifier of the operation event; The operation event is stored in the session corresponding to the session identifier of the message queue.

32. The apparatus of claim 31 , further comprising: The acquisition module is further configured to acquire other identifiers except the event identifier from the identifiers included in the operation event before performing a hash operation on the service identifier included in the operation event using a preset hash algorithm to obtain a hash value; The splicing module is used to splice the other identifiers as the service identifier if there are multiple other identifiers.

33. A database synchronization device, applied to a consumer microservice, comprising: An acquisition module is used to acquire operation events stored by the producer microservice from the message queue, wherein the operation event is an event executed when the source system updates data in the source database, and the operation event includes a business identifier; A consumption module, used to determine a consumption thread corresponding to the business identifier of the operation event, and use the consumption thread to execute the operation event on the target database; There are multiple consumer microservices; the message queue is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; The target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

34. The device according to claim 33, wherein There are multiple consumer microservices; the acquisition module is specifically used to: When monitoring that an operation event exists in the session subscribed by itself in the message queue, the operation event is obtained, wherein each consumer microservice subscribes to a different session; The consumption module is specifically used for: A consumer thread for the session that is to execute the subscription is determined.

35. A database synchronization device, applied to a message queue, comprising: A receiving module, used to receive an operation event stored by a producer microservice, wherein the operation event is an event executed when a source system updates data in a source database, and the operation event includes a business identifier, so that a consumer microservice obtains the operation event from the message queue, determines a consumption thread corresponding to the business identifier of the operation event, and uses the consumption thread to execute the operation event on a target database; There are multiple consumer microservices; the device also includes: A setting module is used to set the target session to a locked state when it is detected that the target consumer microservice obtains an operation event from the target session to which it has subscribed for the first time; wherein the target consumer microservice is any consumer microservice in the database synchronization system, and the locked state is used to indicate that the operation event in the target session is only obtained by the consumer microservice that subscribes to the target session.

36. The apparatus of claim 35, further comprising: The receiving module is further configured to receive an execution completion message sent by the target consumer microservice after the target consumer microservice obtains any operation event in the target session; The deleting module is used to delete the operation event from the target session.

37. The apparatus of claim 36, further comprising: The setting module is further configured to set the target session to an unlocked state if no execution completion message sent by the target consumer microservice is received within a timeout period after the target consumer microservice obtains an operation event in the target session; A selection module, configured to select a consumer microservice from other consumer microservices except the target consumer microservice to subscribe to the target session, so that the selected consumer microservice obtains an operation event from the target session; The setting module is further configured to set the target session to a locked state when it is detected that the selected consumer microservice obtains an operation event from the target session for the first time.

38. The apparatus of claim 36, further comprising: The allocation module is used to allocate the sessions subscribed by each consumer microservice according to a preset load balancing strategy when a new consumer microservice is detected in the database synchronization system.

39. The apparatus of claim 35, further comprising: A determination module, configured to determine the dead letter session to which the operation event that meets the dead letter condition belongs if it is detected that any operation event in the message queue meets a preset dead letter condition; The transfer module is used to store each operation event in the dead letter session in the dead letter queue in sequence.

40. The device according to any one of claims 35 to 39, wherein: The message queue is a service bus servicebus.

41. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 13 to 26.

42. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 13-26.

43. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 13 to 26.

Images