Database for fusing message queues and implementation method for fusing transactions
By developing a message processing center in the database kernel and combining the original transaction mechanism, the integration of message queues is achieved, and the complexity and transaction atomicity of database and message queues in the existing technology is solved, reducing the difficulty of development and avoiding single points risk.
Patent Information
- Application Number
- CN202411092980.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-17
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-09
AI Technical Summary
When integrating databases and message queues, the development of the prior art is complex and difficult, and it is impossible to achieve transaction atomicity across database systems, which easily leads to data inconsistency and single-point risk.
Develop a message processing center in the database kernel, combine the original transaction mechanism to realize the integration of message queues, reduce development difficulty, avoid relying on distributed transaction middleware, and ensure the atomicity of message sending.
It simplifies the integration of applications and message queues, reduces development complexity, avoids single point of risk, and realizes atomic transaction processing across database systems.
Smart Images

Figure CN118963927A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a method for implementing a database and a fusion transaction of a fusion message queue. Background Art
[0002] A message queue is a technology used to pass messages between applications. It is a decoupled communication method between message producers and message consumers, allowing the sender to send a message to a queue without immediately delivering the message to the receiver. Message queues are often used to implement asynchronous communication, where message producers and message consumers can process messages at different times and speeds.
[0003] Common message queue systems include RabbitMQ, Apache Kafka, Amazon SQS, etc. They provide different features and applicable scenarios, and you can choose a suitable message queue system according to specific needs.
[0004] Databases are software systems used to store and organize data. They are designed to manage large amounts of data efficiently, allowing users to quickly retrieve, update, and manage data.
[0005] Databases provide persistent storage for data, ensuring that data remains available after a system shutdown or restart. They also support transactions, allowing users to perform a series of operations and commit or rollback them as an atomic operation when necessary.
[0006] Common database types include relational databases (such as MySQL, PostgreSQL, and Oracle) and non-relational databases (such as MongoDB, Redis, and Elasticsearch), which provide different solutions for different data storage and access requirements.
[0007] Currently, there is a need to integrate databases and message queues in applications that require asynchronous processing and event-driven processing. Figure 1 , Figure 1 The schematic diagram of the structure of a database and message queue integration in the related technology is shown in FIG. Figure 1 As shown, 100 is an application, 101 is a database, and 102 is a message queue. A common practice is that application 100 persists the information to be sent in a library table (which can be a cache table) in database 101, and then extracts the message to be sent from the library table to the message queue through a trigger or a scheduled task to complete the message consumption.
[0008] This integration takes advantage of the database's storage characteristics to facilitate the persistence of messages to be sent, preventing the entire system from causing message loss. Figure 1The application shown needs to connect to database 101 and message queue 102. On the one hand, for R&D engineers who develop applications, they need to develop complex programs to ensure data communication between database 101 and message queue 102. The development workload is large and not user-friendly. On the other hand, in fusion transactions (fusion transactions include database table operations on the local database and notification operations to notify the peer database to perform related operations in scenarios where cross-database system cooperation is required to complete operations), if only the database's own transaction mechanism is used, only the atomicity of database operations and the persistence of messages to be sent in the database table can be guaranteed. Whether the message queue 102 sends the message successfully cannot be included in the transaction, that is, if the message fails to be sent in the end, the transaction cannot be rolled back. In some solutions, such as Figure 2 As shown, the distributed middleware 203 (complex transaction mechanism) is used to implement a complete integrated transaction. This is difficult to develop and relies too much on a single-point distributed middleware, which is prone to data blockage and brings single-point risks, resulting in inconsistent commit / rollback messages received by participants, inconsistent data, and transaction processing failure. Summary of the invention
[0009] In view of this, the present application discloses a database integrating message queues. The database includes a message processing center; the message processing center is communicatively connected with message producers and message consumers through the communication mechanism of the database; the database also includes message queues allocated to the message producers and message consumers. The message producer is used to generate a message and send it to the message processing center; the message processing center is used to write the message to a redo log based on a redo log mechanism supported by the database kernel in response to receiving the message, and write the message to the message queue; the message consumer is used to obtain the message from the message queue through the communication connection to complete the consumption.
[0010] In some embodiments, the message processing center, in response to an abnormal restart, reads an unprocessed target message from the redo log, and writes the target message into the message queue so that the message consumer completes consumption of the target message.
[0011] In some embodiments, the message producer corresponds to a sender cache, and the message consumer corresponds to a receiver cache; after generating a message, the message producer sends the generated message to the sender cache; the sender cache sets a message sequence number and a queue ID of the message queue for the message to associate the message with the message queue, and sends the message to the message processing center, and records the message sequence number in the sender cache as the sent message sequence number; the message processing center writes the message into a redo log, replies to the message producer with a successful sending confirmation, and records the message sequence number in the message processing center as the sending confirmation message sequence number; and writes the message into the message queue corresponding to the queue ID, pushes the message to the receiving end cache through the communication connection, and records the message sequence number in the message processing center as the message receiving sequence number; the receiving end cache pushes the message to the message consumer, and in response to the push success, replies to the message processing center with a successful receiving confirmation, and records the message sequence number in the receiving end cache as the message receiving sequence number; the message consumer receives the message and completes the consumption; the message processing center releases the local message after receiving the successful receiving confirmation fed back by the receiving end cache, and records the receiving confirmation message sequence number in the message processing center.
[0012] In some embodiments, the message processing center, in response to an abnormal restart, reads an unprocessed target message from the redo log, writes the target message to the message queue, pushes the target message to the receiving end cache through the communication connection, and records the message sequence number of the target message in the message processing center as the message reception sequence number; the receiving end cache, in response to receiving the target message, pushes the target message to the message consumer if the message sequence number of the target message is greater than the locally recorded message reception sequence number, replies to the message processing center with a successful reception confirmation in response to a successful push, and records the message sequence number of the target message in the receiving end cache as the message reception sequence number.
[0013] In some embodiments, when the incomplete processing refers to a situation where no sending confirmation is replied to the message producer, before the target message is written to the message queue, a sending confirmation is replied to the message producer for the target message, and the message sequence number of the target message is recorded in the message processing center as the sending confirmation message sequence number.
[0014] In the database shown in any of the previous embodiments of the present application, a message processing center and a compatibility mechanism between the center and the existing mechanism of the database kernel are developed in the database kernel, so that the database adds a message queue mechanism on the basis of the original mechanism. On the one hand, for application developers, they only need to connect the application (message generator) to the database, which reduces the difficulty of development and is more friendly. On the other hand, the original transaction mechanism of the database can be used to encapsulate the use of message queues to send messages in local database transactions, so that there is no need to develop or rely on distributed transaction middleware, and it is possible to monitor whether the message sending is successful to achieve transaction atomicity. The development difficulty is low, the system is simple, and it will also avoid the defects such as easy blocking due to excessive reliance on single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0015] The present application also includes a method for implementing a fusion transaction. The method is implemented based on a database as shown in any of the previous embodiments, the fusion transaction includes a first operation for a first database and a notification operation for notifying a second database to perform a second operation; the first database refers to a database as shown in any of the previous embodiments; the method includes: receiving a processing request for the fusion transaction sent to the first database; the first database responds to the fusion transaction, encapsulating the execution of the first operation and the notification operation into the same transaction, executing the transaction based on the transaction mechanism of its own kernel, ensuring the execution of the first operation, and using the message queue mechanism fused by the first database to complete the atomicity of the notification operation.
[0016] In some embodiments, the transaction is executed based on the transaction mechanism of its own kernel to ensure the execution of the first operation and the atomicity of completing the notification operation using the message queue mechanism integrated with the first database, including: pre-executing the first operation and pre-executing the notification operation using the message queue mechanism integrated with the first database; in response to the success of pre-executing the first operation and the success of pre-executing the notification operation, issuing a commit instruction to complete the execution of the first operation and the notification operation; in response to the failure of pre-executing the first operation or the failure of pre-executing the notification operation, issuing a rollback instruction to withdraw the first operation and the notification operation.
[0017] In some embodiments, the pre-execution of the notification operation using the message queue mechanism integrated with the first database includes: in response to receiving a notification message that notifies the second database to perform a second operation, using the message processing center, writing the notification message into a redo log, and writing into a message queue associated with the notification message, so as to transmit the notification message to a consumer-end application corresponding to the second database; in response to receiving a message reception confirmation returned by the consumer-end application, determining that the pre-execution of the notification operation is successful; in response to not receiving a message reception confirmation, determining that the pre-execution of the notification operation has failed.
[0018] In some embodiments, the consumer-end application corresponds to a receiving-end cache; the transmitting the notification message to the consumer-end application corresponding to the second database includes: transmitting the notification message to the receiving-end cache corresponding to the consumer-end application, so that the receiving-end cache corresponding to the consumer-end application returns a message reception confirmation to the message processing center when the notification message is successfully received; after issuing a commit instruction, the method also includes: in response to receiving a commit instruction regarding the notification message, using the message processing center, writing the commit instruction to a message queue associated with the commit instruction, so as to transmit the commit instruction to the receiving-end cache corresponding to the consumer-end application, so as to push the notification message to the consumer-end application by the receiving-end cache corresponding to the consumer-end application to complete the second operation; after issuing a rollback instruction, the method also includes: in response to receiving a rollback instruction regarding the notification message, using the message processing center, writing the rollback instruction to a message queue associated with the rollback instruction, so as to transmit the rollback instruction to the receiving-end cache corresponding to the consumer-end application, so as to release the notification message by the receiving-end cache corresponding to the consumer-end application.
[0019] In some embodiments, the method further includes: in response to pushing the notification message to the consumer-end application, returning a submission success to the message processing center; in response to the receiving-end cache corresponding to the consumer-end application releasing the notification message, returning a completion rollback to the message processing center; using the message processing center, returning the received submission success or completion rollback to the generator of the notification message to confirm the completion of the fusion transaction.
[0020] In the scheme recorded in any of the aforementioned embodiments, the original transaction mechanism of the database can be utilized to encapsulate the operations on the database and the notification operations of other databases in the same transaction, so that there is no need to develop or rely on distributed transaction middleware, and it is possible to monitor whether the database operations and notification operations are successful to achieve transaction atomicity. The development difficulty is low, it is friendly to application developers, the system is simple, and it will also avoid defects such as easy blocking due to excessive reliance on a single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0023] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.
[0024] Figure 1 The present invention is a schematic diagram of a structure of a database and a message queue integration in the related technology.
[0025] Figure 2 The present invention is an architecture diagram of an implementation mechanism of a fusion transaction in related technologies.
[0026] Figure 3 This is a structural diagram of a database that integrates message queues as shown in this application.
[0027] Figure 4 This is a schematic diagram of the asynchronous message transmission scenario shown in this application.
[0028] Figure 5 This is a schematic diagram of the message asynchronous processing flow shown in this application.
[0029] Figure 6 A flowchart of a method for implementing a fusion transaction shown in the present application.
[0030] Figure 7 This is a flow chart of a method for executing fusion transactions in a database according to the present application.
[0031] Figure 8 This is a flow chart of the process of completing the notification operation based on the message processing center shown in this application.
[0032] Fig. 9 This is a flow chart of the process of completing the notification operation based on the message processing center shown in this application.
[0033] Fig.10 A schematic diagram of the structure of a device for implementing a fusion transaction shown in an embodiment of the present application.
[0034] Fig.11 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION
[0035] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0036] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".
[0037] In related technologies, such as Figure 1 As shown, application 100 is connected to database 101 and message queue 102. Assume that the cross-database operation is account A minus 20, account B plus 20, and account A and account B are distributed in two databases. Account A is in database 101. Application 100 needs to generate an operation of subtracting 20 from account A, as well as a notification message to notify account B to add 20. Application 100 can send the operation and notification message to database 101, and database 101 can encapsulate the execution of A-20 and the storage of the notification message in a library table (which can be a cache table) as a transaction, that is, the storage of A-20 and the completion notification message can ensure atomicity. Subsequent applications can extract the notification message from the library table and send it to the message queue. The database corresponding to account B, as the consumer, can extract the notification message from the message queue. The pseudo code is as follows:
[0038] Start transaction
[0039] Account A: A-20;
[0040] The message is added to the cache table;
[0041] If Success
[0042] Commit;
[0043] Else
[0044] Rollback;
[0045] While the message cache table contains messages
[0046] Read messages from the cache table;
[0047] Send a message to the message queue;
[0048] If successfully received
[0049] Delete this message from the cache table;
[0050] Else
[0051] End this processing and wait for resending.
[0052] This approach has the following defects: it is necessary to implement a data storage mechanism that cooperates with the message queue in the database, which is difficult and complex. Moreover, if the message queue fails to send a message, since the data storage and the operation of account A-20 have been completed normally in database 101, account A-20 cannot be rolled back, and account B does not perform the operation of adding 20, resulting in an error.
[0053] In some related technologies, the following are used: Figure 2 The method shown in the figure ensures the atomicity of database operations and message notifications. Figure 2 , Figure 2 This is a diagram of the implementation mechanism architecture of a fusion transaction in the related technology. Figure 2 As shown, a distributed transaction middleware 203 is added between the application 200, the database 201 and the message queue 202. The middleware can be developed by the application developer or commercial or open source. The distributed transaction middleware 203 can encapsulate the operation of account A and the operation of notifying bank 2 using the message queue into one transaction. Combined with the previous example, the pseudo code is as follows:
[0054] Start transaction
[0055] Account A: A–20;
[0056] Notify account B: B+100;
[0057] If Success
[0058] Commit;
[0059] Else
[0060] Rollback.
[0061] Relying on distributed transaction middleware in this way can monitor the message sending status of the message queue and achieve transaction atomicity, but this solution has the following defects:
[0062] First, the system complexity increases, making development more difficult;
[0063] Second, when participants such as databases and message queues are waiting for instructions from the transaction middleware, synchronization blocking will occur. If a network communication failure causes the participant to be unable to receive instructions in time, the participant will be blocked forever.
[0064] Third, the introduction of distributed transaction middleware as a coordinator will become a single point of risk. Its failure will cause the system to become unavailable. Even if the coordinator adopts a master-slave solution, there is still the problem of transaction information synchronization when the master-slave switch occurs.
[0065] Fourth, due to single point failure of the coordinator or network failure, the commit / rollback messages received by the participants are inconsistent, resulting in inconsistent data between the participants and failure of transaction processing.
[0066] In view of this, the present application proposes a database integrating message queues. The core develops a message processing center in the database kernel and a compatible mechanism between the center and the existing mechanism of the database kernel, so that the database adds a message queue mechanism on the basis of the original mechanism. On the one hand, for application developers, they only need to connect to the database, which reduces the development difficulty and is more friendly; on the other hand, the original transaction mechanism of the database can be used to encapsulate the use of message queues to send messages in local database transactions, so that there is no need to develop or rely on distributed transaction middleware, and whether the message sending is successful can be monitored to achieve transaction atomicity. The development difficulty is low, the system is simple, and it will also avoid the defects such as easy blocking due to over-reliance on single-point distributed transaction middleware, and single-point risks leading to transaction synchronization failure.
[0067] The database is described below with reference to the accompanying drawings.
[0068] See also Figure 3 , Figure 3 This is a structural diagram of a database of a fusion message queue shown in this application. Figure 3 As shown, the database 320 includes a network connection layer 321 , a database core service layer 322 , a transaction processing layer 323 , a database storage layer 324 , and a database management and control tool 325 .
[0069] The core difference between this database and the conventional database in the related technology is that a message processing center 3221 is added to the database core service layer 322, and a message queue object 3241 is added to the database storage layer 324.
[0070] Message processing center 3221: interacts with message producers and consumers through the network connection layer to implement the core process of message interaction processing.
[0071] Message queue object 3241: saves the core data structure information of the message queue object, saves all message information, and provides persistent storage of messages.
[0072] Through the cooperation between the message processing center 3221 and the message queue object 3241, the message queue mechanism is implemented in the database 320. The original transaction processing layer 323 of the database can also be used to monitor the message sending status in the message queue mechanism to achieve the transaction atomicity of database operations and message sending.
[0073] The message processing center 3221 communicates with the message producers and message consumers through the communication mechanism of the database.
[0074] Message producers and consumers need to communicate with the message processing center 3221 and establish a communication connection with the processing center in advance. Since the message processing center is implemented based on the database kernel, the original database network connection layer 321 connection process can be reused:
[0075] Select the protocol port for the database service (e.g. the default port for MySQL is 3306, the default port for PostgreSQL is 5432), and start the service;
[0076] The application creates a socket and specifies the IP address and port of the database service;
[0077] Use the socket's connect method to connect to the database;
[0078] After the connection is successful, the database may initiate authentication for the application;
[0079] After the connection is established and verification is passed, you can use the socket's send, recv and other methods to send and receive data;
[0080] When communication is no longer needed, close the connection through the socket's close method.
[0081] The database 320 also includes message queues allocated to the message producers and message consumers. The message queues are only schematic representations and do not represent that there is only one message queue in the database. There may be one or more message queues between the message producers and message consumers.
[0082] The message producer is used to generate messages and send them to the message processing center.
[0083] The message format of the message may include queue ID, message sequence number, message length, and message content.
[0084] Queue ID: used to uniquely identify a message queue. Multiple message queues can be established between the same producer and consumer, and different queue IDs are assigned to multiple queues.
[0085] Message sequence number: can be incremented to determine whether the message is repeated or lost at different stages, mainly used for processing after various fault recovery. Of course, the message sequence number can also be generated in a decremental manner. This application takes incrementing as an example. For example, add 1 each time.
[0086] Message length: The length of the message information.
[0087] Message content: The entity of the message, written by the message producer and read and processed by the message consumer.
[0088] The message processing center is used for writing the message into a redo log and the message queue based on a redo log mechanism supported by the database kernel in response to receiving the message.
[0089] The redo log is called a redo log, which provides re-write operations and restores the page operations modified by the committed transaction to ensure the durability of the transaction. This application can use the original redo log mechanism of the database to implement the re-write operation of sending messages to ensure its durability.
[0090] The message queue may support a first-in-first-out mode, which is specifically set according to business requirements.
[0091] In some embodiments, the basic data structure of the message queue object includes message queue ID, message producer ID, message consumer ID, message sending sequence number, message sending confirmation sequence number, message receiving sequence number, message receiving confirmation sequence number, maximum number of messages, maximum message length, and message linked list.
[0092] The message queue ID is used to uniquely identify a message queue. Multiple message queues can be established between the same producer and consumer, and different queue IDs are assigned to the multiple queues.
[0093] The message producer ID is used to identify the message producer.
[0094] The message consumer ID is used to identify the message consumer.
[0095] The message sending sequence number is used to record the message sequence number when the message sending end writes the message into the queue.
[0096] The message sending confirmation sequence number is used to record the message sequence number when the message processing center completes the relevant processing of the sent message and replies to the sending end to confirm the sending.
[0097] The message receiving sequence number is used as the message sequence number recorded by the message processing center after the message is pushed to the receiving end.
[0098] The message reception confirmation sequence number is used by the receiving end to reply to the message processing center for reception confirmation after receiving the message, and the message processing center records the confirmation sequence number.
[0099] The maximum number of messages is the maximum number of messages supported by the message queue;
[0100] The maximum message length is the maximum length of a single message supported by the message queue;
[0101] The message linked list is used to store messages written into the message queue.
[0102] When a producer generates a message, the producer can write the queue ID corresponding to the message into the message. In this application, the message processing center can determine the corresponding message queue through the queue ID carried by the message and write it.
[0103] The message consumer is used to obtain the message from the message queue through the communication connection to complete consumption.
[0104] In this application, the process of sending messages from a message queue to a consumer can be completed by actively pushing or passively acquiring. Active push means that the database actively pushes messages in the message queue to the corresponding consumer by setting a trigger task or a scheduled task. Real-time passive acquisition means that the message consumer can periodically query whether there are messages of interest in the message queue, and if so, the acquisition is completed. This application does not specifically limit the message transmission method.
[0105] For example, the message producer generates message A and can access the message processing center 3221 in the database through the network connection layer 321 to complete the sending of message A. The message processing center 3221 can write message A into the redo log and the corresponding message queue object 3241, and push message A to the message consumer by active push to complete the asynchronous delivery of message A.
[0106] In the database shown in any of the previous embodiments of the present application, a message processing center and a compatibility mechanism between the center and the existing mechanism of the database kernel are developed in the database kernel, so that the database adds a message queue mechanism on the basis of the original mechanism. On the one hand, for application developers, they only need to connect the application (message generator) to the database, which reduces the difficulty of development and is more friendly. On the other hand, the original transaction mechanism of the database can be used to encapsulate the use of message queues to send messages in local database transactions, so that there is no need to develop or rely on distributed transaction middleware, and it is possible to monitor whether the message sending is successful to achieve transaction atomicity. The development difficulty is low, the system is simple, and it will also avoid the defects such as easy blocking due to excessive reliance on single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0107] In some embodiments, the message processing center, in response to an abnormal restart, reads an unprocessed target message from the redo log, and writes the target message into the message queue so that the message consumer completes consumption of the target message.
[0108] The target message refers to the message in the redo log that needs to be processed. In some methods, after the message is successfully sent to the consumer, the message will be deleted from the redo log. Therefore, after the system restarts abnormally, the message that remains in the redo log can basically be understood as the target message that has not been processed. In some methods, the message in the redo log can be labeled with whether it has been processed, and the target message can also be selected based on the label.
[0109] After an abnormal restart of the database, some messages may fail to be sent. Therefore, after the restart, the target message in the redolog can be read out and rewritten to the corresponding message queue according to the queue ID carried by the message, so that the consumer can obtain the target message.
[0110] Therefore, the original redo log mechanism of the database can be used to resend messages after the message queue mechanism is abnormal, thereby improving the success rate of message sending.
[0111] The following is a description of the message asynchronous transmission mechanism (message queue mechanism) based on the message processing center with reference to the accompanying drawings. Figure 4 , Figure 4 This is a schematic diagram of the asynchronous message transmission scenario shown in this application. Figure 4 The other basic units of the database are omitted, and only the message processing center is retained. The message producer corresponds to the sender cache, and the message consumer corresponds to the receiver cache. The message producer and the sender cache constitute the message generation end, and the message consumer and the receiver cache constitute the message receiving end.
[0112] Message producer: responsible for generating and sending messages to the message processing center;
[0113] Sender cache: Deployed together with the producer, producer messages are first cached here and sent to the message processing center after preprocessing;
[0114] Message processing center: responsible for message storage, forwarding, confirmation, retry, etc.;
[0115] Receiving end cache: Deployed together with consumers, the message processing center pushes messages to consumers, first cached here, and then the message cache pushes the messages to message consumers.
[0116] Message consumer: responsible for receiving messages and processing them accordingly.
[0117] See also Figure 5 , Figure 5 This is a schematic diagram of the message asynchronous processing flow shown in this application. Figure 5 As shown, the process includes S501-S506.
[0118] S501, after generating a message, the message producer sends the generated message to the sender cache.
[0119] The message at this time includes the content consumed by the consumer. For example, in the above example of the cross-database operation scenario, the message includes notifying account B+20.
[0120] S502, the sending end cache sets a message sequence number and a queue ID of the message queue for the message to associate the message with the message queue, and sends the message to the message processing center, and records the message sequence number in the sending end cache as a sending message sequence number.
[0121] The sender cache is pre-set with a business code that can encapsulate the message into the message format shown above. The message producer and consumer have obtained at least one message queue in the database message queue object in advance. This step can add the queue ID of the message queue based on the message content to associate the message with the message queue.
[0122] The message sequence number has at least two functions. The first function is to add 1 to the message sequence number of the current record in the cache to get the sequence number of the message to be sent. The second function is to compare it with the message sequence number returned by the message processing center after completing the redo log to determine whether the message is sent successfully.
[0123] For example, if the current message sequence number is 10, then after receiving a new message, the new message sequence number is 11, and then the message is sent to the message processing center, and the message sequence number is updated to 11. After the message processing center completes the redo log and returns a successful sending confirmation and a successful message sequence number 11, it can be compared with the updated message sequence number. If the two are consistent, it means that the message with sequence number 11 was indeed sent successfully. If the message sequence number 11 has not been received, it can be determined that the message has failed to be sent, which is convenient for the database transaction mechanism to track the message sending results and ensure atomicity.
[0124] S503, the message processing center writes the message into a redo log, replies to the message producer to confirm the success of the sending, and records the message sequence number as the sending confirmation message sequence number in the message processing center;
[0125] In this step, the sending sequence number of this message can be recorded, and the original redo log mechanism of the database can be used to complete the message persistence. The probe is set inside the database. After detecting the success of the redo log, a sending confirmation success message for the current message can be generated and returned to the message producer and / or the sender cache, and the sending confirmation message sequence number can be updated. For example, after receiving the message with sequence number 11 and completing the redo log, the sending confirmation success message of message 11 can be generated and returned to the message producer and / or the sender cache, and the sending confirmation message sequence number can be updated to 11.
[0126] The function of the sending confirmation message sequence number is to remove duplicates of received messages. If after receiving a message from the sender cache, it is found that the sequence number of the message is less than or equal to the current sending confirmation message sequence number, it means that the message is received repeatedly.
[0127] S504, the message processing center writes the message into the message queue corresponding to the queue ID, pushes the message to the receiving end cache through the communication connection, and records the message sequence number as the message receiving sequence number in the message processing center.
[0128] The message processing center uses the agreed message parsing protocol to parse the queue ID of the message and identify the corresponding message queue object. Then the message can be written into the message queue. After that, the messages in the message queue can be pushed to consumers in a first-in-first-out manner through triggering tasks or scheduled tasks.
[0129] The function of the message reception sequence number is to record the last message sequence number sent, which can be compared with the message sequence number of the successful reception confirmation returned by the receiving end cache after the message is pushed to the consumer to determine whether it is actually sent successfully, so as to facilitate the database to track the pre-execution status of the transaction.
[0130] For example, after the message with sequence number 11 is sent from the message queue to the receiving end cache, the message receiving sequence number can be updated to 11. After the receiving end cache successfully receives the message and pushes it to the message consumer, a successful reception confirmation of message 11 is returned, indicating that message 11 has indeed been received. If a successful reception confirmation is not received, it means that message 11 has failed to be received and the transaction pre-execution has failed, which is convenient for the database transaction mechanism to monitor.
[0131] S505, the receiving end cache pushes the message to the message consumer, responds to a successful push, replies a successful reception confirmation to the message processing center, and records the message sequence number as the message reception sequence number in the receiving end cache.
[0132] The message consumer receives the message and completes consumption.
[0133] The purpose of the message receiving sequence number in the receiving end cache is to deduplicate messages. If the receiving end cache receives a message, it can compare the sequence number of the message with the cached message receiving sequence number. If it is less than or equal to the current message receiving sequence number, it means that the message is duplicated.
[0134] For example, the sequence number of the currently recorded message reception is 11, and the sequence number of the newly received message is 11, which means that the message is received repeatedly and can be discarded.
[0135] S506: Upon receiving the successful reception confirmation from the receiving end buffer, the message processing center releases the local message and records the reception confirmation message sequence number in the message processing center.
[0136] The reception confirmation message sequence number has two functions. First, it can record the message sequence number that has been successfully consumed by the consumer, which is convenient for the database transaction mechanism to monitor. Second, it can deduplicate the message sending after the system restarts abnormally. For example, the reception confirmation message sequence number currently recorded is 11, indicating that message 11 has been consumed. Assuming that after the system restarts abnormally, after reading the target message from the redo log, the sequence number of the target message is 11, indicating that the message has been successfully consumed and no subsequent processing is required.
[0137] Through the process S501-S506, the database can be enhanced with the ability of asynchronous message transmission (message queue) based on the message processing center, and the transaction mechanism can conveniently monitor the results of message sending by recording the message sequence number.
[0138] In some embodiments, after an abnormal restart of the database occurs, resending can be completed based on the messages saved in the redo log, thereby improving the success rate of asynchronous message sending.
[0139] The message processing center, in response to an abnormal restart, reads an unprocessed target message from the redo log, writes the target message into the message queue, pushes the target message to the receiving end cache through the communication connection, and records a message sequence number of the target message as a message receiving sequence number in the message processing center.
[0140] The target message has been introduced in the previous example. In this example, the messages saved in the redo log are all target messages, which need to be read one by one and written into the message queue after the abnormal restart and pushed to the receiving end cache.
[0141] The target message that has not been processed includes two situations, one is the failure to send the message to the message queue or the receiving end cache, and the other is the failure to reply to the message producer for sending confirmation success and the failure to send the message. The sequence number of the read target message can be compared with the sequence number of the sending confirmation message recorded by the message processing center. If the two are consistent, it means that the target message has only failed to send the message to the message queue or the receiving end cache. If the sequence number of the sending confirmation message is smaller than the sequence number of the target message, it means that the sending confirmation success has not been replied to the message producer.
[0142] Regardless of the incomplete processing situation, the message processing center needs to write the target message into the message queue and push it to the receiving end cache.
[0143] The receiving-end cache, in response to receiving the target message, pushes the target message to the message consumer when the message sequence number of the target message is greater than the locally recorded message receiving sequence number, replies to the message processing center with a successful reception confirmation in response to a successful push, and records the message sequence number of the target message in the receiving-end cache as the message receiving sequence number.
[0144] In this step, the receiving end cache needs to perform deduplication check after receiving the target message. Specifically, the message sequence number of the target message can be compared with the current message reception sequence number recorded locally. If the message sequence number of the target message is less than or equal to the current message reception sequence number, it means that the message is a duplicate message and can be released directly, and the target message reception confirmation is returned to the message processing center.
[0145] If the message sequence number of the target message is greater than the current message receiving sequence number (for example, the current message receiving sequence number + 1), it means that the message is a new message and can be successfully received and pushed to the consumer, and the current message receiving sequence number is updated.
[0146] For example, if the target message has a sequence number of 10 and the message received in the receiving end cache record is 11, it means that the message has been processed and can be released to avoid duplication, and the message 10 is returned to indicate that it has been successfully received. The message processing center will update the received confirmation message sequence number and release message 10.
[0147] For another example, the target message sequence number is 12, and the message reception sequence number recorded in the receiving end cache is 11, indicating that the message has not been processed. Message 11 can be pushed to the consumer, and the return message 12 has been successfully received. The message processing center will update the reception confirmation message sequence number and release message 12.
[0148] The above method can be used to reprocess the unfinished messages based on the database redo log mechanism after an abnormal restart, thereby improving the success rate of message sending.
[0149] In some embodiments, when the incomplete processing refers to a situation where no sending confirmation is replied to the message producer, before the target message is written to the message queue, a sending confirmation is replied to the message producer for the target message, and the message sequence number of the target message is recorded in the message processing center as the sending confirmation message sequence number.
[0150] If by comparing the sequence number of the target message and the sending confirmation message sequence number of the message processing center, it is confirmed that the sending confirmation of the target message has not been successfully replied to the message producer, it is necessary to execute the step of replying the sending confirmation successfully and update the sending confirmation message sequence number.
[0151] In this example, these steps are only re-executed when the confirmation is not sent to the message producer successfully, which will not cause the repetition of the step.
[0152] This application proposes a method for implementing fusion transactions. In this method, the original transaction mechanism of the database can be used to encapsulate the operation of the database and the notification operation of other databases in the same transaction, so that the database operation and the notification operation can be monitored to realize the transaction atomicity successfully without developing or relying on distributed transaction middleware. The development difficulty is low, it is friendly to application developers, the system is simple, and it will also avoid the defects such as easy blocking caused by over-reliance on single-point distributed transaction middleware, and single-point risk leading to transaction synchronization failure.
[0153] The following is a description with reference to the accompanying drawings. Figure 6 , Figure 6 FIG. 1 is a flow chart of a method for implementing a fusion transaction shown in this application. Figure 6 As shown, the method includes S602-S604.
[0154] The method is implemented based on the database shown in any of the above embodiments. The fusion transaction includes a first operation on the first database and a notification operation for notifying the second database to perform the second operation.
[0155] The first database refers to the database shown in any of the previous embodiments. Figure 3 The database kernel implements the message asynchronous transmission (message queue) mechanism through the message processing center and the message queue object.
[0156] The second database refers to other databases in a cross-database operation scenario. In this scenario, the number of second databases is not limited and can be set according to actual needs.
[0157] S602: Receive a processing request for the fusion transaction sent to the first database.
[0158] The specific contents of the first operation and the second operation are generated according to business requirements.
[0159] Some fusion transaction operation logic is pre-deployed in the application. After receiving the previous business transaction, the first operation and the notification message that notifies the second database to perform the second operation can be generated to form a notification operation, a fusion transaction, and send it to the first database for execution through instructions. The first database can receive the request. It can be understood that the application is the message producer in the previous embodiment, and the second database and its corresponding consumer application are the message consumers. The consumer application, as the application corresponding to the second database, is used to issue instructions to the second database to complete the database operation.
[0160] For example, if a transaction occurs in the application between account A-20 in the first database and account B+20 in the second database, the first operation is for account A-20 in the first database, and the second operation is for account B+20 in the second database. The application uses A-20 and notification B+20 as a fusion transaction. After the consumer-side application receives the notification message of B+20, it can issue instructions to the second database to perform the operation of B+20. The pseudo code of the fusion transaction on the application side is as follows:
[0161] Start transaction
[0162] Account A-20;
[0163] Notify account B+20;
[0164] If Success
[0165] Commit;
[0166] Else
[0167] Rollback.
[0168] S604, the first database responds to the fused transaction, encapsulates the execution of the first operation and the notification operation into the same transaction, and executes the transaction based on the transaction mechanism of its own kernel to ensure the atomicity of executing the first operation and completing the notification operation using the message queue mechanism fused by the first database.
[0169] After receiving the previous fused transaction, the first database can ensure the execution of the first operation through the internal transaction mechanism, and use the message queue mechanism fused by the first database to complete the atomicity of notifying the second database and its corresponding consumer application to execute the second operation.
[0170] The transaction mechanism may include: an application initiates transaction processing, and the database combines the first operation of the library table and the notification message into a transaction through the transaction processing layer;
[0171] The transaction processing layer records the transaction startup status and coordinates each operation to perform transaction preprocessing: save the current library table status to the undo log, then execute the library table operation, and notify the transaction processing module after the operation is completed; notification message notification preprocessing: the message is pushed from the sending end to the receiving end cache, and the transaction processing module is notified after the operation is completed.
[0172] If the preprocessing of the two operations is successful, the transaction is committed: each operation returns a successful preprocessing, and the application initiates a transaction commit: the first operation is performed on the library table: the undo log is released, and the transaction is returned to be successful; the notification operation: the message is pushed from the receiving end cache to the message consumer, and the transaction is returned to be successful.
[0173] Any operation returns a preprocessing failure, or the application initiates a transaction rollback. Transaction rollback: rolls back the original modification through the undo log, releases the undo log, and returns a rollback success. Notification operation: the message is released from the receiving end cache and returns a rollback success.
[0174] Taking the previous example, the first database will treat account A-20 and notification account B+20 as one transaction to ensure the atomicity of the two.
[0175] Therefore, through the solution recorded in S602-S604, the original transaction mechanism of the database can be used to encapsulate the operations on the database and the notification operations of other databases in the same transaction, so that there is no need to develop or rely on distributed transaction middleware, and it is possible to monitor whether the database operations and notification operations are successful to achieve transaction atomicity. The development difficulty is low, it is friendly to application developers, the system is simple, and it will also avoid the defects such as easy blocking due to over-reliance on single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0176] In some embodiments, the database may execute the fusion transaction in two phases: the first phase is a pre-operation, and the second phase is a commit or rollback operation.
[0177] See also Figure 7 , Figure 7 This is a flow chart of a method for executing fusion transactions in a database according to the present application. Figure 7 The illustrated steps are for explanation of S604. Figure 7 As shown, the method may include S702-S706.
[0178] The first stage is pre-operation.
[0179] S702, pre-execute the first operation, and pre-execute the notification operation by using the message queue mechanism integrated with the first database.
[0180] The first operation shown in the pre-execution can be understood as a regular database operation. Figure 3 In the illustrated database, the pre-operation of account A-20 can be completed by using the core service layer and the storage layer.
[0181] Using the message queue mechanism integrated by the first database to pre-execute the notification operation is a new operation added to the database by this application. Figure 3 In the illustrated database, the message processing center and the message queue object can be used to complete the message push. The successful sending of the message to the consumer can confirm that the pre-notification operation is successful. If the message is not successfully sent, the task pre-notification operation fails.
[0182] The second stage is to commit or rollback the operation.
[0183] S704: In response to the success of pre-execution of the first operation and the success of pre-execution of the notification operation, a commit instruction is issued to complete execution of the first operation and the notification operation.
[0184] S706: In response to a failure in pre-execution of the first operation or a failure in pre-execution of the notification operation, issuing a rollback instruction to withdraw the first operation and the notification operation.
[0185] If the first operation is successful and the notification operation is pre-executed successfully, the application will issue a commit instruction to complete the actual operation. If any operation is pre-executed unsuccessfully, a rollback operation will be issued to ensure the atomicity of the transaction.
[0186] The solution recorded in S702-S706 refers to the logic of ensuring transaction atomicity in related technologies, and also sets a two-stage execution strategy in the database. The transaction will be committed only after all operations are pre-executed successfully, otherwise it will be rolled back, thereby ensuring the atomicity of the transaction.
[0187] In some embodiments, the notification operation can be completed based on the message processing center. Figure 8 , Figure 8 This is a flow chart of the process of completing the notification operation based on the message processing center shown in this application. Figure 8 The illustrated steps are for the description of the first stage S702. Figure 8 As shown, the method may include S802-S806.
[0188] S802, in response to receiving a notification message notifying the second database to perform a second operation, using the message processing center, writing the notification message into a redo log and a message queue associated with the notification message, so as to transmit the notification message to a consumer-end application corresponding to the second database.
[0189] The description of this step can refer to the previous embodiment and will not be described in detail here. The message queue associated with the notification message refers to the queue indicated by the queue ID carried by the notification message. It can be understood that the second database and its corresponding consumer application are the consumers in the previous embodiment. The application is the message producer.
[0190] S804: In response to receiving a message reception confirmation returned by the consumer-side application, determining that the pre-execution of the notification operation is successful.
[0191] S806: In response to not receiving a message reception confirmation, determining that the pre-execution of the notification operation has failed.
[0192] The consumer application corresponding to the second database receives the notification message or returns a successful reception to the message processing center, indicating that the pre-notification operation is successful. If a reception failure is returned or no message is returned, it indicates that the pre-notification operation has failed.
[0193] Through the solution recorded in S802-S806, asynchronous message forwarding can be completed based on the message processing center, and the message forwarding results can be monitored to ensure transaction atomicity.
[0194] In some embodiments, the consumer-end application corresponds to a receiving-end cache; transmitting the notification message to the consumer-end application corresponding to the second database includes transmitting the notification message to the receiving-end cache corresponding to the consumer-end application, so that the receiving-end cache corresponding to the consumer-end application returns a message reception confirmation to the message processing center when the notification message is successfully received.
[0195] After issuing the commit instruction, the method further includes:
[0196] In response to receiving a submission instruction regarding the notification message, using the message processing center, writing the submission instruction into a message queue associated with the submission instruction, so as to transmit the submission instruction to a receiving end cache corresponding to the consumer-end application, so that the receiving end cache corresponding to the consumer-end application pushes the notification message to the consumer-end application to complete the second operation;
[0197] After issuing the rollback instruction, the method further includes:
[0198] In response to receiving a rollback instruction regarding the notification message, the message processing center is used to write the rollback instruction into a message queue associated with the rollback instruction, so as to transmit the rollback instruction to the receiving-end cache corresponding to the consumer-end application, so that the notification message is released by the receiving-end cache corresponding to the consumer-end application.
[0199] In this embodiment, Figure 5 The difference between the schemes shown is that the message is transmitted to the receiving end cache, that is, the task pre-notification is successful. Only after receiving the commit instruction, the message will be pushed from the receiving end cache to the consumer, that is, the consumer application corresponding to the second database, ensuring the two-stage notification operation. Therefore, if there is a pre-execution failure in the fusion transaction, the message in the receiving end cache can be released without affecting the operation of the second database. Otherwise, if it is pushed directly to the consumer application, it may not be possible to roll back.
[0200] In some embodiments, in response to pushing the notification message to the consumer application, returning a submission success to the message processing center;
[0201] In response to the receiving end cache corresponding to the consumer end application releasing the notification message, returning the completion rollback to the message processing center;
[0202] The message processing center is used to return the received submission success or completion rollback to the generator of the notification message to confirm the completion of the fusion transaction.
[0203] The generator of the notification message is the upper layer application.
[0204] For example, if all pre-operations in the first phase are successful, a commit instruction will be issued in the second phase. When the receiving-end cache receives the commit instruction, it will actually push the notification message to the consumer-end application, so that the consumer-end application will issue an instruction to the second database to perform the second operation. After the message push is completed, the message processing center can return the information that the fusion transaction is completed to the upper-level application. If there is a pre-operation failure in the first phase, a rollback instruction will be issued in the second phase. When the receiving-end cache receives the rollback instruction, it will release the notification message locally and will not push it to the consumer-end application. After the message release is completed, the message processing center can return the information that the fusion transaction is rolled back to the upper-level application, indicating that the transaction execution failed. In this way, feedback can be given to the application.
[0205] The following combination Fig. 9 A detailed description of the two-stage process of completing the notification operation based on the message processing center. Fig. 9 , Fig. 9 This is a flow chart of the process of completing the notification operation based on the message processing center shown in this application. Fig. 9 The steps shown are two-stage transaction processing instructions. Fig. 9 As shown, the method may include S901-S910.
[0206] The first stage is preprocessing.
[0207] S901, after generating a notification message, the application pushes the notification message to the sender cache. For related instructions, refer to S501.
[0208] S902, the sending end cache sets a message sequence number and a queue ID of the message queue for the notification message to associate the notification message with the message queue, and sends the notification message to the message processing center, and records the message sequence number in the sending end cache as the sending message sequence number. For related instructions, refer to S502.
[0209] S903, the message processing center writes the message into the redo log, replies to the message producer with a successful sending confirmation, and records the message sequence number as the sending confirmation message sequence number in the message processing center. For related instructions, refer to S903.
[0210] S904, the message processing center writes the message into the message queue corresponding to the queue ID, pushes the message to the receiving end cache through the communication connection, and records the message sequence number as the message receiving sequence number in the message processing center. For related instructions, refer to S904.
[0211] S905, after receiving the notification message, the receiving end cache replies to the message processing center with a successful reception confirmation. The message processing center also replies to the message producer (application) with a successful reception confirmation and records the message sequence number as the message reception sequence number in the receiving end cache.
[0212] The difference between this step and S505 is that after the receiving end cache successfully receives the message, it does not push the message to the consumer first, and returns the receipt confirmation success. The message processing center will also reply the receipt confirmation success to the message producer (application). This allows the upper-layer application to clearly know that the first-stage pre-notification operation is successful. If the upper-layer application does not receive the receipt confirmation success or receives the receipt failure notification for a long time, the first-stage pre-notification of the task fails.
[0213] S906, the message processing center releases the local message after receiving the successful reception confirmation from the receiving end buffer, and records the reception confirmation message sequence number in the message processing center. For related operations, please refer to S506.
[0214] The above first phase is completed.
[0215] Phase 2.
[0216] S907, the message producer generates a commit instruction or a rollback instruction according to the return status of the first phase, and publishes it to the message processing center.
[0217] In this step, the application can generate a corresponding commit instruction or rollback instruction according to the preprocessing result of the first stage, and encapsulate it into a message through the sender cache and send it to the message processing center.
[0218] S908, the message processing center may forward the commit instruction or the rollback instruction to the receiving end cache.
[0219] This step can be accomplished by forwarding through a message queue or directly forwarding.
[0220] S909, if the receiving end cache receives a commit instruction, it pushes a notification message to the consumer, that is, the consumer application corresponding to the second database; if it receives a rollback instruction, it releases the notification message locally.
[0221] S910, after completing the corresponding operation, the receiving end will return a submission confirmation success or a rollback confirmation success to the message processing center, and the message processing center will also return a submission confirmation success or a rollback confirmation success to the message producer, thereby completing the entire fusion transaction.
[0222] Through the solution illustrated by S901-S910, based on the database integrated with the message queue mechanism, the original transaction mechanism of the database can be used to monitor message forwarding in a two-stage manner. Therefore, it is possible to monitor whether the database operation and notification operation are successful to achieve transaction atomicity without developing or relying on distributed transaction middleware. The development difficulty is low, it is friendly to application developers, the system is simple, and it will also avoid defects such as easy blocking due to over-reliance on single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0223] Corresponding to any of the above embodiments, the present application also proposes a device for implementing a fusion transaction. Fig.10 , Fig.10 This is a schematic diagram of a structure of a device for implementing a fusion transaction shown in an embodiment of the present application. The device is implemented based on a database as shown in any of the previous embodiments, and the fusion transaction includes a first operation on a first database and a notification operation for notifying a second database to perform a second operation; the first database refers to a database as shown in any of the previous embodiments. Fig.10 As shown, the implementation device 1000 of the fusion transaction includes:
[0224] A receiving module 1010 receives a processing request for the fusion transaction sent to the first database;
[0225] Execution module 1020, the first database responds to the fused transaction, encapsulates the execution of the first operation and the notification operation into the same transaction, and executes the transaction based on the transaction mechanism of its own kernel to ensure the execution of the first operation and the atomicity of completing the notification operation using the message queue mechanism fused by the first database.
[0226] In some embodiments, the execution module 1020 further:
[0227] Pre-execute the first operation, and pre-execute the notification operation using a message queue mechanism integrated with the first database;
[0228] In response to the success of pre-execution of the first operation and the success of pre-execution of the notification operation, issuing a commit instruction to complete execution of the first operation and the notification operation;
[0229] In response to a failure in pre-execution of the first operation or a failure in pre-execution of the notification operation, a rollback instruction is issued to withdraw the first operation and the notification operation.
[0230] In some embodiments, the execution module 1020 further:
[0231] In response to receiving a notification message notifying the second database to perform a second operation, using the message processing center, writing the notification message into a redo log and a message queue associated with the notification message, so as to transmit the notification message to a consumer application corresponding to the second database;
[0232] In response to receiving a message reception confirmation returned by the consumer-end application, determining that the pre-execution of the notification operation is successful;
[0233] In response to not receiving a message receipt confirmation, it is determined that pre-execution of the notification operation has failed.
[0234] In some embodiments, the consumer-side application corresponds to a receiving-side cache; the execution module 1020 further:
[0235] Transmitting the notification message to a receiving end cache corresponding to the consumer application, so that the receiving end cache corresponding to the consumer application returns a message reception confirmation to the message processing center when the notification message is successfully received;
[0236] After issuing the submission instruction, in response to receiving the submission instruction about the notification message, using the message processing center, writing the submission instruction into the message queue associated with the submission instruction, so as to transmit the submission instruction to the receiving end cache corresponding to the consumer application, so that the receiving end cache corresponding to the consumer application pushes the notification message to the consumer application to complete the second operation;
[0237] After issuing the rollback instruction, in response to receiving the rollback instruction regarding the notification message, the message processing center is used to write the rollback instruction into the message queue associated with the rollback instruction, so as to transmit the rollback instruction to the receiving-end cache corresponding to the consumer-end application, so that the notification message is released by the receiving-end cache corresponding to the consumer-end application.
[0238] In some embodiments, the execution module 1020 further:
[0239] In response to pushing the notification message to the consumer application, returning a submission success to the message processing center;
[0240] In response to the receiving end cache corresponding to the consumer end application releasing the notification message, returning the completion rollback to the message processing center;
[0241] The message processing center is used to return the received submission success or completion rollback to the generator of the notification message to confirm the completion of the fusion transaction.
[0242] In the scheme recorded in the aforementioned device embodiment, the original transaction mechanism of the database can be utilized to encapsulate the operations on the database and the notification operations of other databases in the same transaction, so that there is no need to develop or rely on distributed transaction middleware, and it is possible to monitor whether the database operations and notification operations are successful to achieve transaction atomicity. The development difficulty is low, it is friendly to application developers, the system is simple, and it will also avoid the defects such as easy blocking caused by over-reliance on single-point distributed transaction middleware, and single-point risks that lead to transaction synchronization failure.
[0243] The embodiment of the device for implementing fusion transaction shown in the present application can be applied to electronic devices. Accordingly, the present application discloses an electronic device, which may include: a processor.
[0244] Memory used to store processor-executable instructions.
[0245] The processor is configured to call the executable instructions stored in the memory to implement the method for implementing the fusion transaction shown in any of the above embodiments.
[0246] See also Fig.11 , Fig.11 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application.
[0247] like Fig.11 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operation data for the processor, and a non-volatile memory for storing instructions corresponding to the implementation device of the fusion transaction.
[0248] The embodiments of the device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Fig.11 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.
[0249] It is understandable that, in order to improve the processing speed, the corresponding instructions of the implementation device of the fusion transaction can also be directly stored in the memory, which is not limited here.
[0250] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can be used to enable a processor to execute the implementation method of the fusion transaction shown in any of the aforementioned embodiments.
[0251] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program code.
[0252] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".
[0253] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0254] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0255] The subject matter described in this application and the embodiments of the functional loading can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structure disclosed in this application and its structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in the computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or control the loading of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
[0256] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by loading according to input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.
[0257] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be loadably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.
[0258] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.
[0259] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.
[0260] Similarly, although loads are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these loads be executed in the particular order shown or executed sequentially, or requiring that all illustrated loads be executed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.
[0261] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.
[0262] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.
Claims
1. A database integrating a message queue, characterized in that: The database includes a message processing center; the message processing center is connected to the message producer and the message consumer through the communication mechanism of the database; the database also includes message queues allocated to the message producer and the message consumer; The message producer is used to generate a message and send it to the message processing center; The message processing center is configured to, in response to receiving the message, write the message into a redo log based on a redo log mechanism supported by the database kernel, and write the message into the message queue; The message consumer is used to obtain the message from the message queue through the communication connection to complete consumption.
2. A database for integrating message queues according to claim 1, characterized in that: The message processing center, in response to an abnormal restart, reads the target message that has not been completely processed from the redo log, and writes the target message into the message queue, so that the message consumer completes consumption of the target message.
3. A database for integrating message queues according to claim 2, characterized in that: The message producer corresponds to the sender cache, and the message consumer corresponds to the receiver cache; After generating the message, the message producer sends the generated message to the sender cache; The sending end cache sets a message sequence number and a queue ID of the message queue for the message to associate the message with the message queue, sends the message to the message processing center, and records the message sequence number in the sending end cache as a sending message sequence number; The message processing center writes the message into the redo log, replies to the message producer with a successful sending confirmation, and records the message sequence number as the sending confirmation message sequence number in the message processing center; and writes the message into the message queue corresponding to the queue ID, pushes the message to the receiving end cache through the communication connection, and records the message sequence number as the message receiving sequence number in the message processing center; The receiving end cache pushes the message to the message consumer, responds to the message processing center with a successful push, and records the message sequence number as the message reception sequence number in the receiving end cache; The message consumer receives the message and completes consumption; The message processing center releases the local message upon receiving the successful reception confirmation fed back by the receiving end buffer, and records the reception confirmation message sequence number in the message processing center.
4. A database for integrating message queues according to claim 3, characterized in that: The message processing center, in response to the abnormal restart, reads the target message that has not been processed from the redo log, writes the target message into the message queue, pushes the target message to the receiving end cache through the communication connection, and records the message sequence number of the target message as the message receiving sequence number in the message processing center; The receiving-end cache, in response to receiving the target message, pushes the target message to the message consumer when the message sequence number of the target message is greater than the locally recorded message receiving sequence number, replies to the message processing center with a successful reception confirmation in response to a successful push, and records the message sequence number of the target message in the receiving-end cache as the message receiving sequence number.
5. A database for integrating message queues according to claim 4, characterized in that: In the case where the incomplete processing means that no sending confirmation is replied to the message producer, before the target message is written into the message queue, a sending confirmation of the target message is replied to the message producer, and the message sequence number of the target message is recorded in the message processing center as the sending confirmation message sequence number.
6. A method for implementing a fusion transaction, characterized in that: The method is implemented based on the database according to any one of claims 1 to 5, wherein the fusion transaction includes a first operation on a first database and a notification operation for notifying a second database to perform a second operation; The first database refers to the database according to any one of claims 1 to 5; The method comprises: receiving a processing request for the fusion transaction sent to the first database; In response to the fused transaction, the first database encapsulates the execution of the first operation and the notification operation into the same transaction, and executes the transaction based on the transaction mechanism of its own kernel to ensure the execution of the first operation and the atomicity of completing the notification operation using the message queue mechanism fused by the first database.
7. A method for implementing a fusion transaction according to claim 6, characterized in that: The executing of the transaction is based on the transaction mechanism of the kernel itself, ensuring the execution of the first operation, and utilizing the atomicity of the notification operation by the message queue mechanism integrated by the first database, including: Pre-execute the first operation, and pre-execute the notification operation using a message queue mechanism integrated with the first database; In response to the success of pre-execution of the first operation and the success of pre-execution of the notification operation, issuing a commit instruction to complete execution of the first operation and the notification operation; In response to a failure in pre-execution of the first operation or a failure in pre-execution of the notification operation, a rollback instruction is issued to withdraw the first operation and the notification operation.
8. A method for implementing a fusion transaction according to claim 7, characterized in that: The pre-execution of the notification operation by using the message queue mechanism integrated with the first database includes: In response to receiving a notification message notifying the second database to perform a second operation, using the message processing center, writing the notification message into a redo log and a message queue associated with the notification message, so as to transmit the notification message to a consumer application corresponding to the second database; In response to receiving a message reception confirmation returned by the consumer-end application, determining that the pre-execution of the notification operation is successful; In response to not receiving a message receipt confirmation, it is determined that pre-execution of the notification operation has failed.
9. A method for implementing a fusion transaction according to claim 8, characterized in that: The consumer-side application corresponds to a receiving-side cache; The transmitting the notification message to the consumer application corresponding to the second database includes: Transmitting the notification message to a receiving end cache corresponding to the consumer application, so that the receiving end cache corresponding to the consumer application returns a message reception confirmation to the message processing center when the notification message is successfully received; After issuing the commit instruction, the method further includes: In response to receiving a submission instruction regarding the notification message, using the message processing center, writing the submission instruction into a message queue associated with the submission instruction, so as to transmit the submission instruction to a receiving end cache corresponding to the consumer-end application, so that the receiving end cache corresponding to the consumer-end application pushes the notification message to the consumer-end application to complete the second operation; After issuing the rollback instruction, the method further includes: In response to receiving a rollback instruction regarding the notification message, the message processing center is used to write the rollback instruction into a message queue associated with the rollback instruction, so as to transmit the rollback instruction to the receiving-end cache corresponding to the consumer-end application, so that the notification message is released by the receiving-end cache corresponding to the consumer-end application.
10. A method for implementing a fusion transaction according to claim 9, characterized in that: The method further comprises: In response to pushing the notification message to the consumer application, returning a submission success to the message processing center; In response to the receiving end cache corresponding to the consumer end application releasing the notification message, returning the completion rollback to the message processing center; The message processing center is used to return the received submission success or completion rollback to the generator of the notification message to confirm the completion of the fusion transaction.
Citation Information
Patent Citations
Message synchronization method and system
CN106598762A
Queue message uniformity implementing method and device, equipment and storage medium
CN108009027A
A lightweight message queue implementation method based on database and a storage device
CN109271417A
An asynchronous message reliable delivery and processing method and device
CN113094362A
Information processing method and device based on message queue and computer equipment
CN114637611A
Cited By
Message queue processing method and device, electronic equipment and storage medium
CN119311441A
Atomization submitting method and delivery method for database transaction and asynchronous message
CN120567924A
