A semantic method and device based on message middleware

Abstract

The application discloses a semantic method and device based on a message middleware, and comprises the following steps: in the case that a semantic request for the message middleware is received, a semantic identifier in the semantic request is acquired; a target semantic method is determined based on the semantic identifier, the target semantic method being a semantic method based on an idempotent principle or a semantic method based on a transaction principle; in the case that the target semantic method is the semantic method based on the idempotent principle, the message middleware is used to execute the semantic method on a single message queue in the semantic request only once; and in the case that the target semantic method is the semantic method based on the transaction principle, the message middleware is used to execute the semantic method on multiple message queues in the semantic request only once. The above process executes the semantic method only once based on the idempotent principle or the transaction principle, and no longer needs an additional stream data calculation platform to guarantee the semantic method only once, so that the processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data stream computing technology, and in particular to a semantic method and apparatus based on message middleware. Background Technology

[0002] In stream computing systems, messages have different semantics. Generally, semantics are divided into three types:

[0003] at-most-once: This method involves sending the message at most once, without needing to verify whether it was successfully sent or requiring a return value. (This value can be lost, for example, in scenarios involving request logs.)

[0004] at-least-once: Allows messages to be repeated, but delivery is guaranteed. (Repeated messages are allowed, such as video information; if modified N times, the latest message will be used.)

[0005] Exactly-once: Ensures messages are sent exactly once, guaranteeing no loss and no duplication. (This applies only once, such as in scenarios like payments and transfers that rely on contextual calculations.)

[0006] Existing message middleware at most supports only two semantic types: at-most-once and at-least-once (the easiest to support, requiring only control on the production side). Their primary focus remains on middleware availability, high performance, and channel connectivity. Delegating the processing of these semantics to the respective producers and consumers, who then rely on additional streaming data computing platforms to guarantee exactly-once semantics, results in low processing efficiency. Summary of the Invention

[0007] In view of this, the present invention provides a semantic method and apparatus based on message middleware to address the problem that existing message middleware at most supports only two semantic types: at-most-once and at-least-once (the easiest to support, requiring only production-side control). Its main objectives remain the availability, high performance, and channel performance of the middleware. Furthermore, delegating the processing of related semantics to the corresponding producers and consumers themselves, relying on additional streaming data computing platforms to ensure exactly-once semantics, results in low processing efficiency. The specific solution is as follows:

[0008] A semantic method based on message middleware includes:

[0009] Upon receiving a semantic request to the message middleware, obtain the semantic identifier in the semantic request;

[0010] The target semantic method is determined based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle;

[0011] When the target semantic method is an idempotent semantic method, the semantic method is executed once only once on a single message queue in the semantic request based on the message middleware.

[0012] When the target semantic method is a transaction-based semantic method, the semantic method is executed once only once on multiple message queues in the semantic request based on the message middleware.

[0013] Optionally, in the case that the target semantic method is an idempotent semantic method, the method described above involves executing the semantic method exactly once on a single message queue in the semantic request based on the message middleware, including:

[0014] Assign a first identifier to each message queue, wherein each first identifier is arranged in ascending order according to the generation order of the message queues;

[0015] If the producer successfully sends a semantic request but fails to ACK, a duplicate semantic request is received from the producer, and a second identifier corresponding to the duplicate semantic request is obtained.

[0016] If the first identifier and the second identifier are the same, discard the producer message corresponding to the duplicate semantic request and return an Ack success signal.

[0017] The above methods may also include:

[0018] If any semantic request is detected, obtain the third identifier;

[0019] The third identifier is compared with the latest identifier. If the third identifier is less than the latest identifier, the producer message corresponding to any semantic request is discarded and an Ack success signal is returned.

[0020] The above methods may also include:

[0021] Store the message identifier of each message in the single message queue;

[0022] If a consumer restarts due to a fault or a new consumer restarts, the latest message identifier among all message identifiers is obtained, and consumption is restarted based on the latest message identifier.

[0023] Optionally, in the above method, if the target semantic method is a transaction-based semantic method, the semantic method is executed once only once on multiple message queues in the semantic request based on the message middleware, including:

[0024] Assign transaction identifiers corresponding to the plurality of message queues;

[0025] Execute transactions corresponding to the plurality of message queues based on the transaction identifier;

[0026] Upon receiving a request to end a transaction, the system obtains the end identifier from the request and performs a commit or termination operation on the semantic request based on the end identifier.

[0027] Optionally, the above method involves executing transactions corresponding to the plurality of message queues based on the transaction identifier, including:

[0028] Receive multiple message queues from the semantic request, add the multiple message queues to the corresponding transaction and store them;

[0029] Upon receiving a semantic request for the current message queue, determine whether the current message in the current message queue belongs to a transaction message based on the transaction identifier. If so, write the current message to the transaction buffer.

[0030] Obtain the subscribed message and store it. When a current ACK request for the current message is received, determine whether the current ACK belongs to the corresponding transaction based on the transaction identifier. If it does, mark the current message as PENDING_ACK until the current message is committed or terminated based on the subscribed message.

[0031] The above methods may also include:

[0032] When at least one target message queue among the plurality of message queues performs a semantic request exactly once, the semantic request is performed on the target message queue based on the idempotency principle.

[0033] A semantic device based on message middleware, comprising:

[0034] The acquisition module is used to acquire the semantic identifier in the semantic request when a semantic request for the message middleware is received.

[0035] The determination module is used to determine the target semantic method based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle;

[0036] The first execution module is used to execute the semantic method exactly once on a single message queue in the semantic request based on the message middleware when the target semantic method is a semantic method based on the idempotency principle.

[0037] The second execution module is used to execute the semantic method exactly once on multiple message queues in the semantic request based on the message middleware when the target semantic method is a semantic method based on the transaction principle.

[0038] Optionally, in the aforementioned apparatus, the first execution module includes:

[0039] The first allocation unit is used to allocate a first identifier to the individual message queue, wherein each first identifier is arranged in ascending order according to the generation order of the individual message queues;

[0040] The receiving and obtaining unit is used to receive a repeated semantic request sent by the producer again when the producer successfully sends a semantic request but fails to Ack, and to obtain a second identifier corresponding to the repeated semantic request.

[0041] The discard and return unit is used to discard the producer message corresponding to the duplicate semantic request and return an Ack success signal if the first identifier and the second identifier are the same.

[0042] Optionally, in the aforementioned apparatus, the second execution module includes:

[0043] The second allocation unit is used to pre-allocate transaction identifiers corresponding to the plurality of message queues;

[0044] An execution unit is configured to execute transactions corresponding to the plurality of message queues based on the transaction identifier;

[0045] The acquisition and operation unit is used to acquire the end identifier in the end transaction request when a transaction end request is received, and to perform a submission or termination operation on the semantic request based on the end identifier.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] This invention discloses a semantic method and apparatus based on message middleware, comprising: upon receiving a semantic request to the message middleware, obtaining a semantic identifier in the semantic request; determining a target semantic method based on the semantic identifier, wherein the target semantic method is either an idempotent semantic method or a transaction-based semantic method; if the target semantic method is an idempotent semantic method, executing the semantic method exactly once on a single message queue in the semantic request based on the message middleware; if the target semantic method is a transaction-based semantic method, executing the semantic method exactly once on multiple message queues in the semantic request based on the message middleware. The above process executes the semantic method exactly once based on either the idempotent principle or the transaction principle, respectively, eliminating the need for an additional streaming data computing platform to guarantee exactly-once semantics, thus improving processing efficiency. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a semantic method based on message middleware disclosed in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram illustrating the one and only one semantic implementation of a single message queue disclosed in an embodiment of this application.

[0051] Figure 3 This is a schematic diagram illustrating the one-and-only-once semantic implementation of multiple message queues disclosed in an embodiment of this application.

[0052] Figure 4 This is a block diagram of a semantic device structure based on message middleware disclosed in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0055] This invention discloses a semantic method and apparatus based on message middleware, which is applied to the processing of semantic requests. In the prior art, message middleware is a very important supporting middleware in system architecture, especially in high-concurrency scenarios. It is an indispensable tool for decoupling and smoothing out peaks and valleys in the system. Message middleware transforms "synchronous call" behavior into "asynchronous call".

[0056] Database transactions, on the one hand, refer to data-based transactions, which are implemented based on single-machine logs. Their core principle is to treat a set of behaviors as a logical unit to be executed, either all of them succeed or all of them fail, making them relatively simple to implement.

[0057] Distributed transactions evolved from database transactions, especially with the rise of microservice architectures. The "set of actions" in database transactions may now involve multiple different services executing operations simultaneously. When transitioning from single-machine database transactions to distributed transactions, the relatively reliable method calls, inter-process communication methods, and single-machine logs of single-machine systems become unusable. Furthermore, the instability of network communication can hinder the transmission of information between services. Therefore, distributed transaction solutions such as 2PC, 3PC, XA, and Saga have emerged in microservice architectures. These solutions require highly complex implementations and require collaboration among all participating services, resulting in high coupling, high implementation difficulty, and low processing efficiency.

[0058] To address the aforementioned issues, this invention implements exactly-once semantics under a single queue. While maintaining essentially no performance impact compared to the other two semantic implementation scenarios, it guarantees that a message produced by the producer will be consumed by the consumer exactly once. It also implements exactly-once semantics under multiple queues. Furthermore, in scenarios where a single service generates multiple message sending and acknowledgment (Ack) actions, these message production and acknowledgment (Ack) actions either all succeed or all fail.

[0059] The execution flow of the method is as follows: Figure 1 As shown, the steps include:

[0060] S101. Upon receiving a semantic request for the message middleware, obtain the semantic identifier in the semantic request;

[0061] In this embodiment of the invention, the semantic identifier in the semantic request is set based on experience or specific circumstances. For example, this embodiment of the invention considers the specific application scenario in the semantic request and the number of message queues in the semantic request, and assigns corresponding semantic identifiers based on the number. For example, a single message queue corresponds to one semantic identifier, and multiple message queues correspond to another semantic identifier. Each semantic identifier is assigned a semantic method that can be implemented one and only once, and the semantic identifier is associated with the corresponding semantic method. Preferably, the semantic identifier of a single message queue corresponds to the idempotency principle semantic method, and the semantic identifiers of multiple message queues correspond to the transaction principle semantic method. In this embodiment of the invention, the specific form of the semantic identifier is not limited.

[0062] S102. Determine the target semantic method based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle;

[0063] In this embodiment of the invention, the semantic identifier is identified, and it is determined whether the semantic identifier belongs to a single message queue semantic identifier or a multi-message queue semantic identifier. If the semantic identifier belongs to a single message queue semantic identifier, the target semantic method is a semantic method based on the idempotency principle. If the semantic identifier belongs to a multi-message queue semantic identifier, the target semantic method is a semantic method based on the transaction principle.

[0064] S103. When the target semantic method is a semantic method based on the idempotency principle, the semantic method is executed once only once on a single message queue in the semantic request based on the message middleware.

[0065] In this embodiment of the invention, when the target semantic method is an idempotent semantic method, the message middleware executes the semantic method exactly once on a single message queue topic in the semantic request. The idempotency principle means that the result of one or more operations on the same operation is consistent, and different results will not occur due to multiple operations. For example, if message deduplication is enabled at the cluster, namespace, or topic level, and an idempotent producer is configured on the message producer side, then when an exception causes the producer to resend a message, the duplicate message will only be written once in the broker. Therefore, the exactly-once semantic method for a single message queue is mainly guaranteed by implementing idempotency. The idempotency principle is mainly achieved through setting idempotency for the producer, the broker deduplication mechanism, and idempotent consumption by the consumer. The specific setting process is as follows:

[0066] A unique producer identifier (ProducerName) is set, which is assigned as the first identifier to each message queue. Each first identifier is ordered sequentially according to the generation order of the message queue. The first identifier can be a specified non-repeating, incrementing sequence (Sequenceld), where Sequenceld is a unique, incrementing sequence ID used only under the exactly-once strategy. Sequenceld needs to be persisted to ensure that when a Producer fails and restarts or is switched to a new Producer, a specific sequence ID can be used to restart publishing from a specific message queue. A mapping relationship between each ProducerName and SequenceID is maintained and persisted to ensure that in the event of a Broker failure, other Brokers can reload this mapping relationship from the persisted data.

[0067] If the producer successfully sends a semantic request but the ACK fails, a duplicate semantic request is received from the producer. A second identifier corresponding to a single message queue in the duplicate semantic request is obtained. The second identifier can be a specified non-repeating incrementing sequence (Sequenceld). If it is the same message queue, the corresponding identifiers are the same. The same message queue performs the same operation. It is determined whether the first identifier and the second identifier are the same. If the first identifier and the second identifier are the same, the producer message corresponding to the duplicate semantic request is discarded and an ACK success signal is returned. If the first identifier and the second identifier are different, the normal ACK process is directly followed.

[0068] Furthermore, upon detecting any semantic request, a third identifier is obtained, wherein the third identifier can be a specified non-repeating incrementing sequence (SequenceId). If they are from the same message queue, the corresponding identifiers are the same. The third identifier is compared with the current latest identifier LastSequenceId, wherein LastSequenceId is the messageId of the last generated message. If the third identifier is less than the current latest identifier, it is considered a duplicate message, the producer message corresponding to any semantic identifier is discarded, and an Ack success signal is returned.

[0069] Furthermore, Redis / DB or similar methods are used to store the message identifier (messageId) of each message in the single message queue. The messageId is a unique ID for the message and contains the coordinates of the message stored in the message queue and cluster. The cluster can be a Broker cluster, a store cluster, etc. When a consumer restarts due to a failure or a new consumer is restarted, the latest message identifier (lastMessageId) is retrieved from the list of message identifiers. The lastMessageId is the messageId of the last consumed message. Consumption resumes using a Reader with the latest message identifier lastMessageId. The Reader is a reader interface that reads the cursor position of a Consumer, which is a coordinate containing the MessageId. Subsequent consumption can then begin using this MessageId. After consumption is completed and an ACK is executed, the Consumer's cursor is updated to the next data item.

[0070] Furthermore, one advantage of choosing a non-repeating, incrementing SequenceId is that SequenceId can be compared using <= LastSequenceId, instead of finding a matching SequenceId from all previous SequenceIds. This allows us to obtain the desired result with minimal algorithmic complexity. For example, if 10 data entries are synchronized, with corresponding SequenceIds [1,2,3,4,5,6,7,8,9,10], a non-incremental approach requires comparing the newest ID with all the existing ones to determine if it's a duplicate; while an incrementing approach only needs to compare the latest 10 with the newest ID. When this mapping relationship reaches 1 million or 100 million entries, the non-incremental approach simply cannot proceed. The incrementing approach can complete the comparison with near-zero performance loss (the latest SequenceId mapping exists locally on the Broker in a fixed-length set of the latest SequenceIds, eliminating the need to retrieve it from persistent storage each time).

[0071] The specific execution process is as follows: Figure 2 As shown, the first scenario represents a normal message sending and ACK process. The producer sends (1,m1) to the message middleware pub / sub broker, and the broker returns ACK (1,m1) to the producer and appends (1,m1) to the log topic. In the second sending, due to network issues, the producer has successfully sent the message and sends (2,m2) to the message middleware pub / sub broker, but ACK (2,m2) fails to reach the producer. However, append (2,m2) is still appended to the log topic. In the third sending, when the broker receives the producer send (2,m2), it will find that the message has already been sent in the log topic. At this time, the broker will do two things: discard the message and return a signal indicating that the producer successfully ACKed.

[0072] S104. If the target semantic method is a transaction-based semantic method, the semantic method is executed once only once on multiple message queues in the semantic request based on the message middleware.

[0073] In this embodiment of the invention, when the target semantic method is a transaction-based semantic method, the semantic method is executed exactly once on multiple message queues in the semantic request based on the message middleware. When at least one target message queue among the multiple message queues executes the semantic request exactly once, the semantic request is executed exactly once on the target message queue based on the idempotency principle. The specific execution process is as described in S103, and will not be repeated here. The execution of the semantic method exactly once on multiple message queues is guaranteed by a transaction mechanism, mainly including:

[0074] Add a Transaction Coordinator (TC) component to serve as the logical coordinator for transactions.

[0075] Add a Transaction Buffer (TB) component to serve as a temporary data buffer.

[0076] The acknowledgment (Ack) action adds intermediate states to a transaction for management during transaction processing.

[0077] The transaction metadata and logs generated by TC and TB will be persistently stored through internal meta-topics to ensure that the entire transaction can be recovered from the corresponding topic in the event of failure of any node in each stage.

[0078] The above treatment can achieve the following effects:

[0079] To ensure atomicity, multiple topics are confirmed.

[0080] All operations performed within a transaction either all succeed or all fail.

[0081] Consumers can only read messages indicating that a transaction has been committed.

[0082] The specific execution process is as follows: Figure 3 As shown, TC runs as part of the Broker. Figure 3 For clearer illustration, it has been extracted. Logs are recorded using topics. Many status data in the message middleware are recorded using topics. These topics are called metadata topics. TC is mainly responsible for managing transaction metadata.

[0083] TB is responsible for handling and sending messages with Transaction. Data stored in TB will not be sent to consumers until the transaction is committed. Data stored in TB is persistent.

[0084] Transaction Pending Ack (TP): Responsible for handling Ack requests with Transaction attributes.

[0085] The specific execution process is as follows:

[0086] S1. Start transaction

[0087] Connecting to a broker's TC module, the TC will assign a transaction identifier (TxnID) to the transactions corresponding to the multiple message queues. In the transaction log topic, the TxnID and its status (Status = Open) will be recorded (i.e.,...). Figure 3 In step <1a>), to ensure persistence, it is guaranteed that if the current TC or the broker where the current TC is located has a problem, it can be recovered in other TCs.

[0088] Step 2. Transaction execution

[0089] At this stage, the client uses the Transaction API to perform message operations.

[0090] S2.1. Add message queue partition information to the Txn transaction.

[0091] Before sending a message to a Topic partition, a message is first sent to the broker's Transaction Control (TC) to add the Topic partition to the transaction. The TC then records the Topic partition changes in the transaction in the Transaction Log Topic (i.e., ...). Figure 3 Step <2.1a>) is used to ensure persistence, so that if the current TC or the broker where the current TC is located has a problem, it can be recovered in other TCs.

[0092] S2.2. Produce messages to the Topic partition

[0093] This production process is the same as the normal message production process. The only difference is that the message will contain a TxnID. The broker will check whether the message contains a TxnID to determine whether it belongs to a transaction message. For example, when a client sends a message, if it is a message that requires transaction, a TC parameter will be added to the message. At this time, TC will add the TxnID corresponding to this transaction to the message.

[0094] Txn txn=client.newTxn(...);

[0095] Producer.sender(txn).msg(“Message 1”);

[0096] Producer.sender(txn).msg(“Message 2”);

[0097] txn.end();

[0098] Messages 1 and 2 carry the same TxnID.

[0099] If it is not included, the current message is determined not to be a transaction message, and the normal process of storage and distribution is directly followed. If it is included, the current message is determined to be a transaction message, and the current message is first written into the transaction's TB (Transaction Buffer).

[0100] S2.3. Add the subscription to Txn

[0101] Before subscribing to messages, a transaction support request is sent to the broker's TC. The TC will add the subscribed message to the Transaction Log Topic (step <2.3a>). This step ensures that the TC knows which subscriptions are involved in the transaction so that it can correctly commit or terminate the corresponding subscriptions when the transaction TxnEnds.

[0102] S2.4. Ack operation on subscription

[0103] The consumer's message acknowledgment process is the same as the normal message acknowledgment process. The only difference is that the Ack request will include the TxnID. The broker will check whether the Ack contains the TxnID to determine if it belongs to a specific transaction. For example:

[0104] Txn txn=client.newTxn(...);

[0105] Msg msg=Consumer.consumer(txn);

[0106] / / At this point, the TxnID already exists in the msg.

[0107] / / TODO something

[0108] msg.Ack()

[0109] Producer.sender(txn).msg(“Message 1”);

[0110] Producer.sender(txn).msg(“Message 2”);

[0111] txn.end();

[0112] The ACK operation of msg, along with the sending of messages 1 and 2 below, all carry the same TxnID.

[0113] If a message does not contain a TxnID, it is determined that it does not belong to any transaction, and the normal ACK process is initiated. If a message contains a TxnID, it is determined that it belongs to a transaction, and the broker will mark the message as PENDING_ACK. The PENDING_ACK status means that the message will not be ACKed by other consumers until the transaction is committed or aborted. If two transactions attempt to acknowledge a message, only one transaction will succeed, and the other transaction will abort.

[0114] S3. Transaction complete.

[0115] The client can decide to commit or abort the transaction.

[0116] S3.1. End Transaction Request

[0117] After the client completes the transaction-related operations, it sends a request to the broker's TC to end the transaction. One of the request fields has an end identifier that can be either COMMITTING or ABORTING.

[0118] After TC received this request:

[0119] Write the operation, whether COMMITTING or ABORTING, to the Transaction Log (step <3.1a>).

[0120] Begin performing the corresponding COMMITTING or ABORTING operations on all Topic partitions involved in this transaction. This will be described in detail in S3.2.

[0121] After all corresponding Topic partitions have successfully completed COMMITTING or ABORTING, the TC will write COMMITTED or ABORTED to the Transaction Log. This will be described in detail in S3.3.

[0122] S3.2. Final Processing Procedure

[0123] In this process, TC completes the transaction by sending messages or performing Ack operations on all Topic partitions within the COMMITTING or ABORTING transaction.

[0124] During the COMMITTING operation, the produced message will enter the normal Topic message sending process from TB and then become visible to the consumer (step <3.2a>); the Ack operation status will change from PENDING_ACK to ACK, indicating that consumption is complete and the confirmation is successful, and the message will be stored in the cursor storage component Cursor Ledger.

[0125] During the ABORTING operation, the produced messages will be discarded by TB, and TB will clean up and reclaim space when the transaction is aborted; the PENDING_ACK corresponding to the Ack operation will be cleared (step <2.4a>), and the message that was Acked will be resent to other consumers and stored in the cursor storage component Cursor Ledger.

[0126] S3.3. Mark the transaction as COMMITTING or ABORTING

[0127] After all corresponding partitions have successfully completed COMMITTING or ABORTING, the TC will write COMMITTED or ABORTED to the Transaction Log, indicating that the transaction has been completed (step <3.3a>). At this point, it is safe to delete the log information related to this transaction from the transaction log.

[0128] This invention discloses a semantic method based on message middleware, comprising: upon receiving a semantic request to the message middleware, obtaining a semantic identifier from the semantic request; determining a target semantic method based on the semantic identifier, wherein the target semantic method is either an idempotent semantic method or a transaction-based semantic method; if the target semantic method is an idempotent semantic method, executing the semantic method exactly once on a single message queue in the semantic request based on the message middleware; if the target semantic method is a transaction-based semantic method, executing the semantic method exactly once on multiple message queues in the semantic request based on the message middleware. The above process executes the semantic method exactly once based on either the idempotent principle or the transaction principle, respectively, eliminating the need for an additional streaming data computing platform to guarantee the semantic accuracy exactly once, thus improving processing efficiency.

[0129] To illustrate the above methods, it's important to understand that the idempotency principle and the transaction principle are two independent scenarios and are not mutually exclusive. For example:

[0130] If A transfers 100 to B, there will be 2 messages:

[0131] Message 1-topic-1-a minus 100

[0132] Message 2-topic-2-b plus 100

[0133] Among them, the idempotent method applies to individual messages, meaning that it will not cause message 1 to be subtracted by 100 multiple times or message 2 to be added by 100 multiple times due to network or other reasons.

[0134] The transaction-based approach is a higher-dimensional method (for multiple cross-topic operations). It guarantees that the operations on message 1 and message 2 (sending and receiving messages) either both succeed or both fail. Each operation within a transaction (sending and receiving messages) also needs to implement corresponding semantics (this depends on the specific requirements, such as exactly once, at most once, or at least once semantics). When the chosen semantic is exactly once, it is implemented based on idempotency.

[0135] Therefore, the idempotency principle and the transaction principle are not mutually exclusive. The idempotency principle is based on the one and only one guarantee of a single operation; the transaction principle is based on the one and only one guarantee of multiple operations.

[0136] Based on the above methods, the exactly-once semantics are implemented only once for a single queue. With minimal performance impact compared to the other two semantic implementation scenarios, it guarantees that a message produced by the producer will be consumed by the consumer exactly once. The exactly-once semantics are implemented across multiple queues. In scenarios where a business operation generates multiple message sending and acknowledgment (Ack) actions, these message production and acknowledgment (Ack) actions either all succeed or all fail. These two scenarios greatly simplify the cost of implementing exactly-once semantics for payment, transfer, context-dependent, and computationally related business scenarios. With decoupling, it also simplifies the system architecture complexity and workload of the entire business process.

[0137] Based on the aforementioned message middleware-based semantic method, this embodiment of the invention also provides a message middleware-based semantic device, the structural block diagram of which is shown below. Figure 4 As shown, it includes:

[0138] Acquisition module 201, determination module 202, first execution module 203, and second execution module 204.

[0139] in,

[0140] The acquisition module 201 is used to acquire the semantic identifier in the semantic request when a semantic request for the message middleware is received.

[0141] The determining module 202 is used to determine a target semantic method based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle;

[0142] The first execution module 203 is used to execute a semantic method exactly once on a single message queue in the semantic request based on the message middleware when the target semantic method is a semantic method based on the idempotency principle.

[0143] The second execution module 204 is used to execute the semantic method once only on multiple message queues in the semantic request based on the message middleware when the target semantic method is a semantic method based on the transaction principle.

[0144] This invention discloses a semantic device based on message middleware, comprising: upon receiving a semantic request to the message middleware, obtaining a semantic identifier in the semantic request; determining a target semantic method based on the semantic identifier, wherein the target semantic method is either an idempotent semantic method or a transaction-based semantic method; if the target semantic method is an idempotent semantic method, executing the semantic method exactly once on a single message queue in the semantic request based on the message middleware; if the target semantic method is a transaction-based semantic method, executing the semantic method exactly once on multiple message queues in the semantic request based on the message middleware. The above process executes the semantic method exactly once based on either the idempotent principle or the transaction principle, respectively, eliminating the need for an additional streaming data computing platform to guarantee exactly-once semantics, thus improving processing efficiency.

[0145] In this embodiment of the invention, the first execution module 203 includes:

[0146] The system comprises a first allocation unit 205, a receiving and acquiring unit 206, and a discarding and returning unit 207.

[0147] in,

[0148] The first allocation unit 205 is used to allocate a first identifier to the single message queue, wherein each first identifier is a non-repeating incrementing sequence;

[0149] The receiving and obtaining unit 206 is used to receive a repeated semantic request sent by the producer again when the producer successfully sends a semantic request but fails to Ack, and to obtain a second identifier corresponding to the repeated semantic request.

[0150] The discard and return unit 207 is used to discard the producer message corresponding to the duplicate semantic request and return an Ack success signal when the first identifier and the second identifier are the same.

[0151] In this embodiment of the invention, the second execution module 204 includes:

[0152] The second allocation unit 208, the execution unit 209, and the acquisition and operation unit 210.

[0153] in,

[0154] The second allocation unit 208 is used to pre-allocate transaction identifiers corresponding to the plurality of message queues;

[0155] The execution unit 209 is used to execute transactions corresponding to the plurality of message queues based on the transaction identifier;

[0156] The acquisition and operation unit 210 is used to acquire the end identifier in the end transaction request when a transaction end request is received, and to perform a submission or termination operation on the semantic request based on the end identifier.

[0157] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0163] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0165] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A semantic method based on message middleware, characterized in that, include: Upon receiving a semantic request for the message middleware, obtain the semantic identifier in the semantic request; The target semantic method is determined based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle; When the target semantic method is an idempotent semantic method, the semantic method is executed once only once on a single message queue in the semantic request based on the message middleware. When the target semantic method is a transaction-based semantic method, the semantic method is executed once only once on multiple message queues in the semantic request based on the message middleware. Wherein, if the target semantic method is a transaction-based semantic method, the semantic method is executed once only once on multiple message queues in the semantic request based on the message middleware, including: Assign transaction identifiers corresponding to the plurality of message queues; Execute the transaction corresponding to the plurality of message queues based on the transaction identifier; Upon receiving a request to end a transaction, the end identifier in the request is obtained, and the semantic request is submitted or terminated based on the end identifier. Executing transactions corresponding to the plurality of message queues based on the transaction identifier includes: Receive multiple message queues from the semantic request, add the multiple message queues to the corresponding transaction and store them; Upon receiving a semantic request for the current message queue, determine whether the current message in the current message queue belongs to a transaction message based on the transaction identifier. If so, write the current message to the transaction buffer. Obtain the subscribed message and store it. When a current ACK request for the current message is received, determine whether the current ACK belongs to the corresponding transaction based on the transaction identifier. If it does, mark the current message as PENDING_ACK until the current message is determined to be committed or terminated based on the subscribed message.

2. The method according to claim 1, characterized in that, When the target semantic method is an idempotent semantic method, the semantic method is executed once only once on a single message queue in the semantic request based on the message middleware, including: Assign a first identifier to each message queue, wherein each first identifier is arranged in ascending order according to the generation order of the message queues; If the producer successfully sends a semantic request but fails to ACK, a duplicate semantic request is received from the producer, and a second identifier corresponding to the duplicate semantic request is obtained. If the first identifier and the second identifier are the same, discard the producer message corresponding to the duplicate semantic request and return an Ack success signal.

3. The method according to claim 2, characterized in that, Also includes: If any semantic request is detected, obtain the third identifier; The third identifier is compared with the latest identifier. If the third identifier is less than the latest identifier, the producer message corresponding to any semantic request is discarded and an Ack success signal is returned.

4. The method according to claim 2, characterized in that, Also includes: Store the message identifier of each message in the single message queue; If a consumer restarts due to a fault or a new consumer restarts, the latest message identifier among all message identifiers is obtained, and consumption is restarted based on the latest message identifier.

5. The method according to claim 1, characterized in that, Also includes: When at least one target message queue among the plurality of message queues performs a semantic request exactly once, the semantic request is performed on the target message queue based on the idempotency principle.

6. A semantic device based on message middleware, characterized in that, include: The acquisition module is used to acquire the semantic identifier in the semantic request when a semantic request for the message middleware is received. The determination module is used to determine the target semantic method based on the semantic identifier, wherein the target semantic method is a semantic method based on the idempotency principle or a semantic method based on the transaction principle; The first execution module is used to execute the semantic method exactly once on a single message queue in the semantic request based on the message middleware when the target semantic method is a semantic method based on the idempotency principle. The second execution module is used to execute the semantic method exactly once on multiple message queues in the semantic request based on the message middleware when the target semantic method is a semantic method based on the transaction principle. The second execution module includes: The second allocation unit is used to pre-allocate transaction identifiers corresponding to the plurality of message queues; An execution unit is configured to execute transactions corresponding to the plurality of message queues based on the transaction identifier; The acquisition and operation unit is used to acquire the end identifier in the end transaction request when a transaction end request is received, and to perform a submission or termination operation on the semantic request based on the end identifier. Executing transactions corresponding to the plurality of message queues based on the transaction identifier includes: Receive multiple message queues from the semantic request, add the multiple message queues to the corresponding transaction and store them; Upon receiving a semantic request for the current message queue, determine whether the current message in the current message queue belongs to a transaction message based on the transaction identifier. If so, write the current message to the transaction buffer. Obtain the subscribed message and store it. When a current ACK request for the current message is received, determine whether the current ACK belongs to the corresponding transaction based on the transaction identifier. If it does, mark the current message as PENDING_ACK until the current message is determined to be committed or terminated based on the subscribed message.

7. The apparatus according to claim 6, characterized in that, The first execution module includes: The first allocation unit is used to allocate a first identifier to the individual message queue, wherein each first identifier is arranged in ascending order according to the generation order of the individual message queues; The receiving and obtaining unit is used to receive a repeated semantic request sent by the producer again when the producer successfully sends a semantic request but fails to Ack, and to obtain a second identifier corresponding to the repeated semantic request. The discard and return unit is used to discard the producer message corresponding to the duplicate semantic request and return an Ack success signal if the first identifier and the second identifier are the same.

Images