Distributed transaction consistency guarantee method based on outbox mode

Abstract

The application relates to a distributed transaction consistency guarantee method based on an Outbox mode, which comprises the following steps: simultaneously writing business data into a business data table and writing a to-be-sent message into an Outbox message table in a local transaction of a business database through a business application and submitting the transaction; performing polling on the Outbox message table at a preset polling period through a message relay service, acquiring the to-be-sent message meeting the condition after adding an exclusive lock and skipping the locked record, sending the to-be-sent message to a message queue and updating the state when the sending succeeds; performing idempotent downstream processing based on a message identifier through a consumption end after querying a consumption record table; in response to sending failure, increasing a retry number and performing next-time retry scheduling, and when the retry number reaches an upper limit, migrating the to-be-sent message to a dead letter table. The application has the advantages of guaranteeing atomicity of business data and message persistence, supporting multi-instance concurrent safety, guaranteeing idempotency of a consumption end and controllable retry scheduling.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and in particular to a method for ensuring consistency of distributed transactions based on the Outbox pattern. Background Technology

[0002] With the widespread application of distributed systems in business scenarios such as medical image quality control, order fulfillment, and user behavior analysis, business operations often require triggering asynchronous notifications or processing from downstream services. This process typically relies on message queues to decouple services. However, the local transactions of the business database and message delivery in the message queue belong to two independent resource domains, and the coordination between the two becomes the core challenge of distributed transaction consistency.

[0003] Existing solutions to this problem typically face a dilemma. The approach of directly committing business transactions before message delivery suffers from several drawbacks. Since business database writing and message delivery operate on separate network communication links, a service crash or network interruption before message delivery means that while business data is persisted, downstream messages are lost, causing downstream processing to lose its correspondence with the business state. While a two-phase commit protocol can guarantee cross-resource consistency, its blocking nature forces transaction participants to wait for each other, leading to a collapse in overall system throughput under high concurrency scenarios, making it difficult to meet business requirements for response latency. The local message table approach, while merging message persistence and business transactions into a single local transaction to ensure atomicity of writes, often suffers from fragmented or unmodeled engineering elements such as message delivery failure retry control, mutual exclusion coordination for concurrent multi-instance fetching, and duplicate consumption protection on the consumer side. This results in a lack of coordination across multiple engineering dimensions, including backoff scheduling, concurrency safety, and idempotency protection, leading to a solution that is theoretically feasible but practically unmaintainable. Summary of the Invention

[0004] In order to ensure the atomicity of business data writing and message persistence, and to achieve reliable delivery of downstream messages and eventual consistency of the business system through the systematic coordination of engineering elements such as precise backoff control, multi-instance concurrency security, and consumer-side idempotent protection, this application provides a distributed transaction consistency guarantee method and related apparatus based on the Outbox pattern.

[0005] This application provides a distributed transaction consistency guarantee method based on the Outbox pattern, which adopts the following technical solution: A distributed transaction consistency guarantee method based on the Outbox pattern includes: S1. Responding to business requests through business applications, business data is simultaneously written to the business data table and the message to be sent triggered by the business request is written to the Outbox message table in the local transaction of the business database. The local transaction is then committed. The Outbox message table persists the message to be sent, as well as the status field, retry count field, next retry time field, and creation time field corresponding to the message to be sent. S2. Poll the Outbox message table through the message relay service at a preset polling period, add an exclusive lock to the query results and skip the locked records, and obtain the messages to be sent whose status field is pending and whose next retry time field is no later than the current time; S3. Send the message to be sent to the message queue through the message relay service, and update the status field to "sent" when the sending is successful; S4. The consumer queries the consumption record table based on the message identifier carried in the message to be sent. If the message identifier is not found in the consumption record table, the downstream processing of the message to be sent is performed and the message identifier is written into the consumption record table. S5. In response to the failure of sending a message to be sent through the message relay service, increment the retry count field, perform the next retry scheduling for the message to be sent, and when the retry count field reaches the preset retry count limit, migrate the message to be sent to the dead letter table.

[0006] By adopting the above technical solutions, business data writing and message persistence are merged into the same local transaction for submission, ensuring atomicity of business status and messages to be sent at the writing level. This avoids message loss caused by network interruptions or service crashes in direct dual-write scenarios. The polling of the Outbox message table by the message relay service, combined with the mechanism of exclusive locks and skipping locked records, enables mutual exclusion in the retrieval of different messages to be sent under concurrent pulls from multiple instances, avoiding duplicate sending. The precise control of the next retry time field for sending timing and the idempotent query based on message identifiers on the consumer side ensure that failure retries are executed at a controllable backoff rhythm. The at least one delivery semantic in downstream processing is converged to exactly one, forming a stable engineering safety net between business availability and consistency.

[0007] Optionally, the message relay service includes multiple worker instances running in parallel, and the Outbox message table also contains an aggregation type field corresponding to the message to be sent; in S2, when each worker instance performs polling on the Outbox message table, it obtains the message to be sent belonging to itself based on the aggregation type field according to the preset partitioning rules.

[0008] By adopting the above technical solution, multiple worker instances can be divided into scanning ranges according to business aggregation dimensions, avoiding scanning overlap or competition between instances and improving the overall horizontal scalability of the message relay service.

[0009] Optionally, the preset polling period is an adaptive polling period; in S2, after each round of polling is completed, the message relay service dynamically adjusts the adaptive polling period of the next round of polling according to the number of messages to be sent obtained in this round of polling. When the number of messages to be sent obtained in this round is greater than the preset upper load threshold, the adaptive polling period of the next round is shortened; when the number of messages to be sent obtained in this round is less than the preset lower load threshold, the adaptive polling period of the next round is extended.

[0010] By adopting the above technical solution, the polling frequency can be automatically scaled up and down with the system load. Under high load, the polling interval is shortened to accelerate message delivery, and under low load, the polling interval is extended to reduce the query pressure on the business database.

[0011] Optionally, the next retry scheduling in S5 includes sub-steps S51-S52: S51. The backoff duration is obtained by multiplying the preset basic backoff duration by the backoff coefficient corresponding to the retry count field through the message relay service.

[0012] S52. Use the sum of the current time and the backoff duration as the new value for the next retry time field via the message relay service.

[0013] The backoff coefficient increases exponentially with the number of retries.

[0014] By adopting the above technical solution, the retry interval of the failure message increases exponentially with the number of retries, avoiding excessive impact on downstream services and allowing sufficient time for downstream services to recover on their own.

[0015] Optionally, S2 further includes sub-steps S201-S202: S201. Verify through the message relay service whether the number of retry attempts for the message to be sent is less than the preset maximum number of retry attempts.

[0016] S202. Exclude messages to be sent that have a retry count field not less than the preset retry count limit from the execution object of S3 through the message relay service.

[0017] By adopting the above technical solution, messages that have exceeded the retry limit will no longer enter the sending process, thus avoiding the invalidation of permanently failed messages and providing an accurate input set for subsequent dead letter processing.

[0018] Optionally, S4 includes sub-steps S401-S403: S401. Start a database transaction through the consumer end, and perform an insert operation on the consumer record table with the message identifier as the unique constraint within the database transaction.

[0019] S402. The consumer responds to the failure of the insert operation due to a unique constraint conflict by rolling back the database transaction and terminating this S4.

[0020] S403. Upon successful insertion response from the consumer, downstream processing is executed within the database transaction, and the database transaction is committed.

[0021] By adopting the above technical solution, the unique constraint insertion operation of the message identifier is placed in the downstream processing before the consumer side, so that the consumer side avoids the competition window between query and insertion in the query-then-insert implementation in the concurrent scenario. The downstream processing is only executed once when the message identifier appears for the first time.

[0022] Optionally, the Outbox message table is partitioned and stored according to the creation time field; the message relay service periodically cleans up the pending messages in the Outbox message table whose status field is "sent" according to a preset retention period.

[0023] By adopting the above technical solutions, the size of the Outbox message table is kept within a controllable range, and the query performance does not degrade as historical messages accumulate.

[0024] Optionally, in S1, when the business application writes the message to be sent to the Outbox message table, it also writes the content fingerprint of the content carried by the message to be sent to the Outbox message table; in S3, the message relay service sends the content fingerprint and the message to be sent together to the message queue; S4 includes sub-steps S411-S414: S411. Query the consumption record table based on a combination of message identifier and content fingerprint from the consumer end.

[0025] S412. If a message identifier exists in the consumption record table, skip the current downstream processing.

[0026] S413. If the consumer responds to a record in the consumption record table that does not contain a message identifier but has the same content fingerprint, skip the current downstream processing and write the message identifier into the consumption record table.

[0027] S414. If the consumer responds to the fact that there is no record in the consumption record table that is the same as the message identifier and the content fingerprint, perform downstream processing of the message to be sent and write the message identifier and the content fingerprint into the consumption record table.

[0028] By adopting the above technical solution, idempotency protection is extended from message identifiers to content fingerprints. Even if multiple messages with different message identifiers but the same content are generated due to repeated triggering on the business side, the consumer can still identify them and perform downstream processing only once.

[0029] Optionally, the message relay service maintains the scanned time pointer; S2 includes sub-steps S211-S212: S211. Perform a scan of the Outbox message table using the message relay service with a new write window and a retry expiration window. The scope of the new write window is records whose creation time field is later than the scanned time pointer and whose status field is pending. The scope of the retry expiration window is records whose retry count field is greater than zero, whose next retry time field is not later than the current time, and whose status field is pending.

[0030] S212. After the scan of the newly added write window is completed, update the scan time pointer to the end time of this round of scan via the message relay service.

[0031] By adopting the above technical solution, each round of scanning only processes two types of incremental records: newly added writes and retry expired records, avoiding repeated scanning of the entire table and significantly reducing the query load of the message relay service on the business database.

[0032] Optionally, the Outbox message table also includes a causal dependency identifier field corresponding to the message to be sent, which records the message identifier of the upstream message that the message to be sent depends on; S3 includes sub-steps S31-S32: S31. Verify through the message relay service whether the causal dependency identifier field of the message to be sent is empty, or whether the status field of the upstream message recorded in the causal dependency identifier field in the Outbox message table is "sent".

[0033] S32. The message to be sent is sent to the message queue only when the causal dependency identifier field is empty or the upstream message has been sent, and the status field is updated to sent when the sending is successful.

[0034] By adopting the above technical solution, messages with causal dependencies are delivered to the message queue in the order that upstream messages are sent first, thus avoiding downstream messages receiving a logically disordered sequence of messages.

[0035] Optionally, messages to be sent that are migrated to the dead letter table carry a retry history field in the dead letter table. The retry history field records the occurrence time, failure reason and error stack of each retry of the message to be sent in a structured form.

[0036] By adopting the above technical solution, dead-letter messages retain a complete failure tracing link, which facilitates post-event analysis and attribution of failure modes by operation and maintenance personnel.

[0037] Optionally, manual retries can be supported for messages to be sent in the dead letter table. After fault repair of the dead letter record, the core fields of the dead letter record can be re-inserted into the Outbox message table and the status field, retry count field and next retry time field can be reset.

[0038] By adopting the above technical solutions, dead-letter messages can still re-enter the normal delivery process after the root cause of the failure is resolved, thus avoiding the permanent decoupling of business data.

[0039] Optionally, the message relay service periodically collects runtime metrics from the Outbox message table. These runtime metrics include the number of records with a status field of pending sending, the hourly rolling count of records with a status field of sent, the number of records with a status field of pending sending and a creation time field that is more than a preset threshold from the current time, and the average delay in updating the status field after successful sending.

[0040] By adopting the above technical solutions, the operating status of Outbox mode can be observed in real time, providing a quantitative basis for fault location and capacity planning.

[0041] Optionally, the message relay service applies preset alarm rules to the collected runtime metrics. The preset alarm rules include triggering a backlog alarm when the status field is "pending to send" and the creation time field is more than the preset backlog duration limit, and triggering a failure rate alarm when the proportion of failed sending times to the total number of sending times exceeds the preset failure rate limit.

[0042] By adopting the above technical solutions, abnormal states of the Outbox message table can be proactively detected, preventing the accumulation of faults that have already caused a large-scale impact by the time the business becomes aware of them.

[0043] Optionally, when the message relay service sends a message to be sent to the message queue, it adds a message identifier and a creation timestamp as auxiliary metadata to the message attributes.

[0044] By adopting the above technical solution, the consumer side can perform idempotent deduplication and timing determination based on message attributes, thereby reducing the overhead of message body parsing.

[0045] Optionally, the consumption record table can be indexed using a hash index on the message identifier field.

[0046] By adopting the above technical solution, the consumer-side equality query based on message identifier can directly hit the hash bucket, and the query latency is lower than that of the B-tree index.

[0047] Optionally, the Outbox message table can be partially indexed for records whose status field is "to be sent".

[0048] By adopting the above technical solution, the index size is only related to the number of messages to be sent rather than the total number of tables, thus optimizing both scanning efficiency and index maintenance overhead.

[0049] Optionally, when the message relay service in S2 performs polling on the Outbox message table, the number of messages pulled in a single batch is an adaptive batch size. The message relay service dynamically adjusts the batch size of the next round based on the success rate of sending messages pulled in the previous round. When the success rate of sending messages is lower than the preset batch shrinking threshold, the batch size is shrunk; when the success rate of sending messages is higher than the preset batch expanding threshold, the batch size is expanded.

[0050] By adopting the above technical solution, the batch size can be adaptively adjusted according to the fluctuation of downstream processing capacity, avoiding the concentrated impact of a large number of messages during a brief downstream failure, which would exacerbate the failure.

[0051] Optionally, the consumption record table is physically partitioned by the consumer identifier field, and the query and write operations of each consumer instance only access the corresponding physical partition.

[0052] By adopting the above technical solutions, the consumption records of different consumers are physically isolated, the single partition scan depth is less than that of a full table scan, and the lock conflicts in concurrent scenarios are significantly reduced.

[0053] Optionally, as an alternative implementation to the exclusive lock mechanism in S2, the message relay service in S2 includes multiple worker instances running in parallel. Each worker instance buckets the messages to be sent in the Outbox message table according to the consistent hashing algorithm. Each worker instance exclusively occupies a group of buckets and only scans the messages to be sent belonging to its own bucket, without adding an exclusive lock to the query results.

[0054] By adopting the above technical solution, the concurrent scanning of the Outbox message table by multiple worker instances completely eliminates the overhead of exclusive locks, serving as an equivalent alternative implementation method to the exclusive lock mechanism.

[0055] Optionally, the preset retention period of the consumption record table is not shorter than the upper limit of the lifespan of messages to be sent in the message queue.

[0056] By adopting the above technical solution, the volume of the consumption record table has a clear engineering upper limit, and the idempotency protection will not fail due to premature cleanup.

[0057] Optionally, in S1, business applications can automatically generate messages to be sent and write them to the Outbox message table through object-relational mapping hooks triggered by write operations on business data tables, replacing explicit calls to the Outbox write operation in the business code.

[0058] By adopting the above technical solutions, the intrusion of the Outbox pattern on business code is significantly reduced, and the coupling between business logic and message delivery mechanism is decoupled.

[0059] Optionally, the message relay service in S2 combines polling of the Outbox message table with the change notification mechanism of the business database. Whenever the business database issues a change notification for a new message to be sent, a scan is triggered immediately, while maintaining a preset polling period as a fallback mechanism.

[0060] By adopting the above technical solution, the delivery delay of newly written messages is made close to real-time, and the final reliability can still be guaranteed by polling when the database change notification fails.

[0061] Optionally, this method also includes a failure mode handling mechanism based on message circuit breakers: the Outbox message table also contains an event type field corresponding to the message to be sent, and the business database also has a circuit breaker status table, which is used to persist the cumulative number of failures, circuit breaker flag, and circuit breaker end time for each event type; in S2, before the message relay service executes S3 for each message to be sent, it queries the circuit breaker flag of the event type corresponding to the message to be sent in the circuit breaker status table. If the circuit breaker flag is active and the current time is earlier than the circuit breaker end time, it skips this round of S3 and retains the status field of the message to be sent as "to be sent"; In step 5, after each failed message transmission, the cumulative failure count for the event type corresponding to the message to be sent is updated in the circuit breaker status table. In response to the cumulative failure count reaching the preset circuit breaker threshold, the circuit breaker flag is set to active and the circuit breaker end time is set according to the preset circuit breaker duration. The circuit breaker status table is located in the business database, and all worker instances share the same circuit breaker status table. After the circuit breaker end time is reached, the message relay service performs a trial transmission of the next message to be sent for the event type. If the transmission is successful, the circuit breaker flag is set to inactive and the cumulative failure count is cleared to zero. If the transmission fails, the circuit breaker end time is reset according to the preset circuit breaker duration.

[0062] By adopting the above technical solutions, the failure flood of downstream faults for specific event types can be quickly isolated at the Outbox layer, avoiding continuous impact on message queues and downstream services caused by continuous retries. At the same time, the circuit breaker status is shared among multiple workers to avoid distortion from the perspective of a single worker.

[0063] Optionally, the specific form of multiplying the preset base backoff time and the backoff coefficient in S51 to obtain the backoff time includes: multiplying the preset base backoff time by 2 raised to the power of the retry count field to obtain the backoff time.

[0064] This specific backoff method allows the next retry interval for a failure message to increase exponentially with the number of retries, making it a typical implementation of the backoff strategy.

[0065] Optionally, the content fingerprint is calculated from the message content using a cryptographic hash algorithm, which may include either SHA-256 or MD5.

[0066] By adopting the above technical solution, the content fingerprint is made anti-collision and deterministic. The same content will inevitably produce the same fingerprint, and the probability of collision between different content is negligible.

[0067] Optionally, the scan depth of the newly added write window is limited to the creation time field being later than the scanned time pointer and not exceeding the preset new write window depth from the current time. The scan depth of the retry expiration window is limited to the next retry time field being not exceeding the preset retry expiration window depth from the current time. The specific implementation of updating the scanned time pointer to the end time of this round of scanning in S22 is as follows: the message relay service completes two actions of scanning and reading and pointer updating in the transaction. The new value of the scanned time pointer is the difference between the start time of this round of scanning and the preset safety margin. The preset safety margin is greater than the maximum transaction commit delay of the business database.

[0068] By introducing an upper limit on window depth and a safety margin, the workload of a single round of dual-window scanning is bounded, and messages that are still in the transaction commit phase when the current round of scanning starts will not be missed by pointer advancement.

[0069] Optionally, the preset polling period is 5 seconds, the preset maximum number of retries is 5, the preset basic backoff time is 60 seconds, the preset upper load threshold is 80% of the maximum number of query messages in a single batch in this round, the preset lower load threshold is 5% of the maximum number of query messages in a single batch in this round, the preset retention period is 7 days, the preset circuit breaker threshold is 10 times, and the preset circuit breaker duration is 60 seconds.

[0070] By providing the above typical parameter values, those skilled in the art can directly implement them. The parameter values ​​have been verified to work in typical business scenarios such as medical image quality control.

[0071] In summary, this application includes at least one of the following beneficial technical effects: 1. By merging the writing of business data and the persistence of messages to be sent into the same local transaction, and combining the polling delivery of the message relay service with the idempotent deduplication based on message identifiers at the consumer end, the business system can achieve reliable delivery and eventual consistency of downstream messages without relying on the two-phase commit protocol.

[0072] 2. By combining exclusive locks with mechanisms such as skipping locked records, aggregated type partition scanning, adaptive polling cycle adjustment, exponential backoff retry scheduling, and consumer-side unique constraint fallback, this method achieves production-ready robustness in multiple dimensions, including concurrent deployment of multiple instances, downstream fault tolerance, and consumer-side deduplication.

[0073] 3. Through dual idempotent protection of message identifiers and content fingerprints, dual-window incremental scanning, and the synergy of causal dependency identifier fields, this method has further safeguards against deep engineering risks such as repeated triggering on the business side, maintenance of scanning efficiency, and guarantee of message order. Attached Figure Description

[0074] Figure 1 This is a schematic diagram illustrating the application environment of a distributed transaction consistency guarantee method based on the Outbox pattern in one embodiment of this application.

[0075] Figure 2 This is a flowchart illustrating the overall process of a distributed transaction consistency guarantee method based on the Outbox pattern in one embodiment of this application.

[0076] Figure 3 This is a flowchart of the S2 message relay service polling and fetching process in one embodiment of this application.

[0077] Figure 4 This is a flowchart of the S4 consumer-side idempotent processing in one embodiment of this application.

[0078] Figure 5 This is a flowchart of S5 failure retry and dead letter handling in one embodiment of this application. Detailed Implementation

[0079] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0080] This application provides a distributed transaction consistency guarantee method based on the Outbox pattern. This method is applicable to distributed system scenarios where, after a business operation is completed, notifications or processing messages need to be asynchronously delivered to multiple downstream services. Taking a medical image quality control system as an example, the business application receives DICOM images and executes a quality control algorithm. After quality control is completed, it needs to reliably notify multiple downstream modules such as image archiving services, clinician notification services, and workload statistics services. In such scenarios, the local transaction of the business database and the message delivery of the message queue belong to two independent resource domains, and coordination between the two is crucial to ensuring consistency between the business and downstream processing. This application merges the writing of business data and the persistence of messages to be sent into the same local transaction, combined with the polling delivery of the message relay service and the idempotent deduplication based on message identifiers at the consumer end, enabling the business system to achieve reliable delivery and eventual consistency of downstream messages without relying on the two-phase commit protocol. The following, in conjunction with the appendix... Figure 1 To be continued Figure 5 Each step is explained in detail.

[0081] like Figure 1As shown in the figure, the distributed transaction consistency guarantee method based on the Outbox pattern in this application is applied to a medical image quality control system, which includes core components such as business applications, business databases, message relay services, message queues and consumer terminals.

[0082] Business applications receive medical image quality control requests from external parties, such as performing automated quality control judgments on a single batch of DICOM images.

[0083] The business database stores business data tables and Outbox message tables. The business data tables are used to persist quality control results, and the Outbox message tables are used to persist message records to be sent downstream.

[0084] The message relay service is a stand-alone background process that pulls messages from the Outbox message table and sends them to the message queue.

[0085] Message queues are responsible for message distribution and downstream subscription.

[0086] The consumer side consists of downstream services that subscribe to message queues, such as image archiving services, clinician notification services, and workload statistics services.

[0087] like Figure 2 As shown, this method includes five main steps, S1-S5. Specifically, S1 corresponds to the dual-write phase of the business transaction. After receiving the quality control request, the business application simultaneously writes the quality control result to the business data table and the message to be sent triggered by the business request to the Outbox message table in the local transaction of the business database, and then commits the local transaction. The message to be sent is, for example, the "quality control completion notification" event. The Outbox message table persists the message to be sent, as well as the corresponding status field, retry count field, next retry time field, and creation time field. The status field has two values: pending and sent. The retry count field records the cumulative number of failed transmissions. The next retry time field records the time when the message is next allowed to be picked up by the message relay service. The creation time field records the time when the message is written to the Outbox message table.

[0088] like Figure 3 As shown, S2 corresponds to the polling and fetching phase of the message relay service. The message relay service polls the Outbox message table at a preset polling period, acquires an exclusive lock on the query results, skips records already locked by other message relay service instances, and retrieves messages with a status field of "pending to be sent" and a next retry time field no later than the current time. The combination of exclusive locking and skipping locked records allows multiple concurrently running message relay service instances to safely fetch different messages to be sent simultaneously, avoiding duplicate sending.

[0089] S3 corresponds to the message sending phase. The message relay service will send the retrieved message to the message queue and update the status field of the message to be sent in the Outbox message table to "sent" when the sending is successful.

[0090] like Figure 4 As shown, S4 corresponds to the idempotent consumption phase on the consumer side. After receiving a message to be sent from the message queue, the consumer queries the consumption record table based on the message identifier carried in the message to be sent. If the message identifier does not exist in the consumption record table, the consumer executes the downstream processing corresponding to the message to be sent and writes the message identifier into the consumption record table. The specific content of the downstream processing depends on the business scenario. For example, the image archiving service performs image storage operations, the clinical physician notification service pushes notification messages to the doctor's workstation, and the workload statistics service updates the doctor's workload count.

[0091] like Figure 5 As shown, S5 corresponds to the failure retry and dead letter handling phase. In response to the failure to send the message to be sent in S3, the message relay service increments the retry count field of the message and performs the next retry scheduling for the message. The scheduling result is reflected in the updated value of the next retry time field. When the retry count field reaches the preset retry limit, the message relay service migrates the message to be sent to the dead letter table and stops subsequent retries. The two actions of incrementing the retry count field and migrating to the dead letter table in S5 can be implemented synchronously or asynchronously. That is, they can be completed together in the failure handling process of S3, or they can be completed in an independent processing cycle decoupled from S2 and S3.

[0092] Therefore, the overall synergistic effect of this method is reflected in three aspects: local transaction dual writing eliminates the risk of message loss due to network interruption or service crash in direct dual writing scenarios; the exclusive lock and skip locked record mechanism of message relay service ensure that messages are not sent repeatedly under multi-instance concurrent deployment; and the idempotent query based on message identifier on the consumer end converges the at least one delivery semantic of the message queue into exactly one execution of downstream processing.

[0093] In some embodiments, when the business application in S1 writes the message to be sent to the Outbox message table, it also writes the content fingerprint of the content carried by the message to the Outbox message table. The content fingerprint is a deterministic mapping result of the business payload carried by the message. Two messages with the same business payload will necessarily have the same content fingerprint, and the probability of two messages with different business payloads having the same content fingerprint is negligible. In the medical image quality control scenario, for example, if a doctor clicks the "Submit Quality Control" button twice due to UI lag, the business application will generate two messages to be sent with different message identifiers but the same content fingerprint. Both messages are written to the Outbox message table through S1, waiting for the consumer to identify and deduplicate them in the subsequent dual idempotency mechanism.

[0094] As another implementation of writing to the Outbox message table by business applications in S1, in some embodiments, the business application automatically generates messages to be sent and writes them to the Outbox message table through an object-relational mapping hook triggered by a write operation to the business data table, replacing the explicit call to the Outbox write operation in the business code. The object-relational mapping hook can be a hook function before the database transaction is committed. When the hook function detects a write to the business data table, it generates the corresponding message to be sent according to a preset event type mapping rule and inserts the message to be sent into the same transaction in the Outbox message table. This implementation significantly reduces the intrusion of the Outbox mode on the business code, allowing business developers to focus only on the business logic itself, thus decoupling the business logic from the message delivery mechanism.

[0095] In some embodiments, the message relay service includes multiple worker instances running in parallel, and the Outbox message table also contains an aggregation type field corresponding to the message to be sent. Each worker instance is an independent container process, and each worker instance independently connects to the business database and the message queue. The value of the aggregation type field reflects the business aggregation dimension to which the message to be sent belongs; for example, the value could be a prefix field identifying the research instance, or a combination of the event type and its business domain.

[0096] In S2, when each worker instance polls the Outbox message table, it retrieves the messages to be sent belonging to it based on a predefined partitioning rule using the aggregate type field. The predefined partitioning rule can be a mapping rule where the hash value of the aggregate type field is modulo 4 (the total number of worker instances). For example, when deploying 4 worker instances, the first worker instance only pulls messages to be sent whose aggregate type field hash value modulo 4 is 0, the second worker instance only pulls messages whose modulo 4 hash value is 1, and so on.

[0097] When the number of worker instances deployed changes dynamically, the preset partitioning rules are updated accordingly. Each worker instance redetermines its assigned range of messages to be sent based on the new partitioning rules. For example, when expanding from 4 worker instances to 8 worker instances, the modulo base of the partitioning rules changes from 4 to 8, and the new attribution relationships are uniformly issued by the configuration center or service registry.

[0098] Compared to the method of performing a full table scan of the Outbox message table with a single worker instance and an exclusive lock mechanism, the partitioning mechanism based on the aggregation type field naturally divides the messages to be sent from different business aggregations into multiple independent scan ranges. The scan depth of a single worker instance is only 1 / N of the entire table, where N is the total number of worker instances. Both the scan load and lock contention are significantly reduced.

[0099] In some embodiments, the preset polling period is an adaptive polling period. In S2, after each round of polling is completed, the message relay service dynamically adjusts the adaptive polling period of the next round of polling based on the number of messages to be sent obtained in this round of polling. When the number of messages to be sent obtained in this round is greater than the preset upper load threshold, the adaptive polling period of the next round is shortened; when the number of messages to be sent obtained in this round is less than the preset lower load threshold, the adaptive polling period of the next round is extended. For example, if the initial value of the preset polling period is 5 seconds, and the number of messages fetched in this round reaches 80% of the upper limit of the number of messages queried in a single batch in this round, i.e., 80 messages, the polling period of the next round is shortened to 2.5 seconds to accelerate message delivery; if the number of messages fetched in this round is less than 5% of the upper limit of the number of messages queried in a single batch in this round, i.e., 5 messages, the polling period of the next round is extended to 10 seconds to reduce the empty polling pressure on the business database.

[0100] In some embodiments, as a further filtering mechanism after the message to be sent is retrieved in S2, S2 further includes sub-steps S201-S202.

[0101] S201. Verify through the message relay service whether the number of retry attempts for the message to be sent is less than the preset maximum number of retry attempts.

[0102] S202. Exclude messages to be sent that have a retry count field not less than the preset retry count limit from the execution object of S3 through the message relay service.

[0103] For example, in this round, S2 retrieves 50 messages with a status field indicating they are pending transmission. Of these, 48 messages have a retry count less than the preset retry limit of 5, and 2 messages have a retry count equal to 5. After verification by S201 and filtering by S202, only 48 messages enter the S3 transmission process. The two filtered-out messages retain their status field as pending transmission and are left to be migrated to the dead-letter table during the failure handling process of S5. S201-S202 serve as a fallback protection mechanism for S5 dead-letter migration in asynchronous S5 implementation scenarios. That is, when S5 fails to migrate messages that have reached the retry limit out of the Outbox message table in a timely manner due to execution delays, node crash recovery, or migration operation failures, S201 and S202 immediately identify and exclude the message from S3 upon receiving it, preventing it from being resent. Simultaneously, S5 completes the final migration in the next failure handling cycle. This filtering mechanism prevents messages that have exceeded the retry limit from continuously occupying the message relay service's transmission channel.

[0104] In some embodiments, as another implementation of polling scan in S2, the message relay service maintains a scanned time pointer, and S2 includes sub-steps S211-S212.

[0105] S211. Perform a scan of the Outbox message table using the message relay service with a new write window and a retry expiration window. The new write window is for records whose creation time field is later than the scanned time pointer and whose status field is pending. The retry expiration window is for records whose retry count field is greater than zero, whose next retry time field is no later than the current time, and whose status field is pending.

[0106] S212. After the scan of the newly added write window is completed, update the scan time pointer to the end time of this round of scan via the message relay service.

[0107] For example, if the start time of this round S2 is t0 and the end time of this round of scanning is t1 (t1 = t0 + 200 milliseconds), the current value of the scanned time pointer is t0 minus 5 seconds. In S211, the new write window scans records whose creation time field is within the interval "t0 minus 5 seconds" to t1 and whose status field is "pending to be sent", resulting in 45 new write messages; the retry expiration window scans records whose retry count field is greater than zero, whose next retry time field is no later than t1, and whose status field is "pending to be sent", resulting in 5 expired retry messages. The two windows are merged to obtain 50 messages to be sent in this round, which are then sent in the S3 sending process. In S212, after the new write window scan is completed, the scanned time pointer is updated to t1, and the starting point of the new write window for the next round of scanning is t1.

[0108] Compared to performing a full table scan on the Outbox message table, which scans all records with a status field indicating they are pending transmission in each round, the dual-window scan only processes two types of incremental records per round: newly written records and expired retries. The scan depth is only related to the total number of newly added and expired messages in a single round, and is independent of the total number of records in the entire table, significantly reducing the query load on the business database from the message relay service. The dual-window scan mechanism relies on the correct advancement protocol of the scanned time pointer to ensure that the "later than pointer" condition of the newly added write window does not miss messages pending transmission that are in the middle of a transaction commit.

[0109] In some embodiments, the scan depth of the new write window is limited to a creation time field that is later than the scanned time pointer and no more than a preset new write window depth from the current time. The scan depth of the retry expiration window is limited to a next retry time field that is no more than a preset retry expiration window depth from the current time. For example, if the preset new write window depth is 1 minute and the preset retry expiration window depth is 30 minutes, then the maximum data range of a single round of S211 scan has a clear engineering upper limit.

[0110] In S212, updating the scanned time pointer to the end time of the current scan is implemented as follows: The message relay service completes two actions within a transaction: scanning and reading, and updating the pointer. The new value of the scanned time pointer is the difference between the start time of the current scan and a preset safety margin, which is greater than the maximum transaction commit delay of the business database. For example, if the maximum transaction commit delay of the business database is 100 milliseconds, then the preset safety margin is 200 milliseconds. When the start time of the current scan is t0, the scanned time pointer advances to "t0 minus 200 milliseconds". This safety margin ensures that messages that are still in the middle of a transaction commit when the current scan starts, i.e., messages whose creation time field has not yet been written to disk, will not be missed by the pointer advancement and will still be within the range of the new write window in the next scan.

[0111] In some embodiments, the number of messages pulled by the message relay service when polling the Outbox message table in S2 is an adaptive batch size. The message relay service dynamically adjusts the batch size of the next round according to the success rate of sending messages in the previous round: shrinking the batch size when the success rate is lower than the preset batch shrinking threshold, and expanding the batch size when the success rate is higher than the preset batch expanding threshold.

[0112] As another implementation of the exclusive locking mechanism in S2, in some embodiments, the message relay service includes multiple worker instances running in parallel. Each worker instance buckets the messages to be sent in the Outbox message table according to a consistent hashing algorithm. Each worker instance exclusively owns a set of buckets and only scans the messages to be sent belonging to its own bucket, without adding an exclusive lock to the query results. The input to the consistent hashing algorithm can be the aggregation type field or message identifier of the message to be sent, and the output is the bucket number. For example, if 4 worker instances are deployed and 256 buckets are set, then each worker instance exclusively owns 64 buckets. When the number of worker instances is adjusted from 4 to 8, only about half of the buckets need to be migrated to the new ownership relationship, avoiding the overhead of changing the mapping of all buckets under the modulo partitioning rule. This implementation does not rely on the exclusive locking mechanism of the business database, eliminating the lock contention overhead between multiple worker instances.

[0113] In some embodiments, the polling of the Outbox message table by the message relay service in S2 is combined with the change notification mechanism of the business database. Whenever the business database issues a change notification for a newly added message to be sent, a scan is immediately triggered, while maintaining a preset polling period as a fallback mechanism. The change notification mechanism makes the delivery delay of newly written messages close to real-time, and the preset polling period as a fallback mechanism ensures that messages can still be retrieved eventually if the change notification fails.

[0114] In some embodiments, the Outbox message table creates a partial index for records where the status field is "to be sent". The partial index contains only index entries for records where the status field is "to be sent", and the index size is only related to the number of messages to be sent, not the size of the entire table, thus optimizing both scan efficiency and index maintenance overhead.

[0115] Following the content fingerprint written in S1, the message relay service in S3 sends the content fingerprint along with the message to be sent to the message queue. The content fingerprint is attached to the message body as a message attribute and sent together, for the consumer to use in subsequent double idempotency queries.

[0116] In some embodiments, the Outbox message table also includes a causal dependency identifier field corresponding to the message to be sent. This field records the message identifier of the upstream message that the message to be sent depends on. The causal dependency identifier field can be empty, indicating that the message does not depend on any upstream message, or it can point to the message identifier of another upstream message in the Outbox message table. For example, in an order fulfillment scenario, if an order payment notification message depends on an order creation notification message, then the causal dependency identifier field of the order payment notification message records the message identifier of the order creation notification message.

[0117] S3 includes sub-steps S31-S32.

[0118] S31. Verify through the message relay service whether the causal dependency identifier field of the message to be sent is empty, or whether the status field of the upstream message recorded in the causal dependency identifier field in the Outbox message table is "sent".

[0119] S32. The message to be sent is sent to the message queue only when the causal dependency identifier field is empty or the upstream message has been sent, and the status field is updated to sent when the sending is successful.

[0120] When the status field of an upstream message in the Outbox message table is still "pending," S32 temporarily refrains from sending the currently pending message. The status field of the currently pending message remains "pending," to be retrieved and determined again in the next round by S2. This mechanism ensures that messages with causal dependencies are delivered to the message queue in the order of upstream priority, preventing downstream messages from receiving a logically disordered message sequence.

[0121] In some embodiments, when the message relay service sends a message to be sent to the message queue, it appends a message identifier and a creation timestamp as auxiliary metadata to the message attributes. The consumer performs idempotent deduplication and timing determination based on the message attributes, reducing the overhead of message body parsing.

[0122] In some embodiments, as an implementation of consumption idempotency in S4, S4 includes sub-steps S401-S403.

[0123] S401. Start a database transaction through the consumer end, and perform an insert operation on the consumer record table with the message identifier as the unique constraint within the database transaction.

[0124] S402. The consumer responds to the failure of the insert operation due to a unique constraint conflict by rolling back the database transaction and terminating this S4.

[0125] S403. Upon successful insertion response from the consumer, downstream processing is executed within the database transaction, and the database transaction is committed.

[0126] A unique constraint is established in the message identifier field of the consumption record table. The unique constraint is implemented at the underlying level by the primary key constraint, unique index, or equivalence mechanism of the message identifier field.

[0127] Taking a single message consumption as an example. Suppose that the same message, identified as "msg-001," is pushed to the consumer twice due to the at least one delivery semantic of the message queue. First push: In S401, the consumer starts a database transaction and inserts "msg-001" into the consumption record table, which succeeds. In S403, the consumer performs downstream processing within the database transaction, such as calling an image archiving service to store DICOM images. After the downstream processing is complete, the database transaction is committed. At this point, the consumption record table contains a record with "msg-001." Second push: In S41, the consumer starts a new database transaction and inserts "msg-001" into the consumption record table, but fails due to a unique constraint conflict. In S402, the consumer rolls back the database transaction and terminates this S4 session; the downstream processing is not executed.

[0128] The design in S401, which places the unique constraint insertion operation of the message identifier before downstream processing, avoids the competition window between query and insertion in concurrent scenarios, as seen in the "query-then-insert" implementation. In the "query-then-insert" implementation, two concurrent consumer instances might simultaneously find that "msg-001" does not exist in the consumption record table, then each executes downstream processing and attempts to write "msg-001," resulting in duplicate execution of downstream processing. However, with the aforementioned combination of sub-steps, regardless of the execution order of concurrent instances, downstream processing is only executed once in the instance where the unique constraint insertion operation succeeds.

[0129] As another implementation of consumption idempotency in S4, in some embodiments, the content fingerprint written in S1 is sent together with that in S3, and S4 includes sub-steps S411-S414.

[0130] S411. Query the consumption record table based on a combination of message identifier and content fingerprint from the consumer end.

[0131] S412. If a message identifier exists in the consumption record table, skip the current downstream processing.

[0132] S413. If the consumer responds to a record in the consumption record table that does not contain a message identifier but has the same content fingerprint, skip the current downstream processing and write the message identifier into the consumption record table.

[0133] S414. If the consumer responds to the fact that there is no record in the consumption record table that is the same as the message identifier and the content fingerprint, perform downstream processing of the message to be sent and write the message identifier and the content fingerprint into the consumption record table.

[0134] This implementation extends idempotency protection from message identifiers to content fingerprints, making it particularly suitable for scenarios where multiple messages with different identifiers but the same service payload are generated on the service side due to UI lag, network retransmission, human error, or other reasons. Taking medical image quality control as an example, there are two typical scenarios.

[0135] Scenario 1: After a doctor submits a quality control request, the message queue delivers "msg-001" twice due to a service restart. Both pushes carry the message identifier "msg-001" and the content fingerprint "fp-A". During the first push, the consumption record table does not contain this entry, so S414 performs downstream processing and writes ("msg-001", "fp-A"). During the second push, the consumption record table already contains the message identifier "msg-001", so S412 skips this downstream processing.

[0136] Scenario 2: A doctor clicks the "Submit for Quality Control" button twice due to UI lag. The application generates two messages to be sent, each carrying a message identifier "msg-002" and "msg-003" respectively. However, because the workload is the same, both messages have the content fingerprint "fp-B". When the first message is pushed, there is no such entry in the consumption record table, so it enters S414 for downstream processing and writes ("msg-002", "fp-B"). When the second message is pushed, although the message identifier "msg-003" does not exist in the consumption record table, there is a record with the content fingerprint "fp-B", which is the "msg-002" record from the first message. S413 skips downstream processing and only writes "msg-003" to the consumption record table.

[0137] In S413, “msg-003” is written to the consumption record table but not the content fingerprint is written again. The purpose is to avoid misjudging it as a new message when the same message identifier “msg-003” arrives again. When pushing again, S411 query will hit the branch where “msg-003” already exists and enter S412 to skip downstream processing.

[0138] Compared to single-layer idempotency protection that relies solely on message identifiers, the dual idempotency mechanism extends the boundary of idempotency protection from the message queue protocol layer to application layer bug tolerance, specifically covering the scenario of "repeated triggering of business-side bugs" that cannot be defended against by single message identifier idempotency.

[0139] In some embodiments, the content fingerprint is calculated from the message-carrying content using a cryptographic hash algorithm, which includes either the SHA-256 algorithm or the MD5 algorithm.

[0140] In some embodiments, the preset retention period of the consumption record table is not shorter than the upper limit of the message lifespan of the message queue to be sent. For example, if the upper limit of the message lifespan of the message queue is 24 hours, then the preset retention period of the consumption record table is not shorter than 24 hours; in actual engineering, 48 hours can be used as a safety margin. Messages discarded by the message queue can no longer be delivered to the consumer, and the consumption record table does not need to retain corresponding entries for message identifiers that are no longer reachable. This constraint ensures that the size of the consumption record table has a clear engineering upper bound, i.e., it does not exceed the upper limit of the message queue lifespan multiplied by the average message generation rate, and it also prevents premature cleanup from causing the idempotency protection of messages that have been retained by the message queue but have been cleaned up by the consumption record table to fail.

[0141] In some embodiments, the consumption record table creates a hash index on the message identifier field. Hash indexes provide constant time complexity in equality query scenarios, with lower latency than B-tree indexes.

[0142] In some embodiments, the consumption record table is physically partitioned by the consumer identifier field, and each consumer instance's query and write operations only access the corresponding physical partition. The scan depth of a single partition is lower than that of the entire table, significantly reducing lock contention between different consumer instances in concurrent scenarios.

[0143] In some embodiments, the next retry scheduling in S5 includes sub-steps S51-S52.

[0144] S51. The backoff duration is obtained by multiplying the preset basic backoff duration by the backoff coefficient corresponding to the retry count field through the message relay service.

[0145] S52. Use the sum of the current time and the backoff duration as the new value for the next retry time field via the message relay service.

[0146] The backoff coefficient increases exponentially with the number of retries. For example, when the number of retries is 1, the backoff coefficient is 2; when the number of retries is 2, the backoff coefficient is 4; and when the number of retries is 3, the backoff coefficient is 8. This backoff scheduling makes the next retry interval of a failure message increase exponentially with the number of retries, avoiding excessive impact on downstream services that have failed, while allowing sufficient time for downstream services to recover on their own.

[0147] In some embodiments, the specific form of multiplying the preset basic backoff time and the backoff coefficient in S51 to obtain the backoff time includes: multiplying the preset basic backoff time by 2 raised to the power of the retry count field to obtain the backoff time. For example, if the preset basic backoff time is 60 seconds, and the retry count field is 0, 1, 2, or 3, the corresponding backoff times are 60 seconds, 120 seconds, 240 seconds, and 480 seconds, respectively.

[0148] In some embodiments, this method further includes a failure mode handling mechanism based on message circuit breakers. The Outbox message table also contains an event type field corresponding to the message to be sent. A circuit breaker status table is also set up in the business database. The circuit breaker status table is used to persist the cumulative number of failures, circuit breaker flag, and circuit breaker end time for each event type. The value of the event type field reflects the business event category to which the message to be sent belongs; for example, the value could be "quality control completion notification," "image archiving request," or "workload statistics report." Each record in the circuit breaker status table corresponds to an event type, and the record fields include event type, cumulative number of failures, circuit breaker flag, and circuit breaker end time, where the circuit breaker flag is either active or inactive.

[0149] In S2, before executing S3 for each message to be sent, the message relay service queries the circuit breaker flag of the event type corresponding to the message to be sent in the circuit breaker status table. If the circuit breaker flag is active and the current time is earlier than the circuit breaker end time, the current round of S3 is skipped and the status field of the message to be sent is retained as pending.

[0150] In S5, after each failed message transmission, the cumulative failure count of the event type corresponding to the message to be sent is updated in the circuit breaker status table. In response to the cumulative failure count reaching the preset circuit breaker threshold, the circuit breaker flag is set to active and the circuit breaker end time is set according to the preset circuit breaker duration.

[0151] The circuit breaker status table is located in the business database, and all worker instances share the same circuit breaker status table. After the circuit breaker end time is reached, the message relay service performs a trial send of the next message to be sent for that event type. If the send is successful, the circuit breaker flag is set to invalid and the cumulative failure count is cleared to zero. If the send fails, the circuit breaker end time is reset according to the preset circuit breaker duration.

[0152] For example, a batch failure of the downstream image archiving service causes consecutive failures to send messages of the "Image Archiving Request" event type. When the cumulative number of failures reaches the preset circuit breaker threshold of 10, the circuit breaker flag for the "Image Archiving Request" event type in the circuit breaker status table is set to active, and the circuit breaker end time is set to the current time plus 60 seconds. For the next 60 seconds, all worker instances skip S3 for messages of the "Image Archiving Request" event type to avoid continuous retries causing sustained impact on the message queue and downstream services. After 60 seconds, the message relay service takes the next message of this event type and performs a trial send: successful sending indicates that the downstream service has recovered, the circuit breaker flag is set to inactive, the cumulative number of failures is reset to zero, and normal sending resumes; failed sending indicates that the downstream service has not yet recovered, the circuit breaker end time is reset to the current time plus 60 seconds, and the circuit breaker remains active.

[0153] Compared to local state circuit breakers implemented only within the caller process, this mechanism extends the circuit breaker state from the memory of a single caller process to shared storage in the business database. The failure statistics for the same event type of all worker instances are completely consistent with the circuit breaker state, avoiding the distortion problem of "insufficient statistics to trigger circuit breaker from the perspective of a single worker but a serious global failure has occurred". At the same time, it extends the circuit breaker from "circuit breaker of downstream interface by the caller" to "circuit breaker of event type by the message layer", which is natively coordinated with the overall message delivery architecture of this method.

[0154] In some embodiments, messages to be sent migrated to the dead-letter table carry a retry history field in the dead-letter table. The retry history field records the occurrence time, failure reason, and error stack of each retrieval of the message to be sent in a structured form. The retry history field enables operations and maintenance personnel to perform post-event analysis and attribution of failure modes of dead-letter messages.

[0155] In some embodiments, manual retries are supported for messages to be sent in the dead-letter table. After fault repair of the dead-letter record, the core fields of the dead-letter record are reinserted into the Outbox message table, and the status field, retry count field, and next retry time field are reset. The manual retry mechanism allows dead-letter messages to re-enter the normal delivery process after the root cause of the fault is resolved, avoiding permanent decoupling of business data.

[0156] In some embodiments, the Outbox message table is partitioned and stored based on the creation time field, and the message relay service periodically cleans up pending messages in the Outbox message table whose status field is "sent" according to a preset retention period. The partitioned storage of the Outbox message table can be monthly, with each month having its own independent physical partition. The preset retention period can be 7 days, meaning that pending messages whose creation time is more than 7 days from the current time and whose status field is "sent" are deleted by the periodic cleanup operation. The synergy between partitioned storage and periodic cleanup keeps the overall size of the Outbox message table within a controllable range, and query performance does not degrade as historical messages accumulate.

[0157] In some embodiments, the message relay service periodically collects runtime metrics from the Outbox message table. The runtime metrics include the number of records with a status field of pending transmission, the hourly rolling count of records with a status field of sent, the number of records with a status field of pending transmission and a creation time field that is more than a preset threshold from the current time, and the average delay in updating the status field after successful transmission.

[0158] In some embodiments, the message relay service applies preset alarm rules to the collected runtime metrics. These preset alarm rules include: triggering a backlog alarm when the status field is "pending transmission" and the creation time field is more than a preset backlog duration limit; and triggering a failure rate alarm when the proportion of failed transmissions per unit time to the total number of transmissions exceeds a preset failure rate limit. For example, the preset backlog duration limit is 5 minutes, and the preset failure rate limit is 5%.

[0159] In some embodiments, the preset polling period is 5 seconds, the preset maximum number of retries is 5, the preset basic backoff time is 60 seconds, the preset upper load threshold is 80% of the maximum number of query messages in a single batch in this round, the preset lower load threshold is 5% of the maximum number of query messages in a single batch in this round, the preset retention period is 7 days, the preset circuit breaker threshold is 10 times, and the preset circuit breaker duration is 60 seconds. The above parameter values ​​have been verified to work in typical business scenarios such as medical image quality control, and those skilled in the art can adjust them accordingly based on the actual business load and the failure modes of downstream services.

[0160] This application also provides a computer device, including one or more processors, a memory, and one or more application programs. The one or more application programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs are configured to perform the above-described distributed transaction consistency guarantee method based on the Outbox pattern.

[0161] This application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above; the storage medium stores at least one instruction, at least one program segment, code set, or instruction set, wherein at least one instruction, at least one program segment, code set, or instruction set is loaded and executed by the processor to implement the distributed transaction consistency guarantee method based on the Outbox pattern as described above.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0163] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A distributed transaction consistency guarantee method based on the Outbox pattern, characterized in that, include: S1. Responding to a business request through a business application, simultaneously writing business data to a business data table and writing the message to be sent triggered by the business request to an Outbox message table in a local transaction of the business database, and submitting the local transaction, wherein the Outbox message table persists the message to be sent and the status field, retry count field, next retry time field and creation time field corresponding to the message to be sent; S2. Poll the Outbox message table through the message relay service at a preset polling period, add an exclusive lock to the query results and skip the locked records, and obtain the message to be sent whose status field is pending and whose next retry time field is not later than the current time; S3. The message to be sent is sent to the message queue through the message relay service, and the status field is updated to "sent" when the sending is successful; S4. The consumer queries the consumption record table based on the message identifier carried in the message to be sent. If the message identifier does not exist in the consumption record table, the downstream processing of the message to be sent is performed and the message identifier is written into the consumption record table. S5. In response to the failure to send the message to be sent, the message relay service increments the retry count field, performs the next retry scheduling for the message to be sent, and migrates the message to be sent to the dead letter table when the retry count field reaches the preset retry count limit.

2. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The message relay service includes multiple worker instances running in parallel, and the Outbox message table also contains an aggregation type field corresponding to the message to be sent; in step S2, when each worker instance performs the polling of the Outbox message table, it obtains the message to be sent belonging to itself based on the aggregation type field according to a preset partitioning rule.

3. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The preset polling period is an adaptive polling period; in S2, after each round of polling is completed, the message relay service dynamically adjusts the adaptive polling period of the next round of polling based on the number of messages to be sent obtained in the current round of polling. When the number of messages to be sent obtained in the current round is greater than a preset upper load threshold, the adaptive polling period of the next round is shortened; when the number of messages to be sent obtained in the current round is less than a preset lower load threshold, the adaptive polling period of the next round is extended.

4. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The next retry scheduling in S5 includes the following sub-steps: S51. The backoff duration is obtained by multiplying the preset basic backoff duration by the backoff coefficient corresponding to the retry count field through the message relay service; S52. The sum of the current time and the backoff duration is used as the new value of the next retry time field through the message relay service; The backoff coefficient increases exponentially with the number of retry fields.

5. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, S2 further includes the following sub-steps: S201. Verify through the message relay service whether the retry count field of the message to be sent is less than the preset retry count limit; S202. The message to be sent that is not less than the preset retry count limit is excluded from the execution object of S3 by the message relay service.

6. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, S4 includes the following sub-steps: S401. Start a database transaction through the consumer terminal, and perform an insert operation on the consumption record table with the message identifier as the unique constraint in the database transaction; S402. In response to the failure of the insert operation due to the unique constraint conflict, the consumer rolls back the database transaction and terminates this S4 statement. S403. Upon successful insertion operation, the consumer executes the downstream processing within the database transaction and commits the database transaction.

7. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The Outbox message table is partitioned and stored according to the creation time field; the message relay service periodically cleans up the pending messages in the Outbox message table whose status field is "sent" according to a preset retention period.

8. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, In step S1, when the service application writes the message to be sent into the Outbox message table, it also writes the content fingerprint of the content carried by the message to be sent into the Outbox message table; in step S3, the message relay service sends the content fingerprint and the message to be sent together to the message queue; step S4 includes the following sub-steps: S411. The consumer terminal queries the consumption record table based on the combination of the message identifier and the content fingerprint; S412. If the consumer responds to the existence of the message identifier in the consumption record table, the downstream processing is skipped. S413. If the consumer responds to the fact that the message identifier does not exist in the consumption record table but a record with the same content fingerprint does exist, skip the current downstream processing and write the message identifier into the consumption record table; S414. In response to the fact that neither the message identifier nor the content fingerprint exists in the consumption record table, the consumer performs downstream processing of the message to be sent and writes the message identifier and the content fingerprint together into the consumption record table.

9. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The message relay service maintains the scanned time pointer; step S2 includes the following sub-steps: S211. The Outbox message table is scanned using the message relay service with a new write window and a retry expiration window. The range of the new write window is records whose creation time field is later than the scanned time pointer and whose status field is pending to be sent. The range of the retry expiration window is records whose retry count field is greater than zero, whose next retry time field is not later than the current time and whose status field is pending to be sent. S212. The message relay service updates the scanned time pointer to the end time of the current scan after the scan of the newly added write window is completed.

10. The distributed transaction consistency guarantee method based on the Outbox pattern according to claim 1, characterized in that, The Outbox message table also includes a causal dependency identifier field corresponding to the message to be sent, the causal dependency identifier field recording the message identifier of the upstream message on which the message to be sent depends; S3 includes the following sub-steps: S31. Verify through the message relay service whether the causal dependency identifier field of the message to be sent is empty, or whether the status field of the upstream message recorded by the causal dependency identifier field in the Outbox message table is sent; S32. The message to be sent is sent to the message queue only when the causal dependency identifier field is empty or the upstream message has been sent, and the status field is updated to sent when the sending is successful.

Images