A method for orderly production and consumption in multiple environments based on Kafka
By implementing a technical framework at the application layer, using distributed locking and predecessor linked list mechanisms to ensure orderly and idempotent delivery and consumption of Kafka messages, the problems of out-of-order and repeated consumption of messages under the multi-environment microservice architecture are solved. The reliability and efficiency of messages are achieved.
Patent Information
- Application Number
- CN202310200211.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-28
- Publication Date
- 2025-02-25
- Estimated Expiration
- 2043-02-28
AI Technical Summary
Under the multi-environment microservice architecture, the concurrent consumption of Kafka messages leads to out-of-order messages, and the existing technology is difficult to ensure the orderliness and idempotence of messages, especially when producers are down, Kafka is down or network exceptions, Delivery and consumption may have lost or duplicate problems.
By implementing a technical framework at the application layer, using Maven or Gradle to import dependencies, the reliable delivery of messages and orderly consumption are achieved. This framework places messages in the database within the transaction, and ensures orderly delivery and consumption of messages through partition-grained distributed locking and predecessor linked list mechanisms. At the same time, failure retry and failover mechanisms are provided to ensure the idempotence and reliability of messages.
It realizes the reliability, orderly and idempotent delivery and consumption of messages under a multi-environment and multi-replica architecture, ensuring the integrity and consistency of business data, and avoiding the problems of message loss and repeated consumption.
Smart Images
Figure CN117290122B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a multi-environment orderly production and consumption method based on Kafka. Background Art
[0002] In the process of business logic processing in software systems, MQ is usually used to decouple the coupled calls between services, shorten the response time and improve system performance through asynchronous messages. There are many problems in the asynchronous consumption scenario, which will cause the production and consumption of messages to not be executed in the order of transaction submission in the business. Message disorder may cause serious business impact, and there may be many problems such as dirty data and write overwrite. And with the popularity of microservice architecture, a system may have multiple environments in a multi-copy multi-environment architecture, such as: pre-release environment, production environment V1, and production environment V2. In this multi-environment architecture, there may be multiple senders of MQ messages, and messages in the same queue will be consumed concurrently. In the scenario of concurrent consumption, the message consumption will still be disordered, and orderly consumption faces many challenges in a multi-environment architecture.
[0003] Since business methods are executed concurrently, when producers send messages, if they are not processed through technical means, they will be delivered out of order due to concurrency, network delays, service jams and other issues. When producers send messages to Kafka, even synchronous messages will be out of order due to concurrency. Asynchronous sending cannot ensure reliable message delivery, and a reliable delivery mechanism is required.
[0004] The existing technical solution is suitable for orderly consumption in a single environment, that is, only one consumer will receive the message for a partition. In the scenario of multiple environments, for example, the online business system has a V1 version environment and a V2 version environment. Consumers in the two environments use different consumer groups and can receive messages from the same partition. In the scenario of parallel consumption in multiple environments, the orderliness of the business cannot be guaranteed. For the scenario of consumption failure, Kafka's default retry strategy has poor throughput. A message failure will block the execution of subsequent messages, and in the process of message consumption, repeated consumption may occur. The idempotence of message consumption cannot be absolutely guaranteed. Once repeated consumption occurs, it may have a serious impact on business data.
[0005] Therefore, those skilled in the art provide a method for orderly production and consumption in multiple environments based on Kafka to solve the problems raised in the above background technology. Summary of the invention
[0006] 1. Technical issues to be resolved
[0007] In view of the shortcomings of the prior art, the present invention provides a multi-environment orderly production and consumption method based on Kafka, which can ensure the reliable delivery of message producers. Regardless of the fault scenarios of producer downtime, Kafka downtime, network flash disconnection, and DB abnormality, the producer can still continue to deliver in an orderly manner after the fault is restored. If the consumer crashes after consuming the message but has not submitted the offset, then after restarting, Kafka will re-deliver the message and perform idempotent processing on duplicate messages to avoid duplicate consumption.
[0008] (II) Technical solution
[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0010] A method for orderly production and consumption in multiple environments based on Kafka is implemented through a technical framework, which is characterized in that the technical framework works at the application layer and is not an independent middleware system. Dependencies are imported into the project through Maven or Gradle and it is ready to use out of the box. The technical framework transmits delivery messages and consumption messages to the Kafka middleware through the message frameworks of multiple applications, and queries the messages to be delivered through the DB end.
[0011] The implementation process of this method includes the following steps:
[0012] Step 1: Reliable message delivery
[0013] S1. After the application completes the business operation, it will store the message to be sent in the same transaction in the message table (msg_info). The message table needs to store which queue the message is to be sent to and which partition it should be routed to. The partition index here is hashed by the primary key of the business data and the number of partitions of the Kafka queue. The primary key in the message table uses an auto-increment ID. Since the message data is completed in the same transaction as the business data, the generation of the message can meet the atomicity. Since a row lock is added when a transaction modifies the data, even if the application accesses the database concurrently, at the database level, operating the same data first needs to obtain the row lock of the data. The row lock is an exclusive lock, so concurrent threads will wait. Therefore, the order in which transactions are submitted is the order in which messages are executed.
[0014] S2. Obtain a distributed lock at the partition granularity to prevent multiple producers from concurrently producing messages to the same partition, which would cause producers to deliver messages out of order.
[0015] S3. The producer pulls the undelivered data from the message table, sorts them by SeqId, and stores the business IDs to be operated on these messages for each message to be sent. The ID to be operated on in the current message to be sent is stored in the data table msg_id_info, so that the predecessor message can be generated before the subsequent message is sent.
[0016] S4. After saving the service ID corresponding to the message, it is necessary to determine whether there is a predecessor message for the current message to be sent;
[0017] Predecessor message: If a message is to be consumed, it needs to wait for the predecessor message to be consumed before starting to consume the current message. When sending a message, it is necessary to determine whether there is a predecessor message;
[0018] Predecessor linked list: If the SeqId of a message is 11, and the business data ID to be operated by this message is 3, then according to the reverse order of SeqId, check whether there is a message with operation ID 3 before the current event. If so, add the SeqId of the previous message to the message;
[0019] S5. After the predecessor relationship is bound, the message to be sent is sent to Kafka. The producer ack confirmation mechanism needs to be changed to -1, that is, the message is considered to be delivered successfully only after both the leader and the follower have received the message. After the message is sent, the status of the message in the database is changed to delivered. The message delivery and status modification are within one transaction. If the message delivery fails, the status will not be modified, and the producer will not continue to deliver the next event, but will keep retrying on the current message. If the current message is skipped and continued to be delivered, the previous message will be lost.
[0020] At this point, the producer can deliver messages to a partition under the Kafka topic reliably and in order.
[0021] Step 2: Producer Fault Tolerance and Failover
[0022] If the producer goes down or Kafka goes down, message delivery will fail. For the above scenarios, the producer should have the ability to retry failures and failover. The specific scenarios and solutions are as follows:
[0023] Scenario 1: Producer downtime
[0024] Retry processing: After the producer restarts, it should pull all undelivered messages and retry
[0025] Failover: If there are multiple producers in the cluster and one producer goes down, the message will be pulled up and retried by other producers at regular intervals to achieve failover;
[0026] Scenario 2: Kafka downtime
[0027] If the middleware crashes, the producer will keep retrying to deliver the current message until it succeeds.
[0028] Step 3: Idempotence of message consumption
[0029] If duplicate messages are generated in the queue, if the consumer does not perform idempotence control, duplicate consumption will occur, which may have a serious impact on the business. Duplicate messages may be generated in the following scenarios:
[0030] Scenario 1: If the producer experiences network jitter, the message is actually delivered successfully, but the producer does not receive the ACK from Kafka and determines that the message has failed to be sent. The producer will retry to deliver the message.
[0031] Scenario 2: Since the offset of Kafka consumers is usually submitted in batches, that is, the offset is submitted once after consuming 10 messages. If the message consumption is completed but the offset has not been submitted, the consumer crashes. At this time, Kafka does not receive the offset. After the consumer is restarted again, Kafka will re-deliver the previous message, resulting in duplicate consumption.
[0032] Step 4: Reliable and orderly consumption
[0033] S1. Idempotence check and consumption record initialization;
[0034] S2. According to the partition and the business ID carried in the message, the corresponding thread is found through hashing to process. Different business IDs in the same partition can be executed concurrently, which improves the concurrency of orderly consumption.
[0035] S3. Determine whether there is a predecessor message. If so, determine whether the predecessor message has been consumed. If the predecessor message has not been consumed, change the status of the current consumption record to queued. If it has been consumed or the predecessor message does not exist, start consuming the current message;
[0036] S4. Get the primary key of the business data to be consumed by the current message, and determine whether there are any messages that have failed to be consumed in the current partition. If there are any messages that have failed to be consumed, you need to queue up and wait for the previously failed messages to be successfully retried before you can continue consuming. Otherwise, consumption will be out of order, because there are messages that have failed to be consumed before the queue. If you consume directly, when the previously failed messages start to be retried, consumption will be out of order.
[0037] S5. Track the consumption of the message. If the message consumption is successful, change the status of the consumption record to consumption success. If the message consumption fails, change the consumption status of the current message to failure.
[0038] S6. After consumption is successful, the consumer's offset for the current consumption branch and business ID is submitted, and subsequent queued events are awakened.
[0039] In S3 of the step 1, if the message to be delivered has SeqId = 10, and the business data of the message needs to operate the business data with business ID 3, then the structure stored in msg_id_info is seqId, businessId, and the storage result is 10, 3.
[0040] In S4 of step 1, the predecessor message is "pointed" to the seqId of the previous message. The pointing here only stores a preSeq attribute in the message header of Kafka, which identifies the SeqId of the predecessor message of a message. If the message with SeqId 10 also needs to operate the business data with ID 3, then the preSeq of the message = 10. The predecessor message binding method of subsequent events is similar, and finally a predecessor linked list will be generated in the queue.
[0041] The solution to the scenario in step 3 is as follows:
[0042] Since the SeqId of the message is unique and the business logic of consumption is unique, the executor is used to package the business logic in the framework. The executor is a class that implements the Runnable interface and encapsulates the framework logic and business logic. The uniqueness can be guaranteed by SeqId and consumer group ID plus the method name. Before the message is consumed, the execution record of the executor will be initialized and the consumption record will be inserted into the database. If a unique index conflict occurs, it means that the message has been consumed and does not need to be consumed again.
[0043] In S4 of the step 4, if the consumption order is 1->2->3, if 1 fails, 2 and 3 succeed, then after 1 is retried, the consumption order becomes 2->3->1, resulting in disordered message consumption.
[0044] In step 4, if there is a consumption listener for business 1 in consumer 1, business 1 is a business method, which processes the data of the incoming business parameters. This method can be executed concurrently. Multiple threads execute the business 1 method. It only needs to ensure that the business parameters in the executed method are different. After the consumption is completed, the offset is submitted in the following format: business method 1, business ID, seqId of the message. The offset is the condition for judging whether the preceding message is completed in S3. If a message needs to be judged whether the preceding message is executed before it is executed, it is to judge whether the offset of the current business method is greater than or equal to the seqId of the preceding message.
[0045] (III) Beneficial effects
[0046] The present invention provides a multi-environment orderly production and consumption method based on Kafka. It has the following beneficial effects:
[0047] 1. The present invention provides a method for orderly production and consumption in multiple environments based on Kafka, which ensures that messages are produced reliably and orderly. Even if there is a crash, network interruption, application execution error, or database HA switching, the messages can still be delivered in order and no messages will be lost.
[0048] 2. The present invention provides a multi-environment orderly production and consumption method based on Kafka, which ensures the idempotence of messages. Whether the producer times out and retries, or the consumer crashes, there will be no problem of repeated consumption.
[0049] 3. The present invention provides a method for orderly production and consumption in multiple environments based on Kafka. Under the premise of orderly consumption in partitions, routing is performed according to the business ID, and multi-threading is used to consume data of different business IDs, thereby improving the concurrency of message consumption and avoiding the problem of task blocking to a certain extent.
[0050] 4. The present invention provides a method for orderly production and consumption in multiple environments based on Kafka. In the case of multiple environments and multiple consumers, the orderly consumption of messages can still be guaranteed through the predecessor judgment and queuing wake-up mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 It is a schematic diagram of the internal structure of the technical framework of the present invention;
[0052] Figure 2 It is a schematic diagram of the technical framework of the present invention;
[0053] Figure 3 It is a schematic diagram of the transaction and message storage process of the present invention;
[0054] Figure 4 It is the predecessor linked list flow chart of the present invention;
[0055] Figure 5 Sending a flow chart to the producer of the present invention;
[0056] Figure 6 This is a schematic diagram of the producer fault-tolerant processing and failover technology of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example
[0058] like Figure 1-6 As shown, an embodiment of the present invention provides a method for orderly production and consumption in multiple environments based on Kafka. The method is implemented through a technical framework, which is characterized in that the technical framework works at the application layer and is not an independent middleware system. Dependencies are imported into the project through Maven or Gradle and it is ready to use out of the box. The technical framework transmits delivery messages and consumption messages to the Kafka middleware through the message frameworks of multiple applications, and checks the messages to be delivered through the DB end.
[0059] The implementation process of this method includes the following steps:
[0060] Step 1: Reliable message delivery
[0061] S1. After the application completes the business operation, it will store the message to be sent in the same transaction in the message table (msg_info). The message table needs to store which queue the message is to be sent to and which partition it should be routed to. The partition index here is hashed by the primary key of the business data and the number of partitions of the Kafka queue. The primary key in the message table uses an auto-increment ID. Since the message data is completed in the same transaction as the business data, the generation of the message can meet the atomicity. Since a row lock is added when a transaction modifies the data, even if the application accesses the database concurrently, at the database level, operating the same data first needs to obtain the row lock of the data. The row lock is an exclusive lock, so concurrent threads will wait. Therefore, the order in which transactions are submitted is the order in which messages are executed.
[0062] The order in which messages are sent is the order of the auto-increment IDs. This primary key is the globally ordered SequenceId (SeqId) of the messages. The transaction and message storage process is as follows: Figure 3 shown.
[0063] S2. Obtain a distributed lock at the partition granularity to prevent multiple producers from concurrently producing messages to the same partition, which would cause producers to deliver messages out of order.
[0064] S3. The producer pulls the undelivered data from the message table, sorts them by SeqId, and stores the business IDs to be operated on these messages for each message to be sent. The ID to be operated on in the current message to be sent is stored in the data table msg_id_info, so that the predecessor message can be generated before the subsequent message is sent.
[0065] If the message to be delivered has SeqId = 10, and the business data of the message requires the operation of business data with business ID 3, then the structure stored in msg_id_info is seqId, businessId, and the storage result is 10, 3.
[0066] S4. After saving the service ID corresponding to the message, it is necessary to determine whether there is a predecessor message for the current message to be sent;
[0067] Predecessor message: If a message is to be consumed, it needs to wait for the predecessor message to be consumed before starting to consume the current message. When sending a message, it is necessary to determine whether there is a predecessor message;
[0068] Predecessor linked list: If the SeqId of a message is 11, and the business data ID to be operated by this message is 3, then according to the reverse order of SeqId, check whether there is a message with operation ID 3 before the current event. If so, add the SeqId of the previous message to the message;
[0069] The predecessor message is "pointed" to the seqId of the previous message. The pointing here is just to store a preSeq attribute in the Kafka message header, which identifies the SeqId of the predecessor message of a message. If the message with SeqId 10 also needs to operate the business data with ID 3, then the preSeq of the message is 10. The predecessor message binding method of subsequent events is similar, and finally a predecessor linked list will be generated in the queue. The predecessor linked list is as shown in the attached figure. Figure 4 shown.
[0070] S5. After the predecessor relationship is bound, the message to be sent is sent to Kafka. The producer ack confirmation mechanism needs to be changed to -1, that is, the message is considered to be delivered successfully only after both the leader and the follower have received the message. After the sending is completed, the status of the message in the database is changed to delivered. The message delivery and status modification are within one transaction. If the message delivery fails, the status will not be modified, and the producer will not continue to deliver the next event, but will continue to retry on the current message. If the current message is skipped and continued to be delivered, the previous message will be lost. The producer sending process is as shown in the attached Figure 5 As shown;
[0071] At this point, the producer can deliver messages to a partition under the Kafka topic reliably and in order.
[0072] Step 2: Producer Fault Tolerance and Failover
[0073] If the producer goes down or Kafka goes down, message delivery will fail. For the above scenarios, the producer should have the ability to retry failures and failover. The specific scenarios and solutions are as follows:
[0074] Scenario 1: Producer downtime
[0075] Retry processing: After the producer restarts, it should pull all undelivered messages and retry
[0076] Failover: If there are multiple producers in the cluster and one producer goes down, the message will be pulled up and retried by other producers at regular intervals to achieve failover;
[0077] Scenario 2: Kafka downtime
[0078] If the middleware crashes, the producer will keep retrying to deliver the current message until it succeeds.
[0079] Step 3: Idempotence of message consumption
[0080] If duplicate messages are generated in the queue, if the consumer does not perform idempotence control, duplicate consumption will occur, which may have a serious impact on the business. Duplicate messages may be generated in the following scenarios:
[0081] Scenario 1: If the producer experiences network jitter, the message is actually delivered successfully, but the producer does not receive the ACK from Kafka and determines that the message has failed to be sent. The producer will retry to deliver the message.
[0082] Scenario 2: Since the offset of Kafka consumers is usually submitted in batches, that is, the offset is submitted once after consuming 10 messages, if the message consumption is completed but the offset has not been submitted, the consumer crashes. At this time, Kafka does not receive the offset. After the consumer is restarted again, Kafka will re-deliver the previous message, resulting in duplicate consumption;
[0083] Solution: Since the SeqId of the message is unique and the business logic of consumption is unique, the executor is used to package the business logic in the framework. The executor is a class that implements the Runnable interface and encapsulates the framework logic and business logic. The uniqueness can be guaranteed by SeqId and consumer group ID plus the method name. Before the message is consumed, the execution record of the executor will be initialized and the consumption record will be inserted into the database. If a unique index conflict occurs, it means that the message has been consumed and does not need to be consumed again.
[0084] Step 4: Reliable and orderly consumption
[0085] S1. Idempotence check and consumption record initialization;
[0086] S2. According to the partition and the business ID carried in the message, the corresponding thread is found through hashing to process. Different business IDs in the same partition can be executed concurrently, which improves the concurrency of orderly consumption.
[0087] S3. Determine whether there is a predecessor message. If so, determine whether the predecessor message has been consumed. If the predecessor message has not been consumed, change the status of the current consumption record to queued. If it has been consumed or the predecessor message does not exist, start consuming the current message;
[0088] S4. Get the primary key of the business data to be consumed by the current message, and determine whether there are any messages that have failed to be consumed in the current partition. If there are any messages that have failed to be consumed, you need to queue up and wait for the previously failed messages to be successfully retried before you can continue consuming. Otherwise, consumption will be out of order, because there are messages that have failed to be consumed before the queue. If you consume directly, when the previously failed messages start to be retried, consumption will be out of order.
[0089] For example, if the consumption order is 1->2->3, if 1 fails, but 2 and 3 succeed, then after 1 is retried, the consumption order becomes 2->3->1, resulting in disordered message consumption.
[0090] S5. Track the consumption of the message. If the message consumption is successful, change the status of the consumption record to consumption success. If the message consumption fails, change the consumption status of the current message to failure.
[0091] S6. After consumption is successful, the consumer's offset for the current consumption branch and business ID is submitted, and subsequent queued events are awakened.
[0092] In step 4, if there is a consumption listener for business 1 in consumer 1, business 1 is a business method that processes the data of the incoming business parameters. This method can be executed concurrently. Multiple threads execute the business 1 method. You only need to ensure that the business parameters in the executed method are different. After the consumption is completed, the offset is submitted in the following format: business method 1, business ID, seqId of the message. The offset is the condition for judging whether the preceding message is completed in S3. If a message needs to be judged whether the preceding message is executed before it is executed, it is to judge whether the offset of the current business method is greater than or equal to the seqId of the preceding message.
[0093] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for orderly production and consumption in multiple environments based on Kafka, which is implemented through a technical framework and is characterized by: The technical framework works at the application layer, not an independent middleware system. It can be used out of the box by importing dependencies into the project through Maven or Gradle. The technical framework transmits delivery and consumption messages to the Kafka middleware through the message framework of multiple applications, and checks the messages to be delivered through the DB end; The implementation process of this method includes the following steps: Step 1: Reliable message delivery S1. After the application completes the business operation, it will store the message to be sent in the same transaction in the message table. The message table stores the queue to which the message is to be sent and the partition to which it is to be routed. The partition index is obtained by hashing the primary key of the business data and the number of partitions of the Kafka queue. The primary key in the message table uses an auto-increment ID. The primary key in the message table is the global ordered SeqId of the message. Since the message data is completed in the same transaction as the business data, the generation of the message satisfies atomicity. Since a row lock is added when a transaction modifies the data, even if the application accesses the database concurrently, at the database level, the row lock of the data must be obtained first to operate the same data. The row lock is an exclusive lock, so concurrent threads will wait. Therefore, the order in which transactions are submitted is the order in which messages are executed. S2. Obtain a distributed lock at the partition granularity to prevent multiple producers from concurrently producing messages to the same partition, which would cause producers to deliver messages out of order. S3. The producer pulls the undelivered messages in the message table, sorts them by SeqId, and stores the business IDs of the messages to be operated on for each message to be sent. The producer stores the business ID of the operation in the current message to be sent in the data table msg_id_info, so that the predecessor message can be generated before the subsequent message is sent. S4. After saving the service ID corresponding to the message, it is necessary to determine whether there is a predecessor message for the current message to be sent; Predecessor message: If a message is to be consumed, it needs to wait for the predecessor message to be consumed before starting to consume the current message. When sending a message, it is necessary to determine whether there is a predecessor message; Predecessor linked list: If the SeqId of a message is 11, and the business ID to be operated by the message is 3, then check in reverse order of SeqId whether there is a message with business ID 3 before the current event. If so, add the SeqId of the predecessor message to the current message; S5. After the predecessor relationship is bound, the message to be sent is sent to Kafka, and the producer ack confirmation mechanism is changed to -1, that is, the message is considered to be delivered successfully only after both the leader and the follower have received the message. After the delivery is completed, the status of the message in the database is changed to delivered. The message delivery and status modification are within one transaction. If the message delivery fails, the status will not be modified, and the producer will not continue to deliver the next event, but will continue to retry on the current message. If the current message is skipped and continued to be delivered, the previous message will be lost; Step 2: Producer Fault Tolerance and Failover If the producer goes down or Kafka goes down, message delivery will fail. For the above scenarios, the producer has the ability to retry and failover. The specific scenarios and solutions are as follows: Scenario 1: Producer downtime Retry processing: After the producer restarts, pull all undelivered messages and retry; Failover: If there are multiple producers in the cluster and one producer goes down, the message will be pulled up and retried by other producers at regular intervals to achieve failover; Scenario 2: Kafka downtime If the middleware crashes, the producer will keep retrying to deliver the current message until it succeeds; Step 3: Idempotence of message consumption If duplicate messages are generated in the queue, if the consumer does not perform idempotence control, duplicate consumption will occur. Duplicate messages will be generated in the following scenarios: Scenario 1: If the producer experiences network jitter, the message is actually delivered successfully, but the producer does not receive the ACK from Kafka and determines that the message has failed to be sent. The producer will re-deliver the message. Scenario 2: The offset of the Kafka consumer is submitted in batches. That is, after consuming 10 messages, the offset is submitted once. If the message consumption is completed but the offset has not been submitted, the consumer crashes. At this time, Kafka does not receive the offset. After the consumer is restarted, Kafka will re-deliver the previous message, resulting in duplicate consumption; The solution for the scenario in step 3 is as follows: Since the SeqId of a message is unique and the business logic of consumption is unique, the framework uses an executor to package the business logic. The executor is a class that implements the Runnable interface and encapsulates the framework logic and business logic. The uniqueness is ensured by adding the SeqId and the consumer group ID to the method name. Before the message is consumed, the execution record of the executor is initialized and the consumption record is inserted into the database. If a unique index conflict occurs, it means that the message has been consumed and does not need to be consumed again. Step 4: Reliable and orderly consumption S1. Idempotence check and consumption record initialization; S2. According to the partition and the business ID carried in the message, the corresponding thread is found through hashing to process. Different business IDs in the same partition are executed concurrently, which improves the concurrency of orderly consumption. S3. Determine whether there is a predecessor message. If so, determine whether the predecessor message has been consumed. If the predecessor message has not been consumed, change the status of the current consumption record to queued. If it has been consumed or the predecessor message does not exist, start consuming the current message. S4. Get the primary key of the business data to be consumed by the current message, and determine whether there are any messages that have failed to be consumed in the current partition. If there are any messages that have failed to be consumed, you need to queue up and wait for the previously failed messages to be successfully retried before you can continue consuming. Otherwise, consumption will be out of order, because there are messages that have failed to be consumed before the queue. If you consume directly, when the previously failed messages start to be retried, consumption will be out of order. S5. Track the consumption of the message. If the message consumption is successful, change the status of the consumption record to consumption success. If the message consumption fails, change the consumption status of the current message to failure. S6. After consumption is successful, the consumer's offset for the current consumption branch and business ID is submitted, and subsequent queued events are awakened.
2. According to claim 1, a method for orderly production and consumption in multiple environments based on Kafka is characterized by: In S3 of the step 1, if the message to be delivered has SeqId = 10, and the business data of the message needs to operate the business data with business ID 3, then the structure stored in msg_id_info is seqId, businessId, and the storage result is 10, 3.
3. According to claim 1, a method for orderly production and consumption in multiple environments based on Kafka is characterized by: In S4 of step 1, the predecessor message is pointed to the seqId of the previous message. The pointing here is just to store a preSeq attribute in the message header of Kafka, which identifies the SeqId of the predecessor message of a message. The predecessor message binding method of subsequent events is similar, and finally a predecessor linked list will be generated in the queue.
4. According to claim 1, a method for orderly production and consumption in multiple environments based on Kafka is characterized by: In the step 4, if there is a consumption listener for business 1 in consumer 1, business 1 is a business method, which processes the data of the incoming business parameters. The method is executed concurrently, and multiple threads execute the business 1 method. It only needs to ensure that the business parameters in the executed method are different. After the consumption is completed, the offset is submitted. The format is as follows: business method 1, business ID, seqId of the message. The offset is the condition for judging whether the predecessor message is completed in S3. If a message needs to be judged whether the predecessor message is executed before execution, it is necessary to judge whether the offset of the current business method is greater than or equal to the seqId of the predecessor message.