Message queue consumer processing method based on transaction group

Abstract

The present invention provides a message queue consumer processing method based on transaction group implementation, including: defining order receipt message processing as a transaction group; splitting into multiple sub-level calls according to the external call interface dimension, and defining corresponding message types for each sub-level, the message type definitions are as follows: parent root method, modifying the document receipt quantity, calling the SAP system to confirm the receipt income, calling the points system to add new receipt points, calling the settlement system to add new receipt flow, calling the inventory system to add new receipt flow. The present invention splits a large message processing into multiple small sub-levels, each sub-level has an independent processing status, when the parent calls the sub-level in sequence, if the sub-level execution fails midway, the subsequent retry program will skip the successful sub-level, and only retry the sub-level with a failed processing status, which greatly improves the retry efficiency and reduces the service resource pressure.

Description

Technical Field

[0001] The present invention relates to the field of computer software communication, and in particular to a situation where consumers need to support idempotence in message queue communication between heterogeneous systems and involve calling external systems to write data, and more particularly to a message queue consumer processing method based on transaction groups. Background Art

[0002] In heterogeneous system communication scenarios, for business scenarios that do not require the called party to return processing results in real time, message queues are generally used as communication middleware in the communication architecture. The advantage of message queues is that they can reduce the situation where the caller system is dragged down due to instability or long time consumption of the called party system, and decouple transactions under high concurrency. The message sender only needs to ensure that the message is sent to the message queue system to end. The message handler consumes and processes the message from the message queue asynchronously. The message sender and the consumer group are separated and independent, and the message sender does not need to block and wait for the message to be processed. Common scenarios include: order delivery, order receipt, increase points, and start approval flow in e-commerce systems.

[0003] In the solution of using message queue middleware, in order to prevent the loss of messages, a message resending mechanism is usually provided on the message sending end. In addition, the message queue system usually sends the same message to consumers repeatedly, so the consumer end must ensure the idempotence of the message to prevent repeated processing of the same message transaction. Since most asynchronous transactions must be processed by the consumer end to obtain a result, it is also necessary to support the retry of messages that fail in the middle of processing. For messages that still fail after multiple retries, manual intervention must be supported for operation and maintenance.

[0004] After research, it was found that most of the above scenarios on the market usually create corresponding message log tables on the consumer side to ensure idempotency, and perform operation and maintenance through development pages. This method is effective when the message processing does not involve external system write interfaces. However, if it involves write calls to external system interfaces, especially when the call chain is long (for example, message processing involves calls to external A, B, C, and D data write interfaces), all downstream system interfaces must also implement idempotency to prevent repeated writing due to retries after message failure (for example, an error is reported when calling the last D interface).

[0005] Although the downstream system can achieve idempotence, each retry still takes a long time, has poor performance, and consumes a lot of resources when the call link is long. Summary of the invention

[0006] The present invention provides a message queue consumer processing method based on transaction groups to solve at least one of the above technical problems.

[0007] To solve the above problem, as one aspect of the present invention, a message queue consumer processing method based on a transaction group is provided, comprising:

[0008] Define order receipt message processing as a transaction group;

[0009] According to the external call interface dimension, it is divided into multiple sub-level calls, and the corresponding message type is defined for each sub-level. The message type definitions are as follows: parent root method, modify document receipt quantity, call SAP system to confirm receipt income, call the points system to add receipt points, call the settlement system to add receipt flow, call the inventory system to add receipt flow;

[0010] When a new message is consumed from the message queue, the first root parent message log object is created, and the initial processing status is failure;

[0011] The parent method calls the child method in sequence, and each time the child method is called, a corresponding child message log object is created, which corresponds to a record in the database message log table.

[0012] Each time a child is successfully called, the processing status is marked as successful. When all children are successfully executed, the processing status of all children and the parent becomes successful.

[0013] During the sequential execution process, if an exception occurs when executing a certain child, the exception is thrown externally, and only the current child and parent are marked as failed. The processing status of the previously successfully executed child remains successful.

[0014] Preferably, the method first enters the parent module, and the parent module sequentially performs the following steps:

[0015] Step S100, executing the parent define unit to define the consumption interface;

[0016] Step S110, executing the parent begin unit to start a message transaction, create a transaction group context, and push it into a stack;

[0017] Step S120, execute the parent doWork unit to execute the parent business method. After step S120, enter the child module and execute the following parent units respectively according to the results of the child module:

[0018] Step S130, executing the parent commit unit, which is used to commit the transaction and pop the stack when the business method is successfully executed;

[0019] Step S140, executing the parent rollback unit, which is used to roll back the transaction and pop the stack when the business method fails to execute;

[0020] Step S150, executing the parent clear unit to clear the transaction group context.

[0021] Preferably, the sub-module performs the following steps:

[0022] Step S200, executing the sub-level define unit, which is used to define the consumption interface as required, and executing step S210 after step S200;

[0023] Step S210, executing the sub-level begin unit, which is used to start the message transaction, create the transaction group context, and push it into the stack. After step S210, step S220 is executed;

[0024] Step S220, executing the sub-level doWork unit, which is used to execute the sub-level business method, and then executing step S230;

[0025] Step S230, determining whether the execution of the business method is successful, if successful, executing step 240, if unsuccessful, executing step S250;

[0026] Step S240, executing the sub-level commit unit to commit the transaction and pop the stack;

[0027] Step S250, executing the sub-level rollback unit to roll back the transaction and pop the stack, and executing step S260 after step S250;

[0028] Step S260, executing the sub-level clear unit to clear the transaction group context, and executing step S270 after step S260;

[0029] Step S270, determine whether it is the last sub-method, if yes, execute step 310, otherwise return to step 200.

[0030] Preferably, step S100 includes:

[0031] Define a set of message type enumerations for a message consumption transaction group;

[0032] The parent of the transaction group corresponds to a method, or called the root method, and each child corresponds to a method. The parent method calls each child method synchronously in the business order. All related methods in the transaction group must be annotated with the message transaction AOP aspect annotation set by the present invention, and the message type is defined to the message type attribute of the corresponding method annotation;

[0033] All related methods in the transaction group must take a message provider object parameter as a parameter. The message provider object parameter must implement a set message provider interface. The interface defines four methods that must be implemented by subclasses to obtain message content, message type, message ID, and business keywords.

[0034] If the transaction group involves a cluster multi-node deployment architecture, you also need to annotate the transaction group root method with a distributed lock module annotation to avoid concurrent calls on multiple nodes.

[0035] After the program is defined according to the agreement, when a new message is received, it will be triggered to first determine whether the distributed lock is annotated. If so, it will try to obtain the distributed lock through the specified lock key. If the lock acquisition fails, an error will be reported and the execution will continue if the lock acquisition succeeds.

[0036] Based on the AOP aspect, the message provider object in the current execution method is obtained. If the object cannot be obtained, an error is reported and the log is printed to inform that it does not meet the defined requirements. If the object is obtained, the method continues. In the subsequent stages, the core field content can be obtained through the message provider object: message body, message type, message ID and key business key.

[0037] Preferably, step S110 includes:

[0038] Determine whether the current thread has a transaction group context. If not, create a message transaction group context for the current thread and create an empty first-in-last-out method stack for the context.

[0039] The message type and message ID are obtained through the message provider object of the current method, and then queried from the message log table; if the message log table already exists, it is converted into a message log object based on the records existing in the database. At this time, the processing status of the message log object is based on the last processing result of the database. The message log object will be marked as non-new, and it will be determined whether the processing status of the record is failed. If it is failed, it will be marked as retryable; if the message log table does not exist, a new message log object is created based on the message provider object of the current method. The processing status of the newly created message log object defaults to failure, which is equivalent to pending processing;

[0040] The message log object corresponding to the current method is further encapsulated into a stack frame and pushed into the context of the current transaction group;

[0041] Determine whether the number of current context method stacks = 1. If so, create and start a parent method local transaction.

[0042] Preferably, step S120 includes:

[0043] Obtain the method stack corresponding to the current method from the transaction group context, extract the message log object prepared by the parent begin unit in S110 from the method stack, and determine the idempotence by whether the processing status of the message log object is successful;

[0044] If the processing status is successful, it is considered idempotent and the subsequent actual business logic of the current method will not be executed. At the same time, the message message of the current message log object will be parsed into the next message provider parameter object as the input parameter object of the message provider for calling the next sub-method;

[0045] If the processing status is failure, it is considered non-idempotent, and the new identification of the message log object created by the parent begin unit in S110 is further determined. If the identification is new, the message log object is saved to the database message log table. If it is not new, the number of retries is accumulated and the message log object is updated to the message log table.

[0046] If the processing status is failure, it is considered non-idempotent. Finally, whether it is the parent is determined by whether the number of method stacks = 1. If it is the parent, the actual business logic is called directly. If it is the child, follow the instructions for the child doWork below.

[0047] Preferably, the only difference between the child begin unit in step S210 and the parent begin unit in the previous S110 is that whether it is a parent is determined according to the number of method stacks. Only the parent will start a local transaction in this unit, and the child's local transaction is controlled by the child doWork unit in S220.

[0048] The only difference between the doWork unit in the child in step S220 and the parent doWork unit in S110 is that before the actual business logic is actually executed, the child's own independent local transaction will be started first;

[0049] In step S230, if the execution of the actual business logic processing of the child is successful, the current child-level local transaction is committed in the child-level doWork unit of S220, and the child-level local transaction is completely controlled by the child-level doWork unit of S220; if the execution of the actual business logic fails, the current child-level local transaction is rolled back in the child-level doWork unit of S220, and an exception is thrown to the outside. When any child fails to be executed, other children will not be executed further, and the parent-level rollback unit S140 will be finally executed. At this time, the processing status of the currently failed child and parent-level message log objects are both failed; when the parent-level rollback unit S140 is finally executed, three retries are performed through the automatic retry module; if the three retries still fail, the intelligent early warning module unit of the automatic retry module is triggered according to the configuration, and a visual interface is provided for manual intervention according to the operation and maintenance module unit of the automatic retry module.

[0050] Preferably, step S270 also identifies whether the current parent commit or child commit is determined by whether the number of method stacks = 0. If it is identified as a child commit, it further determines whether the message log object in the stack frame currently popped from the method stack is a new or retry identifier. If so, the message body of the message log object is updated to the message log table; if it is identified as a parent commit, the parent commit unit is executed.

[0051] Preferably, step S270 also identifies whether the current parent rollback or child rollback is performed by checking whether the method stack quantity is 0. If it is a child rollback, the current method stack frame is popped; if it is a parent rollback, the parent rollback unit is executed.

[0052] Preferably, the automatic retry module performs the following steps in sequence:

[0053] Step S310, through the automatic cleaning module, configurable switches, timing frequency cron expressions, cleaning time points and other options are provided, and the child with the parent processing status as successful is physically deleted according to the configuration, and the parent is still retained as idempotent;

[0054] Step S320, through the intelligent early warning module, a configurable switch is provided, and email, SMS or other notification methods are customized to be implemented, and abnormal notification is made to the parent rollback unit according to the configuration;

[0055] Step S330: Provide API interfaces such as paging condition list query and manual retry of parent-child structure through the visual operation and maintenance module, and the front end performs visual operation and maintenance through the docking API interface;

[0056] Step S340, through the distributed lock module, a distributed lock based on Redis lock is provided, which is used to control concurrent execution of multiple nodes in the cluster and ensure the idempotence requirements of multiple nodes;

[0057] Step S350: Automatically create a related message log table when first introduced through an automatic table creation unit.

[0058] The advantages and differences of the transaction group method of the present invention compared with other traditional implementation methods are:

[0059] (1) The present invention splits a large message processing into multiple small sub-levels, each of which has an independent processing status. When the parent level calls the sub-levels in sequence, if the sub-level fails to execute in the middle, the subsequent retry program will skip the successful sub-level and only retry the sub-level with a failed processing status, which greatly improves the retry efficiency and reduces the service resource pressure.

[0060] (2) The present invention can support the situation where the downstream system does not implement idempotence, because the successfully executed child external call will not be executed again during the next retry.

[0061] (3) The present invention implements a set of standard consumer definitions based on transaction groups, supports automatic retry, intelligent early warning, visual operation and maintenance, automatic cleanup, automatic table creation, distributed locks and other units, completely simplifying development and improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of the message processing method provided in this manual.

[0063] Figure 2 A flowchart of a message processing application scenario provided for one embodiment of this specification.

[0064] Figure 3 This is a structural diagram of other device modules of the system provided in one embodiment of this specification. DETAILED DESCRIPTION

[0065] The following is a detailed description of embodiments of the present invention, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0066] The present invention provides a new method for ensuring that consumers process messages based on parent-child relationship transaction groups, which is used to solve the existing problems in the market such as writing data to external systems, long call links leading to low retry execution efficiency, downstream services must be idempotent, and high server resource consumption. Figure 1 The message processing flow is used to clearly and completely illustrate the implementation scheme of the device of the present invention.

[0067] The entire message transaction group in the message queue consumer processing method based on transaction group implementation in the present invention is implemented based on basic technologies such as object-oriented high-level development language, local database transaction, proxy and AOP aspect, database, etc.

[0068] Please refer to Figure 1 , the S100 parent define unit comprises the steps of:

[0069] First, define a set of message type enumerations for a message consumption transaction group. Usually, the message types of different transaction groups in a single system must be internally unique. It is recommended that the child message type use the parent message type as a prefix to unify the style. For example Figure 2 ORDER_RECEIVED, ORDER_RECEIVED_UPDATE, ORDER_RECEIVED_TOSAP, etc. of the order receipt transaction group implementation example of the TOB scenario.

[0070] The parent of the transaction group corresponds to a method, or called the root method, and each child corresponds to a method. The parent method calls each child method synchronously in the business order. All related methods in the transaction group must be annotated with the message transaction AOP aspect annotation set by the present invention, and the message type must be defined on the message type attribute of the corresponding method annotation.

[0071] All related methods in the transaction group must input a message provider object parameter, and the message provider object parameter must implement a message provider interface set by the present invention. The interface defines four methods that must be implemented by subclasses to obtain message messages, message types, message IDs and business keywords.

[0072] If the transaction group involves a cluster multi-node deployment architecture, it is also necessary to annotate the additional information provided by the present invention on the transaction group root method. Figure 3 The S340 distributed lock module annotation is used to avoid concurrent calls on multiple nodes.

[0073] After the program is defined as agreed, when a new message is received, the device of the present invention will be triggered. The device first determines whether the distributed lock of the device of the present invention is annotated. If so, it tries to obtain the distributed lock through the specified lock key. If the lock acquisition fails, an error is reported and the process ends. If the lock acquisition succeeds, the process continues to execute.

[0074] Based on the AOP aspect, the message provider object in the current execution method is obtained. If the object cannot be obtained, an error is reported and the log prints to inform that it does not meet the defined requirements. If it is obtained, it continues. In the subsequent stages, the core field content can be obtained through the message provider object: message message, message type, message ID and key business key.

[0075] Please refer to Figure 1 , S110 parent begin unit, including the steps:

[0076] Determine whether the current thread has a transaction group context. If not, create a message transaction group context for the current thread and create an empty first-in-last-out method stack for the context.

[0077] The message type and message ID are obtained through the message provider object of the current method, and then queried from the message log table. If the message log table already exists, it is converted to a message log object based on the records existing in the database. At this time, the processing status of the message log object is based on the last processing result of the database. The message log object will be marked as non-new, and it will be determined whether the processing status of the record is failed. If it is a failure, it will be marked as retryable. If the message log table does not exist, a new message log object is created based on the message provider object of the current method. The processing status of the newly created message log object defaults to failure, which is equivalent to pending processing.

[0078] The message log object corresponding to the current method is further encapsulated into a stack frame and pushed into the context of the current transaction group.

[0079] Determine whether the number of current context method stacks = 1. If so, create and start a parent method local transaction.

[0080] Please refer to Figure 1 , S120 parent doWork unit, including the steps:

[0081] The method stack corresponding to the current method is obtained from the transaction group context, and the message log object prepared by the parent begin unit in S110 described above is extracted from the method stack. The idempotence is determined by whether the processing status of the message log object is successful.

[0082] If the processing status is successful, it is considered idempotent and the subsequent actual business logic of the current method will not be executed. At the same time, the message message of the current message log object will be parsed into the next message provider parameter object as the input parameter object for calling the next sub-method. The purpose of this is to resolve the dependency between the previous and next parameters between multiple sub-method calls. For example, sub-method B needs to use the data processed by the previous sub-method A.

[0083] If the processing status is failure, it is considered non-idempotent. The new creation flag of the message log object created by the parent begin unit in S110 is further judged. If the flag is new, the message log object is saved to the database message log table. If it is not new, the number of retries is accumulated and the message log object is updated to the message log table.

[0084] If the processing status is failure, it is considered non-idempotent. Finally, whether it is the parent is determined by whether the number of method stacks = 1. If it is the parent, the actual business logic is called directly. If it is the child, the instructions of the child doWork are followed.

[0085] Further explanation: Since the parent and child units are designed to use the same set of method devices, each unit needs to determine whether it is a parent or a child by checking whether the number of method stacks = 1.

[0086] Further explanation: The actual business logic of the parent level is mainly the call of each child level method, and the acquisition and assembly of some basic configuration data before executing the child level method. The actual business logic of the child level is the real business processing of the message.

[0087] Please refer to Figure 1 , S200 sub-level define unit, including the steps:

[0088] The S200 child define unit implements the same logic as the previous S100 parent define unit.

[0089] Please refer to Figure 1 , S210 sub-level begin unit, including the steps:

[0090] The only difference between the S210 child begin unit and the previous S110 parent begin unit is that the parent is determined based on the number of method stacks. Only the parent will start a local transaction in this unit, and the child's local transaction is controlled by the S220 child doWork unit.

[0091] Further explanation of local transactions: the parent corresponds to a local transaction, and each child corresponds to an independent local transaction. After the child method is successfully processed, the corresponding child transaction directly commits the child transaction, and the processing status of the child's message log is set to success before committing. When all children are successfully completed, the parent transaction is committed, and the processing status of the parent's message log is set to success before committing.

[0092] Please refer to Figure 1 , S220 sub-level doWork unit, including the steps:

[0093] The only difference between the S220 child doWork unit and the previous S110 parent doWork unit is that before the actual business logic is actually executed, the child's own independent local transaction will be started first.

[0094] Please refer to Figure 1 , S230, if the execution of the actual business logic processing of the child is successful, the current child-level local transaction is committed in the child-level doWork unit in S220, and the child-level local transaction is completely controlled by the child-level doWork unit in S220.

[0095] If the execution of the actual business logic fails, the current child-level local transaction will be rolled back in the S220 child-level doWork unit, and an exception will be thrown to the outside. When any child fails to be executed, other children will no longer be executed, and the parent-level rollback unit S140 will be executed eventually. At this time, the processing status of the currently failed child and parent-level message log objects are both failed.

[0096] When the S140 parent rollback unit is finally reached, press Figure 3 The S300 automatic retry module performs 3 retries. If the 3 retries still fail, the configuration is triggered. Figure 3 The S320 intelligent warning module unit, and press Figure 3 The S330 operation and maintenance module unit provides a visual interface for manual intervention.

[0097] Please refer to Figure 1 , S240 sub-level commit unit, including the steps: if S220 sub-level doWork unit is executed successfully, the sub-level commit unit will be executed, which first obtains the transaction group context and then pops the current method stack frame.

[0098] Please refer to Figure 1 , S270 unit, identifies whether the current parent commit or child commit is determined by whether the number of method stacks = 0. If it is identified as a child commit, further determine whether the message log object in the stack frame currently popped from the method stack is a new or retry mark. If it is, update the message body of the message log object to the message log table. The update of the message message has been explained in the previous S120 parent doWork unit, which will not be repeated here. If it is identified as a parent commit, refer to the description of the parent commit unit below.

[0099] Additional note: Pushing into the stack will add a method stack frame object to the stack, and popping the stack will reduce a method stack frame object accordingly. The stack currently uses a first-in, last-out method, and the method currently popped out is the last method pushed into the stack.

[0100] Please refer to Figure 1 , S250 sub-level rollback unit, including the steps: if S220 sub-level doWork unit fails to execute, it will enter the sub-level rollback unit, which first obtains the transaction group context and then pops the current method stack frame.

[0101] Please refer to Figure 1 , S270 unit identifies whether the current parent rollback or child rollback is based on whether the number of method stacks is 0. If it is identified as a child rollback, the current method stack frame is popped. If it is identified as a parent rollback, refer to the description of the parent rollback unit below.

[0102] Please refer to Figure 1 , S260 sub-level clear unit, including the steps: no matter whether S220 sub-level doWork unit is executed successfully or failed, the sub-level clear unit will be executed.

[0103] Please refer to Figure 1 , S270 unit obtains the transaction group context, and identifies whether the current parent clear or child clear is by checking whether the method stack number is 0. If the identification is child clear, nothing is processed. If the identification is parent clear, refer to the description of the parent clear unit below.

[0104] Please refer to Figure 1, S130 parent commit unit, including the steps: if all children are executed successfully, at this time the message processing status of all children is successful, and no exception is thrown, then the parent commit unit will be executed, and the unit will commit the parent local transaction.

[0105] Please refer to Figure 1 , S140 parent rollback unit, including the steps: if any child fails to execute, the child will throw an exception and eventually enter the parent rollback unit, which will roll back the parent local transaction.

[0106] Please refer to Figure 1 , S150 parent clear unit, including the steps: the parent clear unit is a symbol of the end of the entire transaction group, and the unit ends the current transaction group by clearing the current transaction group context.

[0107] Next, Figure 3 The relevant unit functions are explained.

[0108] For the S300 unit, the automatic retry module provides configurable switches, timing frequency cron expressions, paging batch sizes, and retry counts, and automatically retries parent processes with a failed status according to the configuration.

[0109] For the S310 unit, the automatic cleanup module provides configurable switches, timing frequency cron expressions, cleanup time points and other options. According to the configuration, the child with a successful parent processing status is physically deleted, and the parent is still retained as an idempotent.

[0110] For the S320 unit, the intelligent warning module provides configurable switches, custom implementations of email, SMS or other notification methods, and notifies the parent rollback unit of abnormalities based on the configuration.

[0111] S330 unit, the visual operation and maintenance module provides API interfaces such as paging condition list query and manual retry of parent-child structure. The front end performs visual operation and maintenance by connecting to the API interface.

[0112] The S340 unit, the distributed lock module provides a distributed lock based on the Redis lock, which is used to control concurrent execution of multiple nodes in the cluster and ensure the idempotence requirements of multiple nodes.

[0113] Unit S350, the automatic table creation unit provides for automatically creating a related message log table when the device of the present invention is first introduced.

[0114] Next, combine Figure 2An embodiment is used to further clearly and completely illustrate the differences and beneficial effects of the method of the present invention compared with the previous technology. In this embodiment, the present invention provides a message queue consumer message processing method based on a transaction group. In the embodiment of the present invention, the TOB order receipt scenario is used as an example for explanation. The method processing flow described is as follows: Figure 2 As shown, the steps include:

[0115] First, define the order receipt message processing as a transaction group.

[0116] In terms of program design, the external call interface is divided into multiple sub-level calls, and the corresponding message type is defined for each sub-level:

[0117] The message types are defined as follows: parent root method, modify document receipt quantity, call SAP system to confirm receipt income, call points system to add receipt points, call settlement system to add receipt flow, call inventory system to add receipt flow.

[0118] When a new message is consumed from the message queue, the first root parent message log object is created, and the initial processing status is failure.

[0119] The parent method calls the child method in sequence, and each time the child is called, a corresponding child message log object is created, which corresponds to a record in the database message log table.

[0120] Each time a child call succeeds, the processing status is marked as successful. When all children are executed successfully, the processing status of all children and the parent becomes successful.

[0121] During the sequential execution process, if an exception occurs when executing a certain child, the exception is thrown externally, and only the current child and parent are marked as failed. The processing status of the previously successfully executed child remains successful.

[0122] The advantages and differences of the transaction group method of the present invention compared with other traditional implementation methods are:

[0123] (1) The present invention splits a large message processing into multiple small sub-levels, each of which has an independent processing status. When the parent level calls the sub-levels in sequence, if the sub-level fails to execute in the middle, the subsequent retry program will skip the successful sub-level and only retry the sub-level with a failed processing status, which greatly improves the retry efficiency and reduces the service resource pressure.

[0124] (2) The present invention can support the situation where the downstream system does not implement idempotence, because the successfully executed child external call will not be executed again during the next retry.

[0125] (3) The present invention implements a set of standard consumer definitions based on transaction groups, supports automatic retry, intelligent early warning, visual operation and maintenance, automatic cleanup, automatic table creation, distributed locks and other units, completely simplifying development and improving development efficiency.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A message queue consumer processing method based on transaction group, characterized in that: include: Define order receipt message processing as a transaction group; According to the external call interface dimension, it is divided into multiple sub-level calls, and the corresponding message type is defined for each sub-level. The message type definitions are as follows: parent root method, modify document receipt quantity, call SAP system to confirm receipt income, call the points system to add receipt points, call the settlement system to add receipt flow, call the inventory system to add receipt flow; When a new message is consumed from the message queue, the first root parent message log object is created, and the initial processing status is failure; The parent method calls the child method in sequence, and each time the child method is called, a corresponding child message log object is created, which corresponds to a record in the database message log table. Each time a child is successfully called, the processing status is marked as successful. When all children are successfully executed, the processing status of all children and the parent becomes successful. During the sequential execution process, if an exception occurs in a child, the exception is thrown externally, and only the current child and parent are marked as failed. The processing status of the previously successfully executed child remains successful. The method first enters the parent module, and the parent module executes the following steps in sequence: Step S100, executing the parent define unit to define the consumption interface; Step S110, executing the parent begin unit to start a message transaction, create a transaction group context, and push it into a stack; Step S120, execute the parent doWork unit to execute the parent business method. After step S120, enter the child module and execute the following parent units respectively according to the results of the child module: Step S130, executing the parent commit unit, which is used to commit the transaction and pop the stack when the business method is successfully executed; Step S140, executing the parent rollback unit, which is used to roll back the transaction and pop the stack when the business method fails to execute; Step S150, executing the parent clear unit to clear the transaction group context; Wherein, step S100 includes: Define a set of message type enumerations for a message consumption transaction group; The parent of the transaction group corresponds to a method, or called the root method, and each child corresponds to a method. The parent method calls each child method synchronously in the business order. All related methods in the transaction group must be annotated with the set message transaction AOP aspect annotation, and the message type must be defined on the message type attribute of the corresponding method annotation; All related methods in the transaction group must take a message provider object parameter as a parameter. The message provider object parameter must implement a set message provider interface. The interface defines four methods that must be implemented by subclasses to obtain message content, message type, message ID, and business keywords. If the transaction group involves a cluster multi-node deployment architecture, you also need to annotate the transaction group root method with a distributed lock module annotation to avoid concurrent calls on multiple nodes. After the program is defined, when a new message is received, it will be triggered to first determine whether the distributed lock is annotated. If so, it will try to obtain the distributed lock through the specified lock key. If the lock acquisition fails, an error will be reported and the execution will continue if the lock acquisition succeeds. Based on the AOP aspect, the message provider object in the current execution method is obtained. If the object cannot be obtained, an error is reported and the log is printed to inform that it does not meet the defined requirements. If the object is obtained, the method continues. In the subsequent stages, the core field content is obtained through the message provider object: message body, message type, message ID and key business key.

2. The message queue consumer processing method based on transaction group implementation according to claim 1 is characterized in that: The child module performs the following steps: Step S200, executing the sub-level define unit, which is used to define the consumption interface as required, and executing step S210 after step S200; Step S210, executing the sub-level begin unit, which is used to start the message transaction, create the transaction group context, and push it into the stack. After step S210, step S220 is executed; Step S220, executing the sub-level doWork unit, which is used to execute the sub-level business method, and then executing step S230; Step S230, determining whether the execution of the business method is successful, if successful, executing step 240, if unsuccessful, executing step S250; Step S240, executing the sub-level commit unit to commit the transaction and pop the stack; Step S250, executing the sub-level rollback unit to roll back the transaction and pop the stack, and executing step S260 after step S250; Step S260, executing the sub-level clear unit to clear the transaction group context, and executing step S270 after step S260; Step S270, determine whether it is the last sub-method, if yes, execute step 310, otherwise return to step 200.

3. The message queue consumer processing method based on transaction group implementation according to claim 1 is characterized in that: Step S110 includes: Determine whether the current thread has a transaction group context. If not, create a message transaction group context for the current thread and create an empty first-in-last-out method stack for the context. The message type and message ID are obtained through the message provider object of the current method, and then queried from the message log table; if the message log table already exists, it is converted into a message log object based on the records existing in the database. At this time, the processing status of the message log object is based on the last processing result of the database. The message log object will be marked as non-new, and it will be determined whether the processing status of the record is failed. If it is failed, it will be marked as retryable; if the message log table does not exist, a new message log object is created based on the message provider object of the current method. The processing status of the newly created message log object defaults to failure, which is equivalent to pending processing; The message log object corresponding to the current method is further encapsulated into a stack frame and pushed into the context of the current transaction group; Determine whether the number of current context method stacks = 1. If so, create and start a parent method local transaction.

4. The message queue consumer processing method based on transaction group implementation according to claim 3 is characterized in that: Step S120 includes: Obtain the method stack corresponding to the current method from the transaction group context, extract the message log object prepared by the parent begin unit in S110 from the method stack, and determine the idempotence by whether the processing status of the message log object is successful; If the processing status is successful, it is considered idempotent and the subsequent actual business logic of the current method will not be executed. At the same time, the message message of the current message log object will be parsed into the next message provider parameter object as the input parameter object of the message provider for calling the next sub-method; If the processing status is failure, it is considered non-idempotent, and the new identification of the message log object created by the parent begin unit in S110 is further determined. If the identification is new, the message log object is saved to the database message log table. If it is not new, the number of retries is accumulated and the message log object is updated to the message log table. If the processing status is failure, it is considered non-idempotent. Finally, whether it is the parent is determined by whether the number of method stacks = 1. If it is the parent, the actual business logic is called directly. If it is the child, follow the instructions for the child doWork below.

5. The message queue consumer processing method based on transaction group implementation according to claim 2 is characterized in that: The only difference between the child begin unit in step S210 and the parent begin unit in the previous S110 is that the parent is determined based on the number of method stacks. Only the parent will start a local transaction in this unit, while the child's local transaction is controlled by the child doWork unit in S220. The only difference between the doWork unit in the child in step S220 and the parent doWork unit in S110 is that before the actual business logic is actually executed, the child's own independent local transaction will be started first; In step S230, if the execution of the sub-level actual business logic processing is successful, the sub-level doWork unit of S220 commits the current sub-level local transaction, and the sub-level local transaction is completely controlled by the sub-level doWork unit of S220; If the execution of the actual business logic fails, the S220 child doWork unit rolls back the current child local transaction and throws an exception. When any child fails, other children will not be executed, and the S140 parent rollback unit will be executed eventually. At this time, the processing status of the currently failed child and parent message log objects are both failed. When the S140 parent rollback unit is finally executed, the automatic retry module will be used to retry three times. If the three retries still fail, the intelligent early warning module unit of the automatic retry module will be triggered according to the configuration, and the operation and maintenance module unit of the automatic retry module will provide a visual interface for manual intervention.

6. The message queue consumer processing method based on transaction group implementation according to claim 5 is characterized in that: Step S270 also identifies whether the current parent commit or child commit is determined by whether the method stack quantity is 0. If it is identified as a child commit, it further determines whether the message log object in the stack frame currently popped from the method stack is a new or retry flag. If so, the message body of the message log object is updated to the message log table. If the identification is a parent commit, the parent commit unit is executed.

7. The message queue consumer processing method based on transaction group implementation according to claim 6 is characterized in that: Step S270 also identifies whether the current parent rollback or child rollback is determined by whether the method stack quantity is 0. If it is a child rollback, the current method stack frame is popped; if it is a parent rollback, the parent rollback unit is executed.

8. The message queue consumer processing method based on transaction group implementation according to claim 5 is characterized in that: The automatic retry mode performs the following steps in sequence: Step S310, through the automatic cleaning module, configurable switches, timing frequency cron expressions, and cleaning time point options are provided, and the child with the parent processing status as successful is physically deleted according to the configuration, and the parent is still retained as idempotent; Step S320, through the intelligent early warning module, a configurable switch is provided, and email, SMS or other notification methods are customized to be implemented, and abnormal notification is made to the parent rollback unit according to the configuration; Step S330: Provide a paging condition list query and manual retry API interface for parent-child structure through the visual operation and maintenance module, and the front end performs visual operation and maintenance through the docking API interface; Step S340, through the distributed lock module, a distributed lock based on Redis lock is provided, which is used to control concurrent execution of multiple nodes in the cluster and ensure the idempotence requirements of multiple nodes; Step S350: Automatically create a related message log table when first introduced through an automatic table creation unit.

Images