Event processing method and device based on Kafka and electronic equipment

Abstract

The invention provides a Kafka-based event processing method and device and electronic equipment, and relates to the technical field of computers. In the application, in a synchronous processing process of at least one synchronous processing event corresponding to a target service, at least one asynchronous processing event corresponding to the target service is preprocessed to obtain at least one first message, and the at least one first message is pushed to a first message queue in Kafka; determining second messages respectively corresponding to the at least one asynchronous processing event, and pushing a target second message meeting a preset asynchronous processing condition in the at least one second message to a second message queue in Kafka; and performing asynchronous processing on an asynchronous processing event corresponding to the target second message included in the second message queue. Through the double message queues, the consistency of the Kafka on the asynchronous processing event processing flow in the business processing process is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to an event processing method, apparatus, and electronic device based on Kafka. Background Art

[0002] As a distributed persistent message queue system, Kafka has characteristics such as high throughput, scalability, and fault tolerance, and is suitable for processing a large number of concurrent asynchronous processing events (or asynchronous command requests). By combining Kafka with asynchronous processing event handling, asynchronous message processing, high concurrency support, high availability, etc. can be effectively achieved.

[0003] However, for business scenarios with a large concurrency and requiring asynchronous processing (such as financial business scenarios, etc.), Kafka is difficult to ensure the consistency of the asynchronous processing event handling process during business processing. For example, for a certain processing node in Kafka, it is impossible to determine whether multiple asynchronous processing events included in Kafka have been completely processed. Therefore, how to ensure the consistency of the asynchronous processing event handling process is an urgent problem to be solved currently. Summary of the Invention

[0004] Embodiments of this application provide an event processing method, apparatus, and electronic device based on Kafka to ensure the consistency of the asynchronous processing event handling process by Kafka during business processing.

[0005] In a first aspect, embodiments of this application provide an event processing method based on Kafka, and the method includes:

[0006] Receiving a business processing request of a target object for a target business, and determining at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business based on the business processing request;

[0007] During the synchronous processing of at least one synchronous processing event, preprocessing at least one asynchronous processing event to obtain at least one first message, and pushing at least one first message to a first message queue in Kafka; where each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed;

[0008] Determining second messages respectively corresponding to at least one asynchronous processing event, and pushing target second messages that meet a preset asynchronous processing condition among at least one second message to a second message queue in Kafka; where a target second message is used to indicate that the target object needs network services provided by the corresponding asynchronous processing event;

[0009] Asynchronously processing the asynchronous processing events corresponding to the target second messages included in the second message queue.

[0010] In an alternative embodiment, preprocessing at least one asynchronous processing event to obtain at least one first message, including:

[0011] Performing the following operations respectively for at least one asynchronous processing event:

[0012] Responding to the start transaction operation for the first asynchronous processing event; wherein, the first asynchronous processing event is any one of the at least one asynchronous processing event;

[0013] Preprocessing the first asynchronous processing event to obtain the first message corresponding to the first asynchronous processing event.

[0014] In an alternative embodiment, determining second messages respectively corresponding to at least one asynchronous processing event, including:

[0015] Performing the following operations respectively for at least one asynchronous processing event:

[0016] Responding to the commit transaction operation for the second asynchronous processing event, and obtaining the commit result of the second asynchronous processing event; wherein, the second asynchronous processing event is any one of the at least one asynchronous processing event, and the commit result is successful commit or failed commit;

[0017] Determining the second message corresponding to the second asynchronous processing event based on the commit result.

[0018] In an alternative embodiment, determining the second message corresponding to the second asynchronous processing event based on the commit result, including:

[0019] If the commit result is successful commit, determining that the second message corresponding to the second asynchronous processing event is a confirmation message; the confirmation message is used to indicate that asynchronous processing of the second asynchronous processing event is required;

[0020] If the commit result is failed commit, determining that the second message corresponding to the second asynchronous processing event is a cancellation message; the cancellation message is used to indicate that asynchronous processing of the second asynchronous processing event is not required.

[0021] In an alternative embodiment, after determining that the second message corresponding to the second asynchronous processing event is a cancellation message, further including:

[0022] Performing a rollback transaction process for the preprocessing operation of the second asynchronous processing event.

[0023] In an alternative embodiment, pushing target second messages that meet a preset asynchronous processing condition among at least one second message to a second message queue in Kafka, including:

[0024] Perform second message pairing on at least one first message in the first message queue to obtain at least one pairing result;

[0025] If there is a first pairing result indicating successful pairing within a set time range among at least one pairing result, when it is determined that the second message associated with the first pairing result meets the asynchronous processing condition, the second message associated with the first pairing result is pushed to the second message queue as the target second message.

[0026] In an alternative embodiment, determining that the second message associated with the first pairing result meets the asynchronous processing condition includes:

[0027] If the second message associated with the first pairing result is a confirmation message, it is determined that the second message associated with the first pairing result meets the asynchronous processing condition; wherein, the confirmation message is used to indicate that asynchronous processing of the asynchronous processing event corresponding to the second message associated with the first pairing result needs to be performed.

[0028] In an alternative embodiment, the method further includes:

[0029] If there is a second pairing result indicating unsuccessful pairing within a set time range among at least one pairing result, query the submission result of the asynchronous processing event corresponding to the second pairing result;

[0030] When it is determined that the submission result is successful submission, the second message associated with the second pairing result is pushed to the second message queue as the target second message.

[0031] In a second aspect, an embodiment of the present application further provides an event processing device based on Kafka, and the device includes:

[0032] A request receiving module, configured to receive a service processing request of a target object for a target service, and determine at least one synchronous processing event and at least one asynchronous processing event corresponding to the target service based on the service processing request;

[0033] A first pushing module, configured to preprocess at least one asynchronous processing event during the synchronous processing of at least one synchronous processing event to obtain at least one first message, and push at least one first message to the first message queue in Kafka; wherein, each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed;

[0034] A second pushing module, configured to determine second messages corresponding to at least one asynchronous processing event respectively, and push the target second messages that meet the preset asynchronous processing conditions among at least one second message to the second message queue in Kafka; wherein, the target second message is used to indicate that the target object needs the network service provided by the corresponding asynchronous processing event;

[0035] An asynchronous processing module for asynchronously processing asynchronous processing events corresponding to target second messages included in a second message queue.

[0036] In an optional embodiment, when preprocessing at least one asynchronous processing event to obtain at least one first message, the first push module is specifically configured to:

[0037] For at least one asynchronous processing event, perform the following operations respectively:

[0038] Respond to the start transaction operation for the first asynchronous processing event; wherein, the first asynchronous processing event is any one of the at least one asynchronous processing event;

[0039] Preprocess the first asynchronous processing event to obtain a first message corresponding to the first asynchronous processing event.

[0040] In an optional embodiment, when determining second messages respectively corresponding to at least one asynchronous processing event, the second push module is specifically configured to:

[0041] For at least one asynchronous processing event, perform the following operations respectively:

[0042] Respond to the commit transaction operation for the second asynchronous processing event, and obtain the commit result of the second asynchronous processing event; wherein, the second asynchronous processing event is any one of the at least one asynchronous processing event, and the commit result is commit success or commit failure;

[0043] Determine the second message corresponding to the second asynchronous processing event based on the commit result.

[0044] In an optional embodiment, when determining the second message corresponding to the second asynchronous processing event based on the commit result, the second push module is specifically configured to:

[0045] If the commit result is commit success, determine that the second message corresponding to the second asynchronous processing event is a confirmation message; the confirmation message is used to indicate that the second asynchronous processing event needs to be asynchronously processed;

[0046] If the commit result is commit failure, determine that the second message corresponding to the second asynchronous processing event is a cancellation message; the cancellation message is used to indicate that the second asynchronous processing event does not need to be asynchronously processed.

[0047] In an optional embodiment, after determining that the second message corresponding to the second asynchronous processing event is a cancellation message, the second push module is further configured to:

[0048] Perform a rollback transaction process on the preprocessing operation for the second asynchronous processing event.

[0049] In an alternative embodiment, when pushing a target second message that meets the preset asynchronous processing condition in at least one second message to the second message queue in Kafka, the second push module is specifically configured to:

[0050] Perform second message pairing on at least one first message in the first message queue to obtain at least one pairing result;

[0051] If there is a first pairing result indicating successful pairing within a set time range in at least one pairing result, when it is determined that the second message associated with the first pairing result meets the asynchronous processing condition, push the second message associated with the first pairing result to the second message queue as the target second message.

[0052] In an alternative embodiment, when it is determined that the second message associated with the first pairing result meets the asynchronous processing condition, the second push module is specifically configured to:

[0053] If the second message associated with the first pairing result is a confirmation message, determine that the second message associated with the first pairing result meets the asynchronous processing condition; wherein, the confirmation message is used to indicate that asynchronous processing of the asynchronous processing event corresponding to the second message associated with the first pairing result needs to be performed.

[0054] In an alternative embodiment, the second push module is further configured to:

[0055] If there is a second pairing result indicating unsuccessful pairing within a set time range in at least one pairing result, query the submission result of the asynchronous processing event corresponding to the second pairing result;

[0056] When it is determined that the submission result is successful submission, push the second message associated with the second pairing result to the second message queue as the target second message.

[0057] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0058] A processor; and

[0059] A memory storing a program,

[0060] wherein the program includes instructions that, when executed by the processor, cause the processor to execute the Kafka-based event processing method as described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the Kafka-based event processing method as described in the first aspect.

[0062] Fifth aspect, the present application provides a computer program product, which, when called by a computer, causes the computer to execute the steps of the Kafka-based event processing method as described in the first aspect.

[0063] The beneficial effects of the present application are as follows:

[0064] In the Kafka-based event processing method provided by the embodiments of the present application, a business processing request of a target object for a target business is received, and based on the business processing request, at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business are determined; then, during the synchronous processing of at least one synchronous processing event, at least one asynchronous processing event is preprocessed to obtain at least one first message, and at least one first message is pushed to a first message queue in Kafka; wherein each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed; further, second messages respectively corresponding to at least one asynchronous processing event are determined, and target second messages that meet a preset asynchronous processing condition among at least one second message are pushed to a second message queue in Kafka; wherein the target second message is used to indicate that the target object requires the network service provided by the corresponding asynchronous processing event; finally, the asynchronous processing events corresponding to the target second messages included in the second message queue are asynchronously processed. In this way, through the first message queue and the second message queue, the consistency of the processing flow of asynchronous processing events during business processing by Kafka is ensured.

[0065] In addition, other features and advantages of the present application will be described in the subsequent specification, and, in part, will be obvious from the specification, or will be understood by implementing the present application. The objectives and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Description of the Drawings

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application and constitute a part of the present application, without unduly limiting the present application. In the drawings:

[0067] Figure 1 It is a schematic flowchart of the implementation process of a Kafka-based event processing method provided by an embodiment of the present application;

[0068] Figure 2 It is a schematic logical diagram of a Kafka-based event processing provided by an embodiment of the present application;

[0069] Figure 3Schematic diagram of the implementation process of a method for confirming a second message provided by an embodiment of the present application;

[0070] Figure 4 Schematic diagram of a specific scenario of event processing based on Kafka provided by an embodiment of the present application;

[0071] Figure 5 Schematic diagram of the structure of an event processing device based on Kafka provided by an embodiment of the present application;

[0072] Figure 6 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0073] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0074] It should be understood that the various steps recorded in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this regard.

[0075] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence relationship of the functions performed by these devices, modules or units.

[0076] It should be noted that the modifications of "one" and "multiple" mentioned in the present application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0077] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0078] First, the design concept of the embodiments of the present application will be briefly introduced as follows:

[0079] Kafka is a distributed and reliable messaging system based on the publish / subscribe model and can be used to process real-time and streaming data. Moreover, as a distributed persistent message queue system, Kafka has characteristics such as high throughput, scalability, and fault tolerance, and is suitable for processing a large number of concurrent asynchronous processing events.

[0080] Therefore, the open-source Kafka can usually be used as an asynchronous event or message processing system. However, although the asynchronous event processing of Kafka has good practicability for most business processing scenarios with a relatively low concurrency volume, for business scenarios with a large concurrency volume and requiring asynchronous processing (such as financial business scenarios, etc.), it cannot guarantee the consistency of the asynchronous processing event processing process or processing flow, and thus cannot better meet the requirements of the business scenario.

[0081] In view of this, to solve or improve the above problems and ensure the consistency of the asynchronous processing event processing flow during the business processing of Kafka. Refer to Figure 1 As shown, it is a schematic diagram of the implementation process of an event processing method based on Kafka provided by the embodiments of the present application. Taking Kafka as an example of the execution subject, the specific implementation process of this method is as follows:

[0082] S101: Receive a business processing request of a target object for a target business, and based on the business processing request, determine at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business.

[0083] Taking the target business as services A and B required by the target object as an example, after Kafka receives the business processing request of the target object for the target business, it can determine that the processing events corresponding to services A and B respectively are the synchronous processing events corresponding to the target business, and other processing events recommended to the target user can be used as asynchronous processing events. Among them, the synchronous processing event is a processing event that needs to be responded to in a timely manner to meet the business processing request of the target object, and the asynchronous processing event is some secondary processing events with a relatively low requirement for the real-time response of event processing.

[0084] The target business can be a business in any business processing scenario with a large concurrency volume, and the embodiments of the present application do not make specific limitations in this regard. Exemplarily, the target business can be a financial business required by the target object, and the aforementioned business processing scenario with a large concurrency volume can be a business scenario deployed by microservices. For a financial system with a large concurrent business volume and requiring asynchronous processing, especially when it is necessary to ensure the consistency of the asynchronous processing event processing flow.

[0085] S102: During the synchronization process of at least one synchronization processing event, preprocess at least one asynchronous processing event to obtain at least one first message, and push the at least one first message to the first message queue in Kafka.

[0086] Each first message can be used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed. Therefore, the first message can also be called a pre- (processing) message. Of course, there can be other names, and the embodiments of this application do not limit this.

[0087] Exemplarily, referring to Figure 2 As shown, when executing step S102, after Kafka determines at least one synchronization processing event and at least one asynchronous processing event corresponding to the target service based on the service processing request, the financial service module in Kafka can perform synchronization processing on the at least one synchronization processing event, and the consumption service module in Kafka can perform asynchronous processing on the at least one asynchronous processing event.

[0088] In an alternative implementation, when Kafka preprocesses at least one asynchronous processing event to obtain at least one first message, for any one of the at least one asynchronous processing events, such as the first asynchronous processing event, the following operations can be performed: In response to the start transaction operation for the first asynchronous processing event, preprocess the first asynchronous processing event to obtain the first message corresponding to the first asynchronous processing event.

[0089] It should be understood that in database operations, since an event (or transaction) is a set of operations, these operations are either all successful or all failed. And starting a transaction or commencing a transaction usually marks the start of an event by a specific command or annotation before performing a series of database operations.

[0090] Based on the above method, it can be quickly determined whether the preprocessing of the asynchronous processing event has been completed, that is, whether it is ready to perform asynchronous processing on the asynchronous processing event, by whether the first message corresponding to the asynchronous processing event is obtained.

[0091] Furthermore, after Kafka obtains the first messages corresponding to the at least one asynchronous processing event respectively, it can push the obtained at least one first message to the first message queue in Kafka. It should be understood that the aforementioned first message queue can also be called a pre-message queue, which is used to record the first messages corresponding to the at least one asynchronous processing event respectively.

[0092] S103: Determine the second messages corresponding to at least one asynchronous processing event respectively, and push the target second messages that meet the preset asynchronous processing conditions among the at least one second messages to the second message queue in Kafka.

[0093] Among them, the target second message can be used to indicate that the target object requires the network service provided by the corresponding asynchronous processing event.

[0094] Specifically, when executing step S103, after Kafka obtains the first messages corresponding to the above at least one asynchronous processing event respectively, that is, after determining that the first messages corresponding to the above at least one asynchronous processing event are saved in the preset first message queue in Kafka, it can start to obtain the second messages for determining whether to perform asynchronous processing on the above at least one asynchronous processing event, that is, determine whether to perform asynchronous processing on the above at least one asynchronous processing event.

[0095] In an optional implementation manner, after Kafka saves the first messages corresponding to the above at least one asynchronous processing event respectively to the preset first message queue in Kafka, for any one of the at least one asynchronous processing event corresponding to the above, such as, the second asynchronous processing event, refer to Figure 3 as shown, the following operations can be performed:

[0096] S301: In response to the commit transaction operation for the second asynchronous processing event, obtain the commit result of the second asynchronous processing event.

[0097] Among them, the above commit transaction means officially recording all the executed operations (such as the preprocessing operations for the second asynchronous processing event) in the transaction (such as the second asynchronous processing event) to the database to make these changes permanent. Usually, the commit transaction is performed after all the operations (i.e., preprocessing operations) of the transaction are successfully executed.

[0098] And the commit result of the second asynchronous processing event can be that the commit transaction corresponding to the second asynchronous processing event is successful or the transaction commit corresponding to the second asynchronous processing event fails. That is, the above commit result is commit success or commit failure.

[0099] S302: Determine the second message corresponding to the second asynchronous processing event based on the commit result.

[0100] In an alternative implementation, when step S302 is executed, if the submission result of the second asynchronous processing event is successful submission, Kafka can determine that the second message corresponding to the second asynchronous processing event is a confirmation message. The confirmation message can be used to indicate that the second asynchronous processing event needs to be asynchronously processed. That is, the target object requires the network service provided by the second asynchronous processing event. If the submission result of the above-mentioned second asynchronous processing event is submission failure, Kafka can determine that the second message corresponding to the second asynchronous processing event is a cancellation message. The cancellation message can be used to indicate that there is no need to asynchronously process the second asynchronous processing event. That is, the target object does not require the network service provided by the second asynchronous processing event.

[0101] Based on the confirmation method of the second message described in the above steps S301 - S302, Kafka can quickly determine the specific content of the second message according to the target object's demand for network services (that is, whether it requires the network service provided by the second asynchronous processing event), and then determine whether it is necessary to perform subsequent asynchronous processing on the second asynchronous processing event.

[0102] Furthermore, after Kafka determines that the second message corresponding to the second asynchronous processing event is a cancellation message, it can also perform a rollback transaction process on the preprocessing operation of the second asynchronous processing event. The aforementioned rollback transaction generally refers to when an operation in a transaction fails, canceling all the executed operations in the transaction and restoring the database to the state before the transaction starts to be processed (i.e., preprocessing). It is an important part of transaction management and is used to ensure data consistency.

[0103] In this way, when Kafka determines that there is no need to asynchronously process the second asynchronous processing event, it can perform a transaction rollback operation on the second asynchronous processing event in the database, thereby releasing the storage space or computing resources occupied by the preprocessing of the second asynchronous processing event, and further reducing the load of Kafka.

[0104] To improve the event processing efficiency of one or more asynchronous processing events that need to be asynchronously processed subsequently, Kafka can push the confirmation messages (i.e., the second messages) corresponding to one or more asynchronous processing events that need to be asynchronously processed to the second message queue in Kafka. And after Kafka completely pushes the confirmation messages (i.e., the second messages) corresponding to one or more asynchronous processing events that need to be asynchronously processed to the second message queue in Kafka, it then pushes at least one confirmation message (i.e., the second message) included in the aforementioned second message queue to the consumption service module, and then asynchronously processes the asynchronous processing events corresponding to the at least one confirmation message.

[0105] After Kafka determines the second messages corresponding to the at least one asynchronous processing event described above, it can combine the at least one first message saved in the first message queue to decide whether to start asynchronous processing of the asynchronous processing events corresponding to the at least one confirmation message (i.e., the second message) included in the second message queue.

[0106] In an alternative implementation, after Kafka determines the second messages corresponding to the at least one asynchronous processing event described above, it can perform second message pairing on the at least one first message in the first message queue to obtain at least one pairing result. If there is a first pairing result indicating successful pairing within a set time range among the at least one pairing result described above, when it is determined that the second message associated with the first pairing result meets the asynchronous processing condition, the second message associated with the first pairing result is pushed to the second message queue as the target second message. Among them, the set time range described above can be several milliseconds or several seconds, and the embodiments of the present application do not make specific limitations in this regard.

[0107] Exemplarily, the asynchronous processing condition described above can be: the second message is a confirmation message. That is, if the second message associated with the first pairing result described above is a confirmation message, then Kafka can determine that the second message associated with the first pairing result meets the asynchronous processing condition described above. Among them, the fact that the second message associated with the first pairing result described above is a confirmation message can be used to indicate that the asynchronous processing event corresponding to the second message associated with the first pairing result needs to be asynchronously processed.

[0108] Optionally, if there is a second pairing result indicating unsuccessful pairing within a set time range among the at least one pairing result described above, the submission result of the asynchronous processing event corresponding to the second pairing result can be queried. When it is determined that the submission result is submission success, the second message associated with the second pairing result is pushed to the second message queue as the target second message.

[0109] Based on the above method, once the first message fails to successfully match the second message within the set time range, the submission result of the corresponding asynchronous processing event can be actively queried, so as to quickly determine whether to push the second message of the asynchronous processing event to the second message queue as the target second message, thereby avoiding the situation where the second message is not pushed in time due to network problems of Kafka, etc., and further affecting the processing efficiency of subsequent asynchronous processing events.

[0110] S104: Asynchronously process the asynchronous processing events corresponding to the target second messages included in the second message queue.

[0111] Exemplarily, when executing step S104, after Kafka obtains the target second message that meets the preset asynchronous processing conditions, that is, after saving the target second message that meets the preset asynchronous processing conditions to the preset second message queue in Kafka, the target second message included in the second message queue can be pushed to the financial service module in sequence, so that the financial service module performs asynchronous processing on the asynchronous processing event corresponding to the target second message.

[0112] Based on the Kafka-based event processing method described in the above steps S101 to S104, refer to Figure 4 As shown, it is a schematic diagram of a specific scenario of Kafka-based event processing provided by an embodiment of the present application. Kafka can include a financial service module and a consumption service module. The financial service module can receive a business processing request of a target object for a target business, start a transaction of the database, and complete all business operations that need to be completed for the target business, which can include database persistent reading and writing, calling a financial asynchronous message system to push asynchronous messages, and committing (local) transactions. The pre-message queue (i.e., the first message queue) in Kafka can receive various communication messages sent by the financial service module, such as the first message (i.e., the pre-message) and the second message (such as, a confirmation message or a cancellation message). The confirmation message queue (i.e., the second message queue) in Kafka can receive the messages that have been confirmed from the pre-message queue (i.e., the second message that is a confirmation message), and can push the second message that is not a confirmation message to the consumption service module. After the consumption service module receives the target second message included in the confirmation message queue, it can perform asynchronous processing on the asynchronous processing event corresponding to the target second message.

[0113] As Figure 4 shown, the above pre-message queue can include 3 first messages (i.e., "No.1 Event 1 Preprocessing", "No.2 Event 2 Preprocessing", and "No.3 Event 3 Preprocessing") and 3 second messages (i.e., "No.1 Event 1 Confirmation Processing", "No.2 Event 2 Confirmation Processing", and "No.3 Event 3 Cancellation Processing"). The above confirmation message queue can include 2 second messages (i.e., "No.1 Event 1 Processing", "No.2 Event 2 Processing").

[0114] Through the above Kafka event processing method, the financial service module can complete the logic processing of this business (i.e., the target business), and record the relevant information of the target business in the storage layer. Then, an asynchronous message is sent. First, a preprocessing message is sent to the pre-queue. After the successful sending, the financial service module performs a local transaction commit. If the transaction commit is successful, a confirmation message is sent to the pre-message queue. If the transaction commit fails, a cancellation message is sent to the pre-message queue. In the pre-message queue, the message queue system will match the messages. If the pre-message matches the confirmation message, the corresponding confirmation message is pushed to the confirmation message queue for push processing. If the pre-message matches the cancellation message, then the push processing of the confirmation message for this pre-message will not be performed. If the pre-message cannot match the confirmation message or the cancellation message for a long time, then the message queue system can query whether the commit transaction operation for the corresponding asynchronous processing event has been completed in the financial service module. If it is confirmed that the commit transaction operation for the corresponding asynchronous processing event has been completed, then when it is determined that the corresponding second message is a confirmation message, the confirmation message can be pushed to the confirmation message queue, otherwise no push processing is performed.

[0115] In summary, in the Kafka-based event processing method provided in the embodiments of the present application, a business processing request of a target object for a target business is received, and based on the business processing request, at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business are determined; then, during the synchronous processing of the at least one synchronous processing event, the at least one asynchronous processing event is preprocessed to obtain at least one first message, and the at least one first message is pushed to a first message queue in Kafka; wherein each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed; further, a second message corresponding to each of the at least one asynchronous processing event is determined, and a target second message that meets a preset asynchronous processing condition in the at least one second message is pushed to a second message queue in Kafka; wherein the target second message is used to indicate that the target object needs the network service provided by the corresponding asynchronous processing event; finally, the asynchronous processing event corresponding to the target second message included in the second message queue is asynchronously processed. In this way, through the first message queue and the second message queue, the consistency of the asynchronous processing event processing flow in the process of Kafka business processing is ensured.

[0116] Further, based on the same technical concept, the embodiments of the present application provide a Kafka-based event processing device, and the Kafka-based event processing device is used to implement the above method flow of the embodiments of the present application. Exemplarily, referring to Figure 5 As shown, the Kafka-based event processing device 500 may include: a request receiving module 501, a first push module 502, a second push module 503, and an asynchronous processing module 504, where:

[0117] A request receiving module 501, configured to receive a service processing request of a target object for a target service, and based on the service processing request, determine at least one synchronous processing event and at least one asynchronous processing event corresponding to the target service;

[0118] A first pushing module 502, configured to preprocess at least one asynchronous processing event during the synchronous processing of at least one synchronous processing event, obtain at least one first message, and push at least one first message to a first message queue in Kafka; wherein each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed;

[0119] A second pushing module 503, configured to determine second messages corresponding to at least one asynchronous processing event respectively, and push target second messages that meet a preset asynchronous processing condition among at least one second message to a second message queue in Kafka; wherein the target second message is used to indicate that the target object requires network services provided by the corresponding asynchronous processing event;

[0120] An asynchronous processing module 504, configured to asynchronously process the asynchronous processing events corresponding to the target second messages included in the second message queue.

[0121] In an optional embodiment, when preprocessing at least one asynchronous processing event to obtain at least one first message, the first pushing module 502 is specifically configured to:

[0122] For at least one asynchronous processing event, respectively perform the following operations:

[0123] Respond to the operation of starting a transaction for the first asynchronous processing event; wherein the first asynchronous processing event is any one of at least one asynchronous processing event;

[0124] Preprocess the first asynchronous processing event to obtain a first message corresponding to the first asynchronous processing event.

[0125] In an optional embodiment, when determining second messages corresponding to at least one asynchronous processing event respectively, the second pushing module 503 is specifically configured to:

[0126] For at least one asynchronous processing event, respectively perform the following operations:

[0127] Respond to the operation of submitting a transaction for the second asynchronous processing event, and obtain the submission result of the second asynchronous processing event; wherein the second asynchronous processing event is any one of at least one asynchronous processing event, and the submission result is submission success or submission failure;

[0128] Determine the second message corresponding to the second asynchronous processing event based on the submission result.

[0129] In an alternative embodiment, when determining the second message corresponding to the second asynchronous processing event based on the submission result, the second push module 503 is specifically configured to:

[0130] If the submission result is successful submission, determine that the second message corresponding to the second asynchronous processing event is a confirmation message; the confirmation message is used to indicate that the second asynchronous processing event needs to be asynchronously processed;

[0131] If the submission result is failed submission, determine that the second message corresponding to the second asynchronous processing event is a cancellation message; the cancellation message is used to indicate that the second asynchronous processing event does not need to be asynchronously processed.

[0132] In an alternative embodiment, after determining that the second message corresponding to the second asynchronous processing event is a cancellation message, the second push module 503 is further configured to:

[0133] Perform a rollback transaction processing on the preprocessing operation of the second asynchronous processing event.

[0134] In an alternative embodiment, when pushing the target second message that meets the preset asynchronous processing condition in at least one second message to the second message queue in Kafka, the second push module is specifically configured to:

[0135] Perform second message pairing on at least one first message in the first message queue to obtain at least one pairing result;

[0136] If there is a first pairing result indicating successful pairing within the set time range in at least one pairing result, when determining that the second message associated with the first pairing result meets the asynchronous processing condition, push the second message associated with the first pairing result to the second message queue as the target second message.

[0137] In an alternative embodiment, when determining that the second message associated with the first pairing result meets the asynchronous processing condition, the second push module 503 is specifically configured to:

[0138] If the second message associated with the first pairing result is a confirmation message, determine that the second message associated with the first pairing result meets the asynchronous processing condition; wherein, the confirmation message is used to indicate that the asynchronous processing event corresponding to the second message associated with the first pairing result needs to be asynchronously processed.

[0139] In an alternative embodiment, the second push module 503 is further configured to:

[0140] If there is a second pairing result indicating unsuccessful pairing within the set time range in at least one pairing result, query the submission result of the asynchronous processing event corresponding to the second pairing result;

[0141] When it is determined that the submission result is successful submission, the second message associated with the second pairing result is pushed to the second message queue as the target second message.

[0142] Based on the descriptions of the above method embodiments and apparatus embodiments, an exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it is used to cause the electronic device to execute the method according to the embodiments of the present invention.

[0143] An embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is used to cause a computer to execute the method according to the embodiments of the present application when executed by a processor of the computer.

[0144] An embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program is used to cause a computer to execute the method according to the embodiments of the present application when executed by a processor of the computer.

[0145] Refer to Figure 6 As shown, the following will describe the block diagram of the electronic device 600 that can be used as a server or a client of the present application, which is an example of a hardware device applicable to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0146] As Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can execute various appropriate actions and processes according to the computer program stored in a read-only memory (ROM) 602 or the computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device capable of inputting information into the electronic device 600. The input unit 606 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device. The output unit 607 can be any type of device capable of presenting information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0148] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 executes the various methods and processes described above. For example, in some embodiments, the above-described Kafka-based event processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 can be configured to execute the above-described Kafka-based event processing method by any other suitable means (e.g., by means of firmware).

[0149] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0150] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] As used in the present application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus for providing machine instructions and / or data to a programmable processor (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)), including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0152] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs that run on the respective computers and have a client-server relationship to each other.

[0155] Also, it should be understood that the above-disclosed are only the preferred embodiments of the present application, and of course cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made in accordance with the claims of the present invention are still within the scope covered by the present application.

Claims

1. A Kafka-based event processing method, characterized in that: include: Receiving a business processing request from a target object for a target business, and determining, based on the business processing request, at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business; During the synchronous processing of the at least one synchronous processing event, preprocessing the at least one asynchronous processing event to obtain at least one first message, and pushing the at least one first message to a first message queue in Kafka; wherein each first message is used to indicate that the preprocessing operation of the corresponding asynchronous processing event has been completed; Determine the second messages corresponding to the at least one asynchronous processing event, and push the target second message that meets the preset asynchronous processing condition in the at least one second message to the second message queue in the Kafka; wherein the target second message is used to indicate that the target object needs the network service provided by the corresponding asynchronous processing event; Asynchronously process the asynchronous processing event corresponding to the target second message included in the second message queue.

2. The method according to claim 1, characterized in that The preprocessing of the at least one asynchronous processing event to obtain at least one first message includes: For the at least one asynchronous processing event, the following operations are performed respectively: In response to a start transaction operation for a first asynchronous processing event; wherein the first asynchronous processing event is any one of the at least one asynchronous processing event; The first asynchronous processing event is preprocessed to obtain a first message corresponding to the first asynchronous processing event.

3. The method according to claim 1, characterized in that The determining the second message corresponding to each of the at least one asynchronous processing events includes: For the at least one asynchronous processing event, the following operations are performed respectively: In response to the commit transaction operation for the second asynchronous processing event, obtaining a commit result of the second asynchronous processing event; wherein the second asynchronous processing event is any one of the at least one asynchronous processing event, and the commit result is a commit success or a commit failure; A second message corresponding to the second asynchronous processing event is determined based on the submission result.

4. The method according to claim 3, characterized in that The determining, based on the submission result, a second message corresponding to the second asynchronous processing event includes: If the submission result is successful submission, determining that the second message corresponding to the second asynchronous processing event is a confirmation message; the confirmation message is used to indicate that the second asynchronous processing event needs to be asynchronously processed; If the submission result is submission failure, it is determined that the second message corresponding to the second asynchronous processing event is a cancellation message; the cancellation message is used to indicate that there is no need to perform asynchronous processing on the second asynchronous processing event.

5. The method according to claim 4, characterized in that After determining that the second message corresponding to the second asynchronous processing event is a cancellation message, the method further includes: Rollback transaction processing is performed on the preprocessing operation of the second asynchronous processing event.

6. The method according to any one of claims 1 to 5, characterized in that The step of pushing the target second message that satisfies the preset asynchronous processing condition in at least one second message to the second message queue in the Kafka comprises: Pairing the at least one first message with a second message in the first message queue to obtain at least one pairing result; If there is a first pairing result indicating successful pairing within a set time range among the at least one pairing result, when it is determined that the second message associated with the first pairing result meets the asynchronous processing condition, the second message associated with the first pairing result is pushed to the second message queue as the target second message.

7. The method according to claim 6, characterized in that The determining that the second message associated with the first pairing result satisfies the asynchronous processing condition includes: If the second message associated with the first pairing result is a confirmation message, it is determined that the second message associated with the first pairing result meets the asynchronous processing condition; wherein the confirmation message is used to indicate that the asynchronous processing event corresponding to the second message associated with the first pairing result needs to be asynchronously processed.

8. The method according to claim 6, characterized in that The method further comprises: If there is a second pairing result in the at least one pairing result indicating that the pairing is not successful within the set time range, querying the submission result of the asynchronous processing event corresponding to the second pairing result; When it is determined that the submission result is successful, the second message associated with the second pairing result is pushed to the second message queue as the target second message.

9. An event processing device based on Kafka, characterized in that: include: A request receiving module, configured to receive a business processing request from a target object for a target business, and determine, based on the business processing request, at least one synchronous processing event and at least one asynchronous processing event corresponding to the target business; A first push module is used to pre-process the at least one asynchronous processing event during the synchronous processing of the at least one synchronous processing event, obtain at least one first message, and push the at least one first message to a first message queue in Kafka; wherein each first message is used to indicate that the pre-processing operation of the corresponding asynchronous processing event has been completed; A second push module is used to determine the second messages corresponding to the at least one asynchronous processing event, and push the target second message that meets the preset asynchronous processing condition in the at least one second message to the second message queue in the Kafka; wherein the target second message is used to indicate that the target object needs the network service provided by the corresponding asynchronous processing event; An asynchronous processing module is used to asynchronously process an asynchronous processing event corresponding to the target second message included in the second message queue.

10. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

Images