A method for implementing a dynamic business pipeline based on event-driven and Reactor pattern

Abstract

The present invention discloses a method for implementing a dynamic service pipeline based on event-driven and Reactor mode, including the following steps: a. Architecture design, where messages on the message middleware are processed through combination by the scheduling center and then corresponding task chains are executed; b. Dynamically configuring the service pipeline, dividing the task chain according to the data composition on the Topic, thereby forming a service pipeline, and different division methods form different service pipelines; c. Execution of the task chain, where the input requirements of the task chain match multiple types of data on multiple Topics on the message middleware, and after being combined into a complete input, the execution of the task chain is triggered. One task chain is associated with one input, and one input is combined by the scheduling center with the data on the message middleware. Different combination or division methods provide a method for dynamically configuring the service pipeline.

Description

Technical Field

[0001] The present invention belongs to the field of communication technologies, and particularly relates to a method for implementing a dynamic service pipeline based on event-driven and Reactor mode. Background Art

[0002] With the rapid development of modern communication technologies, computer network technologies, and enterprise informatization, an increasing number of application systems are used in enterprises. However, since these application systems operate independently of each other, they cannot communicate and share information with each other, resulting in a large amount of information being enclosed in independent application systems, forming so-called "information islands" within the enterprise. Building a unified star-shaped forwarding gateway among multiple application systems can effectively reduce the communication costs of direct interconnection between systems. Further, on this basis, more flexible service pipeline configurations can be achieved among multiple systems, which is actually a scheduling concept. Distributed scheduling tools in business (such as xxl-job and quartz) mainly focus on the scheduling of timed tasks, and the core idea is to abstract Scheduler (task scheduler), Trigger (trigger), and Job (task). Among them, Trigger is used to define the trigger time, that is, according to what time rule to execute the task, Job is used to represent the task to be scheduled, and Scheduler is the actual controller that executes the scheduling. It adopts the method of polling Trigger and then executes a certain Job associated with Trigger. How to implement the service pipeline configuration between systems is a problem that needs to be solved currently. Summary of the Invention

[0003] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a method for implementing a dynamic service pipeline based on event-driven and Reactor mode. A task chain is associated with one input, and one input is combined by the scheduling center for the data on the message middleware. Different combination or division methods provide a method for dynamically configuring the service pipeline.

[0004] The present invention provides the following technical solutions:

[0005] A method for implementing a dynamic service pipeline based on event-driven and Reactor mode, comprising the following steps: a. Architecture design, after the messages on the message middleware are combined and processed by the scheduling center, corresponding task chains are executed;

[0006] b. Dynamically configure the service pipeline, divide the task chain according to the data composition on the Topic, thereby forming a service pipeline, and different division methods form different service pipelines;

[0007] c. Execution of the task chain. The input requirements of the task chain involve matching multiple types of data on multiple Topics in the message middleware. After combining them into a complete input, the execution of the task chain is triggered accordingly.

[0008] Preferably, the task chain includes multiple Handler units for performing business processing. The thread pool layer can ensure that the execution of other task chains is not blocked when the task chain is performing I / O.

[0009] Preferably, the nearest matching point algorithm is used to match multiple types of data on multiple Topics in the message middleware for matching the complete input, and a fault tolerance mechanism is provided simultaneously.

[0010] Preferably, the data on the message middleware comes from multiple systems, and there are differences in the data processing speeds of each system. The messages on the message middleware ensure the eventual consistency of the data but cannot guarantee the real-time consistency of the messages. The scheduling center needs to be responsible for fault tolerance processing when assembling the input of the task chain, saving the data that cannot be matched within a certain period for secondary processing.

[0011] Preferably, implementing the Exactly Once semantic model has a greater impact on the complexity and performance of the system. The scheduling center needs to be able to correctly handle the situations of duplicate information and lost information, that is, find the nearest matching point and save the unmatched data for secondary processing.

[0012] Preferably, finding the nearest matching point is achieved through a matching algorithm. The pseudo-code of the matching algorithm is as follows:

[0013] a. Specify the timeout time;

[0014] b. Driven by middleware events, the received data is passed to the scheduling center, and this data is added to the corresponding queue. The scheduling center records the arrival time of the data;

[0015] c1. If there is an empty queue among the queues associated with the task chain input, then the task chain is not triggered;

[0016] c2. If there is at least one value in each queue associated with the task chain input, based on the latest data, scan other queues. If a matching point is found, mark the intermediate data as unmatched and record it in the database. If no matching point is found, repeat step a;

[0017] c3. If the time of the data in the queue exceeds the threshold, mark the corresponding data as unmatched, remove it from the queue and record it in the database, and repeat step a.

[0018] Preferably, the unit Handler contains a Feign call.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) The method for implementing a dynamic service pipeline based on event-driven and Reactor mode in the present invention enables the output of each subsystem to be sent to the middleware through an interface. Multiple service pipelines can register with the scheduling center and consume the same data, reducing network traffic. It is a generalization of the producer-consumer model. At the same time, the way the scheduling center combines data for combined triggering can meet the actual business needs and provide flexibility in triggering.

[0021] (2) The method for implementing a dynamic service pipeline based on event-driven and Reactor mode in the present invention associates a task chain with one input. The scheduling center combines the data on the message middleware for one input, combines the data on different Topics to obtain a complete input, providing the ability to dynamically configure the service pipeline.

[0022] (3) The method for implementing a dynamic service pipeline based on event-driven and Reactor mode in the present invention involves a streaming data transfer from the message middleware to the scheduling center. To find the nearest matching point for a complete input in multiple streams of data, the present invention provides a matching algorithm that can quickly find the nearest matching point.

[0023] (4) The method for implementing a dynamic service pipeline based on event-driven and Reactor mode in the present invention has a task chain that includes multiple units Handler for executing business processes. Feign calls can also be included in the Handler. The thread pool layer can ensure that other task chains are not blocked when the task chain performs IO. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is the architecture diagram of the present invention.

[0026] Figure 2 It is a diagram showing two different divisions of the input of a task chain of the present invention.

[0027] Figure 3 It is the data flow closed-loop diagram of the present invention.

[0028] Figure 4 It is the data flow closed-loop diagram of different input methods of the present invention.

[0029] Figure 5 is the typical queue data diagram of the dispatching center of the present invention.

[0030] Figure 6 is the flowchart of the algorithm for finding the nearest matching point of the present invention.

[0031] Figure 7 is the execution diagram of the dispatching method of the present invention. Detailed implementation manners

[0032] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Embodiment 1:

[0035] As Figure 1 shown, a method for implementing a dynamic service pipeline based on event-driven and Reactor mode includes the following steps:

[0036] a. Architecture design: After the messages on the message middleware are combined and processed by the dispatching center, the corresponding task chains are executed;

[0037] b. Dynamically configuring the service pipeline: Divide the task chain according to the data composition on the Topic, so as to form a service pipeline, and different division methods form different service pipelines;

[0038] c. Execution of the task chain: The input requirements of the task chain match multiple types of data on multiple Topics on the message middleware, and after being combined into a complete input, the execution of the task chain is triggered.

[0039] The task chain includes multiple units Handler for executing business processing, and the thread pool layer can ensure that other task chains are not blocked when the task chain performs IO. When matching multiple types of data on multiple Topics on the message middleware, the nearest matching point algorithm is used to match the complete input, and a fault tolerance mechanism is provided.

[0040] The data on the message middleware comes from multiple systems. The data processing speeds of each system are different. The messages on the message middleware ensure the eventual consistency of the data, but cannot guarantee the real-time consistency of the messages. When assembling the input of the task chain, the scheduling center needs to be responsible for fault tolerance processing, saving the data that cannot be matched within a certain period of time for secondary processing. Implementing the Exactly Once semantic model has a great impact on the complexity and performance of the system. The scheduling center needs to be able to correctly handle the situation of duplicate information and lost information, that is, find the nearest matching point and save the unmatched data for secondary processing.

[0041] Finding the nearest matching point is achieved through a matching algorithm. The pseudo-code of the matching algorithm is as follows: a. Specify the timeout period; b. Driven by middleware events, the received data is passed to the scheduling center, and this data is added to the corresponding queue. The scheduling center records the arrival time of the data; c1. If there is an empty queue among the queues associated with the input of the task chain, then the task chain is not triggered; c2. If each queue associated with the input of the task chain has at least one value, based on the latest data, scan other queues. If a matching point is found, mark the intermediate data as unmatched and record it in the database. If no matching point is found, repeat step a; c3. If the time of the data in the queue exceeds the threshold, mark the corresponding data as unmatched, remove it from the queue and record it in the database, and repeat step a.

[0042] Embodiment 2

[0043] a. Architecture design

[0044] Combined with Figure 1 As shown, the message middleware is the source of data, and the data on the message middleware comes from each subsystem. After the messages on the message middleware are combined and processed by the scheduling center, the corresponding task chain is executed. The task chain contains multiple units Handler for executing business processing, and Feign calls can also be included in the Handler. The thread pool layer can ensure that other task chains are not blocked when the task chain performs IO.

[0045] b. Scheduling mechanism

[0046] The input of a task chain is determined, but there are multiple ways to form the input. A complete input of a task chain can have multiple "divisions". The task chain can set a division, that is, which data on which Topics the input consists of. Since this data comes from other subsystems, a business pipeline is formed in this way. At the same time, if another different division is set, another business pipeline is formed.

[0047] Among various partitioning methods, the input requirements of the task chain involve matching multiple types of data on multiple Topics of the message middleware (there are several fields for matching on the data). After combining them into a complete input, the execution of the task chain is triggered. The present invention proposes a nearest matching point algorithm for matching the complete input, and this algorithm can also provide a fault tolerance mechanism.

[0048] For example, the input requirements of a task chain are personal identification information, window information, and matter information. Different partitions are configured to form the input required by the task chain, that is, to achieve the dynamic configuration of the business system. An example of a partitioning method is: Subsystem 1 outputs personal identification information and window information (located on Topic1), and matter information (Topic2). As Figure 2 shown.

[0049] c. Fault tolerance mechanism Since the data on the message middleware comes from multiple systems, and the data processing speeds of each system are different, the messages on the message middleware ensure the ultimate consistency of the data, but do not guarantee the real-time consistency of the messages. The scheduling center needs to be responsible for fault tolerance processing when assembling the input of the task chain, and save the data that cannot be matched within a certain period of time for secondary processing.

[0050] Scheduling method, taking the Figure 1 architecture diagram as an example, there are two task chains in the figure. Suppose the first chain subscribes to the data of personal identification information and case handling information, and the second chain subscribes to the data of case handling information. This means that after the personal identification information and case handling information form a complete set of matching data, the execution of the first chain can be triggered. The data of the case handling information is complete for the second chain, so the execution of the second chain can be directly triggered. The typical execution sequence is as Figure 7 shown.

[0051] If the case handling information comes from the case handling system and the personal identification information comes from the identity verification system. Then the complete data flow closed loop is as Figure 3 shown. If a certain application subsystem generates both case handling data and personal identification data at the same time, then the task chain can switch the input "partitioning" method and subscribe to the data generated by this application subsystem. As Figure 4 shown.

[0052] Fault tolerance method, implementing the Exactly Once semantic model has a greater impact on the complexity and performance of the system. The scheduling center needs to be able to ensure correct processing in the case of duplicate information (At Most Once semantic model) and lost information (At Least Once semantic model), that is, to find the nearest matching point and save the unmatched data for secondary processing.

[0053] Suppose a task chain requires witness information and case handling information as inputs. There are two ways to process the inputs: stream processing and batch processing. Batch processing means combining multiple inputs within a length window and passing them to the task chain for execution at once. Stream processing means that once the scheduling center combines an input, it immediately passes it to the task chain for execution. An example data in the scheduling center is as follows Figure 5 as shown

[0054] The pseudo-code of the matching algorithm is as follows

[0055] Combined with Figure 6 as shown, a. Specify the timeout; b. Event-driven middleware, pass the received data to the scheduling center, add this data to the corresponding queue, and the scheduling center records the arrival time of the data; c1. If there is an empty queue among the queues associated with the task chain input, then the task chain is not triggered; c2. If there is at least one value in each queue associated with the task chain input, based on the latest data, scan other queues. If a matching point is found, mark the intermediate data as unmatched and record it in the database. If no matching point is found, repeat step a

[0056] c3. If the time of the data in the queue exceeds the threshold, mark the corresponding data as unmatched, remove it from the queue and record it in the database, and repeat step a

[0057] The device obtained by the above technical solution is a method for implementing a dynamic service pipeline based on event-driven and Reactor mode. The outputs of each subsystem can be sent to the middleware through interfaces. The outputs of each subsystem are equivalent to producers, but the consumers are determined by the scheduling center. Multiple service pipelines can register with the scheduling center to consume the same data, reducing network traffic, which is a generalization of the producer-consumer model. At the same time, the way the scheduling center combines data for combined triggering can meet the actual business needs and provide flexibility in triggering. Dynamically configure the service pipeline, one task chain is associated with one input, and one input is combined by the scheduling center for the data on the message middleware. Combine the data on different Topics to obtain the complete input, providing the ability to dynamically configure the service pipeline

[0058] In the present invention, the source of events is middleware. The events and the data carried by the events are sent to the scheduling center through a unified interface. The scheduling center will combine the data carried by the events and decide whether to trigger the task chain. The task chain part also extends the Reactor concept in network programming, expanding the single processing unit (Handler) on a single machine to a Handler chain and also expanding the scope of the Handler itself. A Handler is not only a local execution unit, but can also be the encapsulation of one or several microservice calls or the encapsulation of operators, such as filters and data format verification.

[0059] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications; any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for implementing a dynamic service pipeline based on event-driven and Reactor pattern, characterized in that, it includes the following steps: a. Architecture design: After the messages on the message middleware are combined and processed by the scheduling center, the corresponding task chain is executed; b. Dynamically configure the service pipeline: Divide the task chain according to the data composition on the Topic, thereby forming a service pipeline, and different division methods form different service pipelines; c. Execution of the task chain: The input requirements of the task chain match multiple types of data on multiple Topics on the message middleware. After being combined into a complete input, the execution of the task chain is triggered; When matching multiple types of data on multiple Topics on the message middleware, the nearest matching point algorithm is used to match the complete input, and a fault tolerance mechanism is provided. The data on the message middleware comes from multiple systems, and the data processing speeds of each system are different. The messages on the message middleware ensure the eventual consistency of the data, but cannot guarantee the real-time consistency of the messages. The scheduling center needs to be responsible for fault tolerance processing when assembling the input of the task chain, save the data that cannot be matched within a period of time for secondary processing. Implementing the Exactly Once semantic model has a greater impact on the complexity and performance of the system. The scheduling center needs to be able to correctly handle in the case of duplicate information and lost information, that is, find the nearest matching point, and save the unmatched data for secondary processing. Finding the nearest matching point is achieved through a matching algorithm. The matching algorithm is as follows: a. Specify a timeout period; b. Middleware event-driven: Transmit the received data to the scheduling center, and this data is added to the corresponding queue. The scheduling center records the arrival time of this data; c1. If there is an empty queue among the queues associated with the task chain input, then the task chain is not triggered; c2. If there is at least one value in each queue associated with the task chain input, based on the latest data, scan other queues. If a matching point is found, mark the intermediate data as unmatched and record it in the database. If no matching point is found, repeat step a; c3. If the time of the data in the queue exceeds the threshold, mark the corresponding data as unmatched, remove it from the queue and record it in the database, and repeat step a.

2. The method for implementing a dynamic service pipeline based on event-driven and Reactor pattern according to claim 1, characterized in that, the task chain contains multiple units Handler for executing business processing, and the thread pool layer can ensure that other task chains are not blocked when the task chain performs IO.

3. The method for implementing a dynamic service pipeline based on event-driven and Reactor pattern according to claim 2, characterized in that, the unit Handler contains a Feign call.

Images