Distributed processing engine integration method, storage medium and computer equipment

Abstract

The invention discloses a distributed processing engine integration method, a storage medium and computer equipment. The distributed processing engine integration method comprises the following steps: performing customized development of a running environment, a grammar rule and a processing rule on a Siddhi engine so as to adapt to a Storm engine; the method comprises the following steps: decomposing a Siddhi engine into a plurality of operators according to a preset decomposition rule, and mapping all the operators into a Storm topology; by means of the mode, the Siddhi can be integrated into the Storm, and big data processing is carried out in a complete scripting mode.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a distributed processing engine integration method, storage medium and computer equipment. Background Art

[0002] In the existing technology, the CEP computing process is still the single-threaded serial processing logic of the Siddhi full-process logic, and the performance is still not up to standard. In addition, Siddhi itself is a Spout / Bolt. The upstream data is divided by the group and then enters the respective Siddhi Spout / Bolt Task. After that, the data actually loses its relevance in CEP. If the business logic is complex, it can only jump out of the current Siddhi Task, repartition and then enter a new Siddhi script; this will cause CEP to be divided into many Siddhi scripts, which loses the ease of use of the script, and thus has poor processing capabilities for complex business requirements. At the same time, Storm itself does not provide an advanced development entry to directly access Siddhi's data interface, so each business needs to manually write the Topology code, and only the part involving CEP can reuse a Bolt code, and business development cannot be fully scripted. Summary of the invention

[0003] This application mainly provides a distributed processing engine integration method, storage medium and computer device to solve the problem that Siddhi cannot be fully scripted for big data processing when integrated into Storm.

[0004] To solve the above technical problems, a technical solution adopted in this application is: to provide a distributed processing engine integration method based on Storm, including: customizing and developing the operating environment, grammar rules and processing rules of the Siddhi engine to adapt to the Storm engine; decomposing the Siddhi engine into multiple operators according to preset decomposition rules, and mapping all the operators into a Storm topology; the operators include Source, Sink, Query and Window.

[0005] In some embodiments, mapping all the operators into a Storm topology includes: building a stream transmission channel between the Siddhi engine and the Storm engine to process messages between the Siddhi engine and the Storm engine; using components based on the Storm engine to process the operators corresponding to each component; and building the Storm topology based on the processed operators and the Storm engine.

[0006] In some embodiments, constructing a stream transmission channel between the Siddhi engine and the Storm engine includes: constructing a converter between the message unit of the Storm engine and the message unit of the Siddhi engine; connecting the stream of the Storm engine and the message receiver and message processor of the Siddhi engine.

[0007] In some embodiments, the use of components based on the Storm engine to process the operators corresponding to each component includes: constructing Storm External to replace the Source operator or the Sink operator, or parsing the components generated by the corresponding syntax of the Siddhi engine to expand the plug-in of the Siddhi engine; constructing Bolt to replace the Query operator and the computing operation of the Siddhi engine; replacing the Partition of the Siddhi engine with the Group capability of the Storm engine.

[0008] In some embodiments, the constructing the Storm topology based on the processed operators and the Storm engine includes: parsing the Siddhi engine to generate a Siddhi Context context environment; generating corresponding Spouts and Bolts based on the Storm engine according to all processed operators in the SiddhiContext context environment, and combining the Bolts to generate a corresponding Storm topology; generating a language environment based on the Siddhi Context context environment to process the streams and data corresponding to the Spouts and the Bolts.

[0009] In some embodiments, the customized development of the Siddhi engine's operating environment to adapt to the Storm engine includes: opening operators, window functions, aggregation functions, and expressions stored in the operating environment to allow the Storm engine to access or control the operating environment.

[0010] In some embodiments, the custom development of grammar rules for the Siddhi engine to adapt to the Storm engine includes: adding distribution parameters, data response parameters, and resource configuration parameters to the domain specific language of the Siddhi engine.

[0011] In some embodiments, the custom development of processing rules for the Siddhi engine to adapt to the Storm engine includes: adding exception handling rules to pass exceptions to the Storm engine, and handling the exceptions through the exception response processing logic of the Storm engine.

[0012] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the distributed processing engine integration method as described above are implemented.

[0013] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the distributed processing engine integration method as described above are implemented.

[0014] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a distributed processing engine integration method, storage medium and computer device. The operating environment, grammar rules and processing rules of the Siddhi engine are customized and developed to adapt to the Storm engine. The Siddhi engine is decomposed into multiple operators according to preset decomposition rules, and all operators are mapped to a Storm topology, thereby realizing Siddhi distributed integration based on Storm. Full scripting is achieved, which improves development efficiency. Users no longer need to write Storm code, but only need to write an independent Siddhi to complete the business development of complex event processing, and the functional complexity will not be affected. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flow chart of an embodiment of a distributed processing engine integration method provided by the present application; Figure 2 Yes Figure 1 The flowchart of the method step 200 is shown as an embodiment; Figure 3 Yes Figure 2 The flowchart of the method step 210 is shown as an embodiment; Figure 4 Yes Figure 2 The flowchart of the method step 220 is shown as an embodiment; Figure 5 Yes Figure 2 The flowchart of the embodiment of the method step 230 is shown; Figure 6 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application; Figure 7 It is a structural diagram of an embodiment of a computer device provided by the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0018] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] See also Figure 1 , Figure 1: is a flow chart of an embodiment of a distributed processing engine integration method provided by the present application. The distributed processing engine integration method based on Storm includes: 100: Customize the operating environment, grammar rules, and processing rules of the Siddhi engine to adapt it to the Storm engine.

[0020] Some custom development is done on Siddhi to connect with Storm's high performance and reliability, such as opening up API connection with the real-time runtime environment and modifying the grammar rules of DSL.

[0021] Storm is a distributed real-time computing engine used to process unbounded streaming data. It provides linear expansion, fast recovery, and data consistency guarantee, with a single node reaching millions per second. It is often used in real-time analysis, online machine learning, real-time data cleaning, and other scenarios. As a distributed, high-performance, and highly reliable computing framework, Storm can be horizontally expanded and is suitable for businesses that process massive amounts of data.

[0022] Siddhi is a complex event processing (CEP) engine that can use a SQL-like DSL for real-time analysis, data processing, etc. It provides a high-level development interface through a customized SQL-like DSL, and provides common functions for complex event processing such as PatternSequence, which provides more development capabilities than SQL, and Siddhi itself provides a lot of rich operators, which helps save development costs.

[0023] Optionally, the operating environment of the Siddhi engine is customized and developed to adapt to the Storm engine, including: opening operators, window functions, aggregation functions, and expressions saved in the operating environment to allow the Storm engine to access or control the operating environment.

[0024] Split the runtime environment API into multiple modules, each module is responsible for different functions, such as data source management, data processing, data storage, etc. Develop a customized runtime environment API.

[0025] Integrate Source and Sink with Storm's Spout and Bolt respectively to read and write data streams. Use Storm's Topology Builder to build real-time processing topologies and connect the above components according to business requirements. Submit Topologies through Storm's client API and use Storm UI for monitoring and management.

[0026] Among them, Storm's API provides Spout and Bolt interfaces to process data streams and supports customized parallelism and grouping strategies.

[0027] Spouts are the source of the message flow in the topology. They are used to ingest external data and submit it to the topology. They can record some external states such as offsets and respond to the replies (ack) of downstream bolts to ensure data reliability.

[0028] Bolts are computing units used to actually process data in the topology. They can filter, aggregate, and associate data. They can send processed message streams to the downstream according to the flow direction of the topology. They can also reply (ack) messages to tell Spouts that the current message has been processed normally.

[0029] Optionally, the grammar rules of the Siddhi engine are customized and developed to adapt to the Storm engine, including: adding distribution parameters, data response parameters, and resource configuration parameters to the domain specific language of the Siddhi engine.

[0030] Added syntax to Siddhi's query language to support distribution parameters such as parallelism and partition fields. Added specific query options or functions to identify streams to reply to upstream messages. Set resource configuration parameters through Siddhi's configuration files or specific options in the query language.

[0031] Among them, the domain-specific language (DSL) of the Siddhi engine allows users to define streams, execute queries, process events, etc.

[0032] Optionally, the processing rules of the Siddhi engine are customized and developed to adapt to the Storm engine, including: adding exception handling rules to pass exceptions to the Storm engine, and handling the exceptions through the exception response processing logic of the Storm engine.

[0033] The Storm engine provides a more powerful exception handling mechanism, including retry and failover when a task fails. Storm's Topology can be configured to take specific actions for exception handling when a Spout or Bolt task fails.

[0034] By adding a try-catch block in Siddhi's query logic, possible exceptions that may occur during Siddhi's query execution are captured. The captured exceptions are passed to the Storm engine for processing through the interface between Siddhi and Storm.

[0035] Receive exceptions passed from Siddhi in Storm's Bolt or Spout by listening to a specific output stream or checking the exception information in the context. Based on Storm's exception handling mechanism, configure the Topology to take appropriate actions when an exception is received.

[0036] Integrate the customized Siddhi engine with the Storm engine to ensure they can communicate and propagate exceptions correctly.

[0037] 200: Decompose the Siddhi engine into multiple operators according to the preset decomposition rules, and map all operators into a Storm topology.

[0038] Analyze Siddhi queries and identify the data processing steps in each query, such as filtering, aggregation, window operations, etc. Define each processing step as an operator. These units should be independent, reusable, and capable of performing specific data processing tasks.

[0039] For each input stream, create a Storm Spout to read the data and pass it to the first Bolt in the topology. For each operator, create a Storm Bolt to perform the corresponding data processing task. Each Bolt can receive data from the upstream Bolt or Spout, process it, and pass the result to the downstream Bolt or output it to an external system, mapping all operators into a Storm topology.

[0040] The Storm topology is composed of Spouts and Bolts linked in a directed acyclic graph (DAG). Spout is responsible for reading data from external sources, while Bolt is responsible for processing data and passing it to the next Bolt or outputting it to an external system.

[0041] In a Storm topology, data flow is defined by connections between bolts.

[0042] See also Figure 2 , 200 further includes: The Siddhi engine is decomposed into multiple operators according to the preset decomposition rules. Operators include Source, Sink, Query and Window.

[0043] In Storm, the Source operator corresponds to the Spout component, which is used to implement a Storm Spout to read data streams from the Siddhi engine or other external sources. Make sure that the Spout can continuously receive data from the Siddhi engine and pass it to the downstream Bolt as a Tuple (Storm's data structure).

[0044] The Sink operator is not implemented as a standalone component in Storm, but is processed as part of a Bolt or as the final output. Implement a Bolt to process data from an upstream Bolt and send it to an external system (such as a database, file system, or another service).

[0045] Query and Window operators correspond to Bolt components in Storm. For each Query and Window operator, a Storm Bolt is implemented to process input data, perform corresponding query or window operations, and generate output data. Bolts can be connected through Storm's stream mechanism to transfer data.

[0046] 210: Build a streaming channel between the Siddhi engine and the Storm engine to process messages between the Siddhi engine and the Storm engine.

[0047] Further, see Figure 3 , 210 including: 211: Build a converter between the Storm engine's message unit and the Siddhi engine's message unit.

[0048] The Storm engine's message unit (Storm Tuple) is the basic data unit in Storm and is used to transfer data between different components in the topology.

[0049] The message unit (Siddhi Event) of the Siddhi engine is used to represent the data structure of an event, which usually contains the timestamp and data fields of the event.

[0050] The conversion from the Storm engine to the Siddhi engine specifically includes mapping the fields in the Tuple to the corresponding fields of the Event, thereby extracting the data in the Storm Tuple and constructing a Siddhi Event object.

[0051] Converting from Siddhi engine to Storm engine involves extracting data from Siddhi Event and constructing a Storm Tuple object, which also requires field mapping.

[0052] 212: Connect the stream of the Storm engine and the message receiver and message processor of the Siddhi engine.

[0053] Define a Spout in the Storm topology to read the data source and generate Storm Tuple.

[0054] Define one or more Bolts to process these Tuples and pass the processed data to the next Bolt or output it to an external system, which is the stream of the Storm engine.

[0055] The setup of Siddhi message receivers and handlers specifically involves creating a Siddhi application and defining input and output streams. Use Siddhi's API to add queries that define how to extract data from input streams and generate output streams. Use Siddhi's message receivers (such as InputHandler) to receive external events and use handlers (such as QueryCallback) to process query results.

[0056] In Storm's Bolt, use Siddhi's InputHandler to send Tuple to Siddhi's input stream. In Siddhi's QueryCallback, convert the query result into Storm Tuple and use Storm's OutputCollector to send it to the next Bolt or output it to an external system, thus connecting Storm's stream with Siddhi's message receiver and message processor.

[0057] By building converters for Storm Tuple and Siddhi Event, and interoperating Storm streams with Siddhi message receivers and processors, we can integrate Siddhi into Storm to enhance Storm's functionality by leveraging Siddhi's powerful complex event processing capabilities.

[0058] 220: Use Storm engine-based components to process operators corresponding to each component.

[0059] In the process of parsing Siddhi scripts into Storm Topology, it is necessary to map each operator in Siddhi (such as Source, Sink, Query, Window, etc.) to the corresponding components in Storm (such as Spout, Bolt, etc.).

[0060] For each operator, select or customize the appropriate Storm component to handle it according to its specific processing logic and functional requirements. For example, for the Window operator in Siddhi, you need to create a dedicated Bolt to handle the window calculation logic.

[0061] In this way, the complex event processing logic in the Siddhi script is distributed to multiple nodes in the Storm cluster for parallel execution, thereby improving Storm's processing performance and scalability.

[0062] See also Figure 4 , 220 further includes: 221: Build Storm External to replace the Source operator or Sink operator, or parse the components generated by the corresponding syntax of the Siddhi engine to expand the plug-in of the Siddhi engine.

[0063] Use Storm External to replace Siddhi's Source and Sink operators to take advantage of Storm External's high performance and data consistency design, thereby improving the data processing capability and reliability of the entire system.

[0064] For Storm Externals such as Kafka, HBase, Hive, and HDFS that have high-performance alternatives or for which Siddhi does not exist, the corresponding Siddhi syntax is parsed to generate corresponding KafkaSpout / Bolt or HBaseBolt, HDFSBolt, and other plug-ins to expand Siddhi itself.

[0065] Parse the corresponding syntax of the Siddhi engine and generate components compatible with the Storm engine based on these syntaxes to expand the plug-in library of the Siddhi engine. These newly generated components can be seamlessly integrated into the Storm topology, further enhancing the flexibility and scalability of the system. Using the data consistency design of Storm's External itself is conducive to ensuring the data reliability of the system.

[0066] 222: Build Bolt to replace the calculation operations of the Query operator and Siddhi engine.

[0067] Specialized Bolts are built to replace Siddhi's query operators and computing operations, such as general QueryBolt to process general queries and Pattern / Sequence, etc., to solve the performance bottleneck faced by the Siddhi engine when processing complex queries. Bolt is designed to efficiently process various complex query logics, including pattern matching, sequence detection, multi-stream association, etc.

[0068] Specifically, the continuous convergence bolt corresponds to the Siddhi continuous convergence calculation.

[0069] Optionally, two types of bolts are created for Siddhi window calculations, a global window bolt and a MapReduce distributed grouping window, to improve the computing capabilities of massive data windows.

[0070] By encapsulating these computing operations into Bolt, it is helpful to make full use of the asynchronous processing, caching mechanism, and back pressure control features of the Storm engine, thereby improving the overall performance and throughput of the system.

[0071] 223: Use the Group capability of the Storm engine to replace the Partition of the Siddhi engine.

[0072] Use the Group capability of the Storm engine to replace the Partition mechanism of Siddhi. The Group capability of the Storm engine allows data streams to be grouped according to specific fields or rules, and these grouped data streams are distributed to different Bolts for processing.

[0073] In this way, finer-grained data control and more efficient resource utilization can be achieved. At the same time, the Storm engine also provides high-performance caching and concurrent processing mechanisms to ensure the consistency and reliability of data during grouping and processing. Make full use of the high-performance caching, concurrency, and back pressure processing methods provided by Storm itself to ensure performance.

[0074] 230: Build the Storm topology based on the processed operators and the Storm engine.

[0075] Define the structure of the Storm topology based on the operators and their relationships defined in the Siddhi engine. Determine which operators correspond to Spouts, which correspond to Bolts, and the data flow between them. Configure the appropriate parallelism for each Spout and Bolt based on processing requirements. Set appropriate grouping strategies for the streams in the Storm topology based on the operator relationships defined in the Siddhi engine. Submit the configured Storm topology to the Storm cluster for deployment.

[0076] Further, see Figure 5 , 230 including: 231: Parse the Siddhi engine to generate the Siddhi Context context.

[0077] Read the Siddhi script file, parse the Source, Sink, Query, Store and other elements in the script, as well as the dependencies and data flows between them. Generate the Siddhi Context environment, which contains all operators defined in the script and their configuration information.

[0078] Parse the configuration and rules of the Siddhi engine to generate a SiddhiContext containing all processed operators. The Siddhi Context is used to reflect the dependencies between operators, the direction of data flow, and the specific processing logic of each operator.

[0079] 232: Generate corresponding Spouts and Bolts based on the Storm engine according to all processed operators in the Siddhi Context context, and combine the Bolts to generate the corresponding Storm topology.

[0080] Generate Storm Spouts based on the Source operator defined in Siddhi Context. Generate Storm Bolts based on the Query, Filter, Window, Aggregation and other operators defined in SiddhiContext. Connect Spouts and Bolts in Storm Topology to form data streams based on the name and direction of the Stream. Configure the parallelism of each Spout and Bolt, as well as the grouping strategy of the data stream.

[0081] 233: Generate a language environment based on the Siddhi Context context to process the streams and data corresponding to Spout and Bolt.

[0082] Submit the generated Storm Topology to the Storm cluster for deployment. The Storm cluster will start the corresponding Worker process and Task according to the configuration information of the Topology. When the Storm Task is initialized, the Siddhi runtime environment is regenerated for the current Task based on the previously generated Siddhi Context context environment. Each Task will have an independent Siddhi runtime environment instance to process the data flow it is responsible for.

[0083] Storm Topology starts running, and data flows in from the Source operator (i.e., Spout). Data flows in StormTopology according to the defined flow direction and grouping strategy, and the corresponding calculation logic is executed in each Bolt. The processing results are finally output to the external system or storage through the Sink operator.

[0084] Siddhi actually uses different Streams to directly send and receive messages to complete the flow of data between different components. This application replaces or links the Publisher and Receiver of Siddhi's StreamJunction with the data sending and receiving interfaces inside the computing platform, correctly encapsulates the `Context`, and then starts the corresponding components in the `Launcher` to complete the splitting and deployment of Siddhi.

[0085] Specifically, the structure of a basic Siddhi Query in the Runtime is encapsulated as an independent Storm Bolt. According to the running structure of Siddhi Query in Siddhi App, StreamJunction is used as an instance of Siddhi Stream. When it is encapsulated in Storm's Bolt and the running structure is encapsulated, Storm's interfaces execute and emit need to be connected to Siddhi's InputHandler and Query Callback, and the tuple and Event are converted to complete the basic connection.

[0086] According to SiddhiQL, the App is split into a DAG graph, where Source is the starting point, Sink is the end point, and each Query is a computing node; then the DAG is transformed into an actual deployment graph through the predefined Parallelism. Among them, the processing logic of multi-stream query: Pattern, Sequence, Join is consistent with the above.

[0087] In addition to basic queries, Siddhi also has loaded Window Queries. There are two types of Siddhi window queries: one is chained Window Query, which is embedded in ordinary queries and does not require special processing; the other is explicitly defined Window Query, which can subsequently provide any number of window query statements.

[0088] In Siddhi Runtime, Window Query is implemented by multiple components. A StreamJunction is a StreamJunction of a defined Window Stream. A Query is a query called insert into [window stream]. N Sub Queries are query statements that perform window queries and aggregation operations on the window stream. By merging the Window Query and its SubQuery into a Storm Bolt through operator merging, the data of the InputHandler is still a simple Event, but the output of the Query callback of the window is not a simple Event, but a Complex Event, which is actually a data set of the entire window cache and cannot be sent as a message of the computing platform. Therefore, the downstream queries of the Window, that is, those Sub Queries, are merged into the operator where the Window Query is located to ensure that the data finally output by the operator is still a simple Event.

[0089] Optionally, the Window Query distributed window computing transformation method includes: In response to the existence of Group by in the Window sub query, the Window sub query containing this keyword can easily set its concurrency. It only needs to distribute its upstream according to fields to complete the distributed transformation, and only one WindowQueryBolt is needed to complete it.

[0090] In response to the absence of Group by in the Window sub query, the Window sub query is a global window and needs to be split into two Storm Bolts, namely WindowQueryMapBolt and WindowQueryReduceBolt; the former sets the concurrency to consume data according to performance requirements and aggregates temporary results to send to Reducer, while the latter is responsible for aggregating intermediate results and outputting them downstream.

[0091] The new grammar rules added to Siddhi to adapt to Storm include: Siddhi Source is used to generate Spouts of corresponding concurrency in Storm and perform field grouping based on its keys and concurrency.

[0092] Siddhi Query is used to consume the data stream distributed by keys from upstream according to the generated Bolt with corresponding concurrency.

[0093] Optionally, `delaySecs` is also included to indicate the maximum allowed message delay for the REDUCE Window; messages exceeding the delay will not be aggregated but will be output as independent data.

[0094] Add two new annotations, dcep_source and dcep_sink, in Siddhi, and use Storm External to encapsulate them as Spout / Bolt to ensure query performance and data reliability.

[0095] Use Storm External to replace Siddhi's Source and Sink, use granular Siddhi components to complete the pipeline construction of the entire data flow, and complete a complete Siddhi App mapping to StormTopology.

[0096] Optionally, if a simple Siddhi Query does not contain heavy intermediate states, such as Join / Pattern / Sequence / chained Window, and is completely consistent with the upstream or downstream stream and grouping method, two Siddhi Bolts can be merged into one to reduce the consumption of Topology computing resource slots and useless message communication IO, etc. Of course, it is also possible to specify several queries to be merged into one Bolt by adding parameters in the Siddhi script.

[0097] Based on the above embodiment of integrating Storm, those skilled in the art should understand that there are many other options for distributed real-time computing engines besides Storm, such as Heron, Spark Streaming, Flink, etc. They also provide flexible underlying APIs to connect to Siddhi and provide different data reliability guarantees.

[0098] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0099] The storage medium 300 stores program data 310. When the program data 310 is executed by the processor, the following is achieved: Figure 1The steps of the distributed processing engine integration method are described.

[0100] The program data 310 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0101] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0102] See also Figure 7 , Figure 7 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.

[0103] The computer device 400 includes a processor 420 and a memory 410 which are connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the distributed processing engine integration method described above are implemented.

[0104] Different from the prior art, this application splits the script into Source, Sink, ordinary Query, multi-stream Query, merged Window operator (Window definition, Window stream, Sub Query stream), grouped Window, global Window and other centralized implementations according to script analysis; so that they can distinguish the upstream and downstream and can be independently deployed to the distributed computing engine tasks, thus realizing the distributed integration of Siddhi based on Storm. Siddhi is distributed and configured in Storm to achieve full scripting. Only an independent Siddhi script needs to be written, and no Storm Topology code needs to be written to complete the business development of complex event processing, thus improving development efficiency. The lengthy serial processing logic of Siddhi script is divided into different Storm components and deployed on different nodes, which can make full use of Storm's own asynchronous, caching, back pressure and other mechanisms to ensure processing performance. At the same time, the concurrency of each Siddhi operator is set as needed. For example, statements with large data intake and high computing density can be set to high concurrency, and vice versa, low concurrency can be set, so that the resources of the Storm cluster can be fully utilized. In addition, after the Siddhi script is split and connected to Storm's Spout and Bolt, Storm's emit and ack mechanisms can be used to verify and reenter data consistency, which is conducive to ensuring the data consistency of the system. In the process of encapsulating Siddhi as a Storm computing engine operator, Siddhi can fully use the high performance and data reliability brought by the Storm computing engine by customizing Siddhi syntax, opening context parameters, replacing External plug-ins, and customizing different Bolts according to different queries.

[0105] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] The present application can be used in many general or special computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A distributed processing engine integration method based on Storm, characterized in that: include: Customize and develop the operating environment, grammar rules and processing rules of the Siddhi engine to adapt to the Storm engine; The Siddhi engine is decomposed into multiple operators according to a preset decomposition rule, and all the operators are mapped into a Storm topology; the operators include Source, Sink, Query and Window.

2. The distributed processing engine integration method according to claim 1, characterized in that: Mapping all the operators into a Storm topology includes: Building a stream transmission channel between the Siddhi engine and the Storm engine to process messages between the Siddhi engine and the Storm engine; Using components based on the Storm engine to process the operators corresponding to each of the components; The Storm topology is constructed based on the processed operators and the Storm engine.

3. The distributed processing engine integration method according to claim 2, characterized in that: The step of constructing a stream transmission channel between the Siddhi engine and the Storm engine includes: Constructing a converter between the message unit of the Storm engine and the message unit of the Siddhi engine; A stream connecting the Storm engine and a message receiver and a message processor of the Siddhi engine.

4. The distributed processing engine integration method according to claim 2, characterized in that: The using the components based on the Storm engine to process the operators corresponding to the components includes: Building Storm External to replace the Source operator or the Sink operator, or parsing the components generated by the corresponding grammar of the Siddhi engine to expand the plug-in of the Siddhi engine; Building Bolt to replace the calculation operation of the Query operator and the Siddhi engine; The Partition of the Siddhi engine is replaced by the Group capability of the Storm engine.

5. The distributed processing engine integration method according to claim 2, characterized in that: The step of constructing the Storm topology based on the processed operators and the Storm engine includes: Parsing the Siddhi engine to generate a Siddhi Context context; Generate corresponding Spouts and Bolts based on the Storm engine according to all processed operators in the Siddhi Context, and combine the Bolts to generate corresponding Storm topology; A language environment is generated based on the Siddhi Context context environment to process the streams and data corresponding to the Spout and the Bolt.

6. The distributed processing engine integration method according to claim 1, characterized in that: The customized development of the operating environment of the Siddhi engine to adapt to the Storm engine includes: Operators, window functions, aggregation functions, and expressions stored in the operating environment are opened to allow the Storm engine to access or control the operating environment.

7. The distributed processing engine integration method according to claim 1, characterized in that: The custom development of grammar rules for the Siddhi engine to adapt to the Storm engine includes: New distribution parameters, data response parameters and resource configuration parameters are added to the domain specific language of the Siddhi engine.

8. The distributed processing engine integration method according to claim 1, characterized in that: The custom development of processing rules for the Siddhi engine to adapt to the Storm engine includes: New exception handling rules are added to pass exceptions to the Storm engine, and the exceptions are handled by the exception response processing logic of the Storm engine.

9. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the distributed processing engine integration method according to any one of claims 1 to 8 are implemented.

10. A computer device, characterized in that: It comprises a processor and a memory connected to each other, the memory stores a computer program, and when the processor executes the computer program, the steps of the distributed processing engine integration method as described in any one of claims 1 to 8 are implemented.

Images