Data serialization in distributed event processing system

Abstract

The invention relates to data serialization in a distributed event processing system. A distributed event processing system is disclosed that receives a batch of events via a continuous data stream and performs serialization of data in the batch of events. In some embodiments, the system identifies a first data type for a first attribute of each event in the batch of events and determines a first type of data compression to be performed on data values represented by the first attribute. The system determines a first type of data compression to be performed on data values represented by a first attribute based on a first data type of the first attribute. The system then generates a first set of serialized data values for the first attribute. The system processes a first set of serialized data values for a set of one or more persistent queries to generate a first set of output events.

Description

[0001] This application is a divisional application of invention patent application 202310828134.1 entitled "Data Serialization in a Distributed Event Processing System" filed on September 12, 2017, which in turn is a divisional application of invention patent application 201780054700.0 entitled "Data Serialization in a Distributed Event Processing System" filed on September 12, 2017. Background Technology

[0002] In traditional database systems, data is typically stored in tables within one or more databases. Data management languages ​​such as Structured Query Language (SQL) are then used to query and manipulate the stored data. For example, SQL queries can be defined and executed to identify relevant data from the data stored in the database. Therefore, SQL queries are executed on a finite dataset stored in the database. Furthermore, when an SQL query is executed, it is performed once on the finite dataset and produces a finite set of static results. Thus, databases are best suited for running queries on finite stored datasets.

[0003] However, many modern applications and systems generate data as continuous streams of data or events rather than finite datasets. Examples of such applications include, but are not limited to, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and vehicle traffic monitoring. These applications have created a demand for new types of applications capable of processing data streams. For example, a temperature sensor can be configured to emit temperature readings.

[0004] Managing and processing data for these types of event-stream-based applications involves building data management and querying capabilities with a strong temporal focus. Different types of query mechanisms are needed, including long-running queries on persistent, unbounded datasets. While some vendors now offer product suites for event-stream processing, these products still lack the processing flexibility required to handle today's event processing needs. Summary of the Invention

[0005] Techniques are provided for serializing event data (e.g., methods, systems, or non-transitory computer-readable media storing code or instructions executable by one or more processors) received via a continuous stream of events. In one embodiment, a distributed event processing system is disclosed, configured to receive a batch of events via a continuous data stream and perform serialization of data within that batch of events. The distributed event processing system can be configured to process event data against a set of continuous queries to generate a set of output events. The distributed event processing system can also be configured to send the set of output events to users of the distributed event processing system.

[0006] In one embodiment, a distributed event processing system comprising multiple computing devices is disclosed. The computing devices are configured to receive a batch of events from an event stream. The event stream has an associated schema. The schema identifies one or more attributes of each event received via the event stream. The computing devices are included in a cluster of computing nodes within the distributed event processing system, and the cluster of computing nodes comprises at least a subset of the multiple computing devices in the distributed event processing system.

[0007] In some embodiments, the computing device is configured to identify a first data type of a first attribute among one or more attributes of each event in the batch of events, and determine a first type of data compression to be performed on the data value represented by the first attribute. In some examples, the first type of data compression is determined based on the first data type of the first attribute. The computing device is then configured to generate a first set of serialized data values ​​for the first attribute and store the first set of serialized data values ​​represented by the first attribute.

[0008] In some embodiments, the computing device is further configured to process a first set of serialized data values ​​corresponding to a first attribute for a set of one or more continuous queries to generate a first set of output events. The computing device is then configured to send the first set of output events to a user.

[0009] In some embodiments, the computing device is configured to generate a first set of deserialized data values ​​corresponding to a first attribute based on a first set of serialized data values, and to process the first set of deserialized data values ​​corresponding to the first attribute for a set of one or more continuous queries to generate a first set of output events.

[0010] In some embodiments, the computing device is configured to generate a first set of deserialized data values ​​corresponding to the first attribute by identifying a first type of data compression to be performed on data values ​​represented by a first attribute and deserializing a first set of serialized data values ​​represented by the first attribute according to the first type of data compression.

[0011] In some embodiments, the computing device is configured to identify a second data type of a second attribute for each event in the batch of events. In one example, the second data type differs from the first data type. The computing device is configured to determine, for each event in the batch of events, a second type of data compression to be performed on the data values ​​represented by the second attribute. In the example, the second type of data compression differs from the first type of data compression. The computing device is also configured to generate a second set of serialized data values ​​represented by the second attribute based on the determined second type of data compression, and to store the second set of serialized data values ​​represented by the second attribute. In some examples, the second type of data compression is determined based on the second data type of the second attribute.

[0012] In some embodiments, the computing device is configured to generate a second set of deserialized data values ​​corresponding to a second attribute based on a second set of serialized data values, and to process the second set of deserialized data values ​​corresponding to the second attribute for a set of one or more continuous queries to generate a first set of output events.

[0013] In some examples, the first type of data compression or the second type of data compression includes at least one of base compression, value index compression, or precision reduction and value index compression.

[0014] In some embodiments, the computing device is configured to process a first set of serialized data values ​​corresponding to a first attribute by identifying a set of one or more operations to be performed on each event in the batch of events based on a set of one or more persistent queries, representing the set of one or more operations as a directed acyclic graph (DAG) of a transformed Continuous Query Language (CQL) Resilient Distributed Dataset (RDD), and processing the first set of serialized data values ​​corresponding to a first attribute for the transformed CQL RDDDAG, thereby generating a first set of output events.

[0015] In some examples, the first data type of the first property is a numeric data type, and the second data type of the second property is a non-numeric data type.

[0016] The techniques described above and below can be implemented in many ways and in many contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts represent only a small fraction of them. Attached Figure Description

[0017] Figure 1 Aspects of an example event processing system architecture according to embodiments of the present disclosure are described, which provides an environment that can handle event processing applications for different execution environments.

[0018] Figure 2 This is a graphical representation of an event processing network (EPN) for an event processing application according to an embodiment of the present disclosure.

[0019] Figure 3 This is a simplified block diagram illustrating the components of an application processing engine according to an embodiment of the present disclosure.

[0020] Figure 4 This is an example of a representation of a "common application runtime model" generated by a common application model generator according to embodiments of this disclosure.

[0021] Figure 5 This is an example of a runtime DAG of components generated by a directed acyclic graph (DAG) generator according to embodiments of this disclosure.

[0022] Figure 6 This is a simplified high-level diagram of an event handling system that can be incorporated into embodiments of this disclosure.

[0023] Figure 7 This is a block diagram illustrating components of a distributed event processing system according to an embodiment of the present disclosure.

[0024] Figure 8 This is an example flowchart illustrating a set of operations for processing an event handling application to generate a common application runtime model, according to one embodiment of the present disclosure.

[0025] Figure 9 This is an example flowchart illustrating a set of operations for processing an event-handling application to generate a common application runtime model, according to another embodiment of this disclosure.

[0026] Figure 10 This is a simplified block diagram illustrating components of a distributed event processing system according to an embodiment of the present disclosure.

[0027] Figure 11 It is a high-level data stream according to embodiments of the present disclosure for performing the serialization and deserialization process of data in an Elastic Distributed Dataset (RDD) object.

[0028] Figure 12 This is an example flowchart illustrating a process, according to embodiments of the present disclosure, of serializing a set of operations that can be performed on data included in a batch of events.

[0029] Figure 13A This is an example flowchart illustrating a set of operations for generating a set of serialized data values ​​for a numerical attribute of an event, according to embodiments of the present disclosure.

[0030] Figure 13BThis is an example flowchart illustrating a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using precision reduction compression techniques, according to embodiments of the present disclosure.

[0031] Figure 13C This is an example flowchart illustrating a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using conventional compression techniques, according to embodiments of the present disclosure.

[0032] Figure 13D This is an example flowchart illustrating a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using a precision-reduced value index compression technique, according to embodiments of the present disclosure.

[0033] Figure 14 This is an example flowchart illustrating a set of operations for generating a set of serialized data values ​​for a non-numerical attribute of an event, according to embodiments of the present disclosure.

[0034] Figure 15 This is an example of a method for serializing event stream data based on the data type of the attributes of events in the event stream, according to embodiments of the present disclosure.

[0035] Figure 16 This is an example flowchart illustrating a set of operations that can be used to deserialize data included in a batch of events, according to embodiments of the present disclosure.

[0036] Figure 17 This is an example flowchart illustrating a set of operations for generating a set of deserialized data values ​​for one or more attributes of an event in a batch of events, according to embodiments of the present disclosure.

[0037] Figure 18 This is an example flowchart illustrating a set of operations, according to embodiments of the present disclosure, for generating a set of deserialized data values ​​corresponding to numerical or non-numerical attributes of events in a batch of events using value index compression.

[0038] Figure 19 This is an example flowchart illustrating a set of operations for generating a set of deserialized data values ​​corresponding to the numerical attributes of events in a batch of events, according to embodiments of the present disclosure, using precision reduction compression techniques.

[0039] Figure 20 This is an example flowchart illustrating a set of operations for generating a set of deserialized data values ​​corresponding to numerical attributes of events in a batch of events, according to embodiments of the present disclosure.

[0040] Figure 21This is an example flowchart illustrating a set of operations for generating a set of deserialized data values ​​corresponding to numerical attributes of events in a batch of events, according to embodiments of the present disclosure.

[0041] Figure 22 This is a simplified block diagram illustrating components of a distributed event processing system configured for scheduling and managing multiple CEP engines according to an embodiment of the present disclosure.

[0042] Figure 23 This is an example flowchart illustrating a process for scheduling and managing a set of operations for multiple CEP engines, according to embodiments of the present disclosure.

[0043] Figure 24 This is a simplified block diagram illustrating components of a distributed event processing system configured for data partitioning and parallelism according to an embodiment of the present disclosure.

[0044] Figure 25 This is an example flowchart illustrating a set of operations for automatically partitioning and parallelizing data using query clauses and object IDs, according to embodiments of the present disclosure.

[0045] Figure 26 A simplified diagram of a distributed system for implementing embodiments of the present disclosure is depicted.

[0046] Figure 27 This is a simplified block diagram of one or more components of a system environment according to an embodiment of the present disclosure, through which services provided by one or more components of the embodiment system can be provided as cloud services.

[0047] Figure 28 An example computer system that can be used to implement embodiments of the present disclosure is illustrated. Detailed Implementation

[0048] In the following description, various embodiments will be described. Specific configurations and details are set forth for illustrative purposes to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that these embodiments can be practiced without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0049] Overview of Complex Event Processing (CEP)

[0050] Complex Event Processing (CEP) provides a modular platform for building applications using event-driven architectures. At the heart of the CEP platform is Continuous Query Language (CQL), which allows applications to filter, query, and perform type-matching operations on data streams using a declarative, SQL-like language. Developers can combine CQL with a lightweight Java programming model to write applications. Other platform modules include a feature-rich IDE, management console, clustering, distributed caching, event storage, and monitoring.

[0051] As event-driven architectures and complex event processing have become prominent features of enterprise computing, a growing number of enterprises are using CEP (Content Execution Programming) technologies to build mission-critical applications. Today, mission-critical CEP applications can be found in many different industries. For example, CEP technology is used in the power industry to make facilities more efficient by allowing them to react instantly to changes in power demand. CEP technology is used in the credit card industry to detect potentially fraudulent transactions in real time. The list of mission-critical CEP applications continues to grow. Building mission-critical applications using CEP technology necessitates making these applications highly available and fault-tolerant.

[0052] Today's information technology (IT) environment generates a continuous stream of data for everything from monitoring financial markets and network performance to executing business processes and tracking RFID-tagged assets. CEP provides a rich, declarative environment for developing event processing applications to improve the effectiveness of business operations. CEP can process multiple event streams in real time to detect patterns and trends, and provide enterprises with the necessary visibility to capitalize on emerging opportunities or mitigate evolving risks.

[0053] A persistent data stream (also known as an event stream) can include either a data stream or an event stream, and can be persistent or unbounded in nature, without a definite end. Logically, an event stream or data stream can be a sequence of data elements (also known as events), each with an associated timestamp. A persistent event stream can be logically represented as a bag or set of elements (s, T), where "s" represents the data portion and "T" is in the time domain. The portion "s" is generally referred to as a tuple or event. Therefore, an event stream can be a sequence of timestamped tuples or events.

[0054] In some respects, the timestamp associated with an event in a stream can be equal to clock time. However, in other examples, the time associated with an event in an event stream can be defined by the application domain and may not correspond to clock time, but can instead be represented, for example, by a sequence number. Thus, the time information associated with an event in an event stream can be represented by numbers, timestamps, or any other information representing a time concept. For a system receiving an input event stream, events arrive at the system in ascending order of their timestamps. Multiple events with the same timestamp can exist.

[0055] In some examples, an event in an event stream can represent the occurrence of a secular event (e.g., when a temperature sensor changes its value to a new value, or when the price of a stock changes), and the timing information associated with that event can indicate when the secular event represented by the data stream event occurred.

[0056] For events received via an event stream, the time information associated with the events can be used to ensure that events in the event stream arrive in ascending order of timestamp values. This allows events received in the event stream to be ordered based on the time information associated with them. To achieve this ordering, timestamps can be associated with events in the event stream in a non-decreasing manner, such that later-generated events have later timestamps compared to earlier-generated events. As another example, if sequence numbers are used as time information, then the sequence number associated with a later-generated event can be greater than the sequence number associated with an earlier-generated event. In some examples, multiple events can be associated with the same timestamp or sequence number, for example, when secular events represented by data stream events occur simultaneously. Events belonging to the same event stream can generally be processed according to the order imposed on the events by the associated time information, such that earlier events are processed before later events.

[0057] The timing information (e.g., timestamps) associated with events in an event stream can be set by the source of the stream, or alternatively by the system receiving the stream. For example, in some embodiments, a heartbeat can be maintained on the system receiving the event stream, and the timing associated with an event can be based on the time the event arrives at the system, as measured by the heartbeat. Two events in an event stream may have the same timing information. It is important to note that while timestamp ordering requires specificity to an event stream, events from different streams can be arbitrarily interleaved.

[0058] An event stream has an associated schema "S", which includes time information and a set of one or more named attributes. All events belonging to a specific event stream conform to the schema associated with that specific event stream. Therefore, for an event stream (s, T), the event stream can have a schema "S" such as (<time_stamp> ,<attribute(s)> (i.e., (<timestamp>, <attribute>)), where <attributes>The data portion representing the schema can include one or more attributes. For example, a schema for a stock quote machine event stream can include attributes.<stock symbol> (Stock Code) and <stockprice>(Stock Price). Each event received via this stream will have a timestamp and two attributes. For example, a stock quote machine event stream might receive the following events and their associated timestamps:

[0059] …

[0060] (<timestamp_N> ,<NVDA,4> )

[0061] (<timestamp_N+1> ,<ORCL,62> )

[0062] (<timestamp_N+2> ,<PCAR,38> )

[0063] (<timestamp_N+3> ,<SPOT,53> )

[0064] (<timestamp_N+4> ,<PDCO,44> )

[0065] (<timestamp_N+5> ,<PTEN,50> )

[0066] …

[0067] In the above flow, for the flow element (<timestamp_N+1> ,<ORCL,62> The event is<ORCL,62> Each event stream has the attributes "stock_symbol" ("stock symbol") and "stock_value" ("stock value"). The timestamp associated with each stream element is "timestamp_N+1". Therefore, a continuous event stream is a stream of events, each with the same set of attributes.

[0068] As described above, a stream can be the primary source of data that a CQL query can operate on. A stream S can be a bundle (also called a "multiset") of elements (s, T), where "s" is in the schema of S and "T" is in the time domain. Furthermore, stream elements can be tuple-timestamp pairs, which can be represented as a sequence of timestamped tuple insertions. In other words, a stream can be a sequence of timestamped tuples. In some cases, there can be more than one tuple with the same timestamp. Additionally, tuples from the input stream can be requested to arrive at the system in ascending order of timestamps. Alternatively, a relation (also called a "time-varying relation," and not to be confused with "relational data," which can include data from a relational database) can be a mapping from the time domain to an unbounded bundle of tuples in schema R. In some examples, a relation can be an unordered, time-varying bundle of tuples (i.e., an instantaneous relation). In some cases, a relation can be a bounded set at each time instance. It can also be represented as a sequence of timestamped tuples, which can include insertions, deletions, and / or updates to capture the changed state of the relation. Similar to streams, relations can have a fixed schema, and each tuple in the relation can conform to that schema. Furthermore, as used herein, continuous queries are generally able to process data from both streams and / or relations (i.e., queries against them). Additionally, relations can reference data from streams.

[0069] Event handling applications

[0070] In IT environments, the volume and speed of raw infrastructure and business events are growing exponentially. Whether it's streaming stock data for financial services, streaming satellite data for the military, or real-time vehicle location data for transportation and logistics, companies across multiple industries must process massive amounts of complex data in real time. Furthermore, the explosion of mobile devices and ubiquitous high-speed connectivity has amplified the explosion of mobile data. Simultaneously, the demand for business process agility and execution is also growing. These two trends put pressure on organizations to increase their capabilities to support event-driven architecture implementations. Real-time event processing requires both infrastructure and application development environments to perform at event processing requirements. These requirements often include scaling from everyday use cases to extremely high speeds of data and event throughput (and potentially latency measured in microseconds rather than seconds). Moreover, event processing applications must frequently detect complex patterns within the streams of these events.

[0071] The Oracle Stream Analytics platform caters to a wide range of industries and functionalities. Here are some use cases:

[0072] Telecommunications: The ability to perform distributed denial-of-service attack detection and real-time call detail (CDR) logging monitoring.

[0073] Financial services: The ability to capitalize on arbitrage opportunities existing within millisecond or microsecond windows. The ability to perform real-time risk analysis, monitoring, and reporting of financial securities transactions, and to calculate foreign exchange prices.

[0074] Transportation: The ability to create passenger alerts and detect baggage location in cases of flight discrepancies caused by local or destination city weather, ground staff operations, airport security, etc.

[0075] Public sector / military: The ability to detect dispersed, geographically dispersed enemy information, abstract it, and interpret the high probability of enemy attacks. The ability to alert the most appropriate resources to respond to emergencies.

[0076] Insurance: The ability to learn and detect potential fraudulent claims.

[0077] IT systems: The ability to detect faulty applications or servers in real time and trigger corrective actions.

[0078] Supply chain and logistics: The ability to track goods in real time and detect and report potential arrival delays.

[0079] Real-time streaming and event processing analysis

[0080] With the explosive growth of data from an increasing number of connected devices, massive amounts of dynamically changing data are emerging; data is moving not only within organizations but also outside firewalls. High-speed data delivers high value, especially for uncertain business processes. However, some of this data loses its operational value within a short period. Big data makes the processing time available for gaining actionable insights a luxury. On the other hand, fast data demands the extraction of maximum value from highly dynamic and strategic data. It requires faster processing and facilitates timely action on data generated as close as possible. The Oracle Stream Analytics platform delivers fast data and responsiveness. Oracle Edge Analytics pushes processing to the network edge in real time, connecting, filtering, and analyzing data to gain actionable insights.

[0081] The Oracle Stream Analytics platform provides the ability to join incoming streaming events with persistent data, enabling context-aware filtering, association, aggregation, and type matching. It offers lightweight, out-of-the-box adapters for common event sources. It also provides an easy-to-use adapter framework for developing custom adapters. Using this platform, organizations can identify and anticipate opportunities and threats represented by seemingly unrelated events. Its incremental processing paradigm allows for event processing with minimal resources, providing extremely low latency. It also allows for the creation of highly timely alerts and immediate detection of missing or delayed events, such as:

[0082] Related events: If event A occurs, then event B will almost always follow within 2 seconds of event A.

[0083] Missing or out-of-order events: Events A, B, and C should occur in sequence. C is seen immediately after A, but B is missing.

[0084] Causal event: The weight of the manufactured item slowly tends to decrease or the reading falls outside the acceptable range. This indicates a potential problem or future maintenance needs.

[0085] Beyond real-time event sourcing, the Oracle Stream Analytics platform's design environment and runtime execution support standards-based continuous query execution across both event streams and persistent data stores (such as databases and high-performance data grids). This enables the platform to act as the intelligent core of systems requiring responses within microseconds or minutes, identifying patterns and trends that would otherwise be overlooked. Event processing use cases demand the speed of in-memory processing and the mathematical accuracy and reliability of standard database SQL. The platform listens for incoming event streams and continuously executes registered queries in memory for each event, while leveraging advanced automated algorithms for query optimization. However, while operating on an in-memory execution model, the platform fully utilizes standard ANSI SQL syntax for query development, ensuring accuracy and scalability in query construction. Fully compliant with the ANSI SQL '99 standard, the platform is one of the industry's first available extensions to standard SQL that support ANSI SQL review for real-time continuous query pattern matching. The CQL engine optimizes in-processor query execution, allowing developers to focus more on business logic than optimization.

[0086] The Oracle Stream Analytics platform allows combining SQL and Java code to deliver robust event processing applications. Leveraging standard industry terminology to describe event sources, processors, and event outputs or sinks, the platform provides a metadata-driven approach to defining and manipulating events within an application. Developers use a visual, directed canvas and palette for application design to quickly outline the flow of events and processing across both events and data sources. Processes are developed via drag-and-drop modeling and configuration wizards, allowing developers to input appropriate metadata definitions to connect the design with the implementation. Where necessary or preferred, developers can jump into custom Java code development or directly code high-level concepts into their applications using the Spring® framework with a single click.

[0087] Event-driven applications often require low and deterministic latency when handling extremely high-rate streaming input data. The Oracle Stream Analytics platform is based on a lightweight Java container with an OSGi® backplane. It incorporates mature components from WebLogic JEE application servers, such as security, logging, and job management algorithms, but fully leverages these services in a real-time event processing environment. An integrated real-time kernel provides unique services to optimize thread and memory management supported by the JMX framework, enabling interaction with the container for performance and configuration. Web 2.0 rich internet applications can communicate with the platform using HTTP publish and subscribe services, allowing them to subscribe to application channels and push events to clients. This platform has a small footprint and is a lightweight Java-based container that provides faster time-to-production and lower total cost of ownership.

[0088] The Oracle Stream Analytics platform has the capability to process millions of events per second with processing latency of a few microseconds on standard commercial hardware (or optimally using Oracle Exalogic and its combination with other engineered systems). This is achieved through a complete top-down layered solution that not only has a design focus on high-performance event processing use cases but also tight integration with enterprise-grade real-time processing infrastructure components. The performance-oriented server cluster platform architecture focuses on reliability, fault tolerance, and extreme flexibility through tight integration with Oracle Coherence technologies, enabling enterprises to predictably scale mission-critical applications across data grids, thereby ensuring continuous data availability and transaction integrity.

[0089] Furthermore, this platform allows for deterministic processing, meaning the same event can be fed to multiple servers or the same server at different rates, achieving the same result each time. This offers an incredible advantage compared to systems that rely solely on the system clock of a running server.

[0090] The techniques described above and below can be implemented in a variety of ways and in many contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts represent only a small subset of them.

[0091] Distributed event processing

[0092] In some situations, a company's users may expect to quickly identify and respond to critical events occurring within the organization so they can take immediate action once these events are identified. For example, users might want to identify critical events related to sales orders that have exceeded a threshold within the organization. In this case, users could submit one or more queries to a data repository / data warehouse and expect to see the results within seconds rather than minutes or hours, allowing them to act immediately if an anomaly is detected. When acting within seconds or minutes is significant, businesses can use real-time data processing and data analytics to process the event stream in real time for more reactive decision-making and immediate action.

[0093] According to embodiments of this disclosure, a distributed event processing system is disclosed that can use a combination of CEP (Continuous Event Processing) and distributed event stream processing to process or query very large amounts of data relatively quickly and in real time. The distributed event processing system can perform real-time processing of data streams by executing queries (e.g., CQL queries) against continuously received data streams (e.g., live feeds). The distributed event processing system can receive one or more continuous data streams, register continuous queries against the data streams, and continuously execute queries as new data appears in the streams. Because this type of continuous query runs for a long time, the distributed event processing system can provide users with a continuous stream of results.

[0094] In some embodiments, the disclosed distributed event processing system can be configured to deploy and execute the application (e.g., an event processing application) by distributing the execution of the application across a cluster of machines within the system. The event processing application described herein may include a set of rules, which may be expressed in the form of a continuous query for processing an input stream. A continuous query may include instructions (e.g., logic) to identify processing to be performed on received events, including what events to select as noteworthy events and output as the result of the query processing. Continuous queries may typically perform filtering and aggregation functions to discover and extract noteworthy events from the input event stream. The application may be configured to listen to one or more input event streams, execute logic (e.g., queries) to select one or more noteworthy events from the one or more input event streams, and output the selected noteworthy events via one or more output event streams.

[0095] For example, an event processing application could include a word counting application that counts the number of references to a specific word within a set of input text. Such an application could include, for example, a persistent query that reads a set of text and counts the number of times each word appears in each text. The input text could contain, for example, short messages received from a stream of an online application such as Facebook® or Twitter®. As mentioned above, persistent queries can be configured using the CQL language. For example, to specify a word counting task / operation to be performed in a word counting stream application, a user could write a CQL query that takes the form of: FROM location GROUP BY word SELECT count. This query could collect all sentences from a specified location, group the unique words from those sentences into different groups, and then count the number of words in each group.

[0096] By distributing application execution across a cluster of machines, a publicly disclosed distributed event processing system can be configured to provide users with fast and real-time results related to application execution. The distributed event processing system can be configured to partition application-related data across separate compute nodes, with each compute node maintaining a separate file on a separate computing machine. Each such machine can be configured to execute queries within the application in parallel with other machines, relative to the data maintained on that machine.

[0097] Efficient DAG generation for distributed event processing systems

[0098] In some embodiments of this disclosure, an application processing engine is disclosed for processing information related to an application (e.g., an event processing application). The application processing engine is configured to receive information identifying the event processing application. In some examples, the event processing application is represented as an event processing network (EPN) of components, and the information identifying the event processing application includes information related to various components of the event processing application (e.g., adapters, processors, streams, or event beans). For example, the information identifying the event processing application may include configuration information, query information, and other types of application-related information.

[0099] In some embodiments, the application processing engine can be configured to process information identifying the application and generate a "common application runtime model" for the application. As described herein, the application's "common application runtime model" is a representation of the application as a set of one or more configuration blocks, where each configuration block represents a processing stage with associated metadata describing the application. The application processing engine can be configured to transform the application's "common application runtime model" into one or more generic representations of the application. The application processing engine can then be configured to cause these one or more generic representations of the application to execute in different execution (runtime) environments supported by different target event processing systems.

[0100] The generation of a "common application runtime model" according to embodiments of this disclosure enables a general representation of an application to be executed in different physical execution (runtime) environments without requiring application developers (e.g., users) to rewrite application code to adapt to the specific physical execution (runtime) environment of the target engine before executing the application in the target engine.

[0101] The above techniques can be implemented in various ways and in multiple contexts. See below. Figure 1-9 Several example implementations and contexts are provided, which describe additional details of how the disclosed distributed event processing system can perform operations related to the deployment, processing, and execution of event processing applications.

[0102] Figure 1 Aspects of an example event processing system architecture 100 according to embodiments of the present disclosure are described, which provides an environment for processing event processing applications that can be tailored to different execution environments. In an embodiment, the architecture (event processing system) 100 includes an application processing engine 110 communicatively connected to one or more user devices 102 via a network 108.

[0103] Network 108 facilitates data communication and exchange between user equipment 102 and the application processing engine. Network 108 can be any type of network familiar to those skilled in the art, and can support data communication using any protocol from a variety of commercially available protocols, including but not limited to TCP / IP, SNA, IPX, AppleTalk, etc. By way of example only, network 108 can be a local area network (LAN) (such as Ethernet, Token Ring, etc.), a wide area network, a virtual network (including but not limited to a virtual private network (VPN)), the Internet, an intranet, an extranet, the public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any IEEE 802.1X protocol suite, Bluetooth protocol known in the art, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0104] User device 102 can be a general-purpose personal computer (including, for example, personal computers and / or laptops running various versions of Microsoft Windows and / or Apple Macintosh operating systems), a cellular phone or PDA (running software such as Microsoft Windows Mobile and enabling internet, email, SMS, Blackberry, or other communication protocols), a workstation computer running various commercial UNIX or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems), or any other computing device. For example, user device 102 can be any other electronic device capable of communicating over a network (e.g., network 108), such as a thin client computer, an internet-enabled gaming system, and / or a personal messaging device. While the example system environment 100 is shown as having one user device, any number of user and / or client computing devices can be supported in other embodiments.

[0105] In some embodiments, application processing engine 110 may be configured to process event processing applications for execution in different runtime environments. In some examples, the event processing application may be generated by the user of user device 102. For example, user 102 may use application design user interface 106 provided by client application 104 (e.g., a browser) on the user device to build an application (e.g., an event processing application). As described above, the event processing application may include a set of rules (e.g., expressed in the form of continuous queries) for processing an input stream of data from event sources. Event sources may include various data sources, such as surveillance equipment, financial services companies, or motor vehicles. Using this data, the event processing application may identify and respond to patterns, locate specific events and alert other applications, or perform other tasks based on rapidly changing data that require immediate action.

[0106] Application processing engine 110 may include one or more computers and / or servers, which may be general-purpose computers, dedicated server computers (including, for example, PC servers, UNIX servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. The computing devices comprising application processing engine 110 can run any operating system or various additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, Java servers, database servers, etc. Example database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM, etc.

[0107] In some embodiments, application processing engine 110 may be configured to receive an application (e.g., an event processing application) as described above from user device 102 and process information within the application to generate a "common application runtime model" for the application. As described above, the "common application runtime model" is a representation of the application as a set of one or more configuration blocks, where each configuration block represents an event bean having associated metadata describing the application. Application processing engine 110 may be configured to transform the application's "common application runtime model" into one or more generic representations of the application. In some embodiments, application processing engine 110 may then be configured to cause these more generic representations of the application to execute in different execution (runtime) environments supported by different target event processing systems.

[0108] In some embodiments, the application processing engine 110 may include a common application runtime model generator 112, a common application runtime model optimizer 114, and a target DAG generator 116. These components may be implemented in hardware, firmware, software, or a combination thereof. The common application runtime model generator 112 may be configured to generate a "common application runtime model" for the application based on information associated with the application. The common application runtime model optimizer 114 may be configured to optimize the "common application runtime model" to generate an optimized common application runtime model for the application. The target DAG generator 116 may be configured to transform the optimized common application runtime model into one or more generic representations of the application that can be executed by one of the target event stream processing engines (systems). The operations performed by components 112, 114, and 116 of the application processing engine 110 will be described below. Figure 2 Let's discuss this in detail.

[0109] In some embodiments, the target event processing engine 118 can be configured to receive a common application runtime model from the application processing engine 110 and transform the information in the common application runtime model into a platform-specific implementation of the application (i.e., the target event processing application 120), which can be executed in a runtime (execution) environment provided by the target event stream processing engine 118. The target event stream processing engine 118 can then be configured to continuously execute the target event processing application 120 as new data appears in the stream, providing a continuous stream of results to the user. The target event stream processing engine 118 can perform real-time processing of the data stream by executing one or more operations (e.g., CQL queries) defined in the target event processing application 120 for the continuously received data stream (e.g., live feeds). For example, the target event stream processing engine 118 can receive one or more continuous data streams, register the target event processing application 120 for the data streams, and continuously execute one or more queries defined in the target event processing application 120 as new data appears in the stream. Because this type of continuous query runs for a long time, the target event stream processing engine can provide a continuous stream of results to the user. Additional operations performed by the target event stream processing engine 118 regarding Figure 3 Let's discuss this in detail.

[0110] In some embodiments, each target event stream processing engine 118 may represent a specific physical execution environment for executing the target event processing application. For example, a first target event stream processing engine may include a first event streaming platform configured to execute the target event processing application in a first physical execution (runtime) environment, a second target event stream processing engine may include a second event streaming platform configured to execute the application in a second physical execution (runtime) environment, a third target event stream processing engine may include a third event streaming platform configured to execute the application in a third physical execution (runtime) environment, and so on. The first, second, and third event streaming platforms may be different from each other. For example, the first event streaming platform may represent an Oracle Event Processor (OEP) system managed by Oracle®. The second event streaming platform may represent a first type of distributed event processing platform (such as the Spark® framework managed by a Spark® system), and the third event streaming platform may represent a third type of distributed event processing platform (such as the Flink® framework managed by a Flink® system).

[0111] Figure 2 This is a graphical representation of an Event Processing Network (EPN) 200 for an event processing application according to embodiments of the present disclosure. In some examples, the event processing application can be represented as a network of components. Such a network of components is generally referred to as an Event Processing Network (EPN) 200. EPN 200 is a conceptual model for expressing event-based interactions and event processing specifications among the components of an event processing application. Components of the event processing application may include adapters, streams, processors, business logic plain old Java objects (POJOs), and beans. Each component in EPN 200 has a role in processing data received via event streams. As described above, the Event Processing Network (EPN) 200 may include information describing these various components, how the components are connected together, the types of events processed by the application, continuous queries or logic for selecting events used by the application, business rules defined in the application, and so on.

[0112] In some embodiments, a user of user equipment 102 may use an application design user interface (e.g., 106) provided by a client application (e.g., 104) in the user equipment to generate an EPN 200 for the event processing application. In other embodiments, the user may provide information identifying the application via the application design user interface. This information may include, for example, one or more persistent queries defined in the application, application parameters specifying the deployment type of the application, runtime configuration of the application (e.g., the number of executors to be used, parallelism parameters, memory size, high availability parameters), etc., and the client application may build and / or generate an EPN 200 for the event processing application based on this information.

[0113] In some embodiments, and as Figure 2 As shown, the EPN 200 for event handling applications can consist of the following component types:

[0114] (1) One or more adapters (202, 204) that directly interface to input and output streams, as well as relational sources and sinks. The adapters are configured to understand the input and output stream protocols and are responsible for converting event data into a normalized form that can be queried by the application processor. The adapters can forward the normalized event data to channels or output streams and relational sinks. Event adapters can be defined for various data sources and sinks. Figure 2 In the illustrated embodiment, the adapters include StreamOrRelationSource1 adapter 202 and StreamOrRelationSource2 adapter 204.

[0115] (2) One or more channels (206, 208, 210) that act as event processing endpoints. Among other things, the channels are specifically responsible for queuing event data until the event processing agent can act on the event data.

[0116] (3) One or more application processors (or event processing agents) 212 are configured to consume normalized event data from the channel, process it using queries to select noteworthy events, and forward (or copy) the selected noteworthy events to the output channel 210.

[0117] (4) One or more beans 214, 216, and 218 are configured to listen to output channel 220 and be triggered by inserting a new event into output channel 220. In some embodiments, this user code is a simple plain old Java object (POJO). The user application may use a set of external services, such as JMS, web services, and file writers, to forward the generated events to an external event sink.

[0118] (5) Event beans 214, 216 and 218 can be registered to listen to output channel 220 and be triggered by inserting new events into the output channel. In some embodiments, this user code can use the Oracle CEP event bean API so that the bean can be managed by Oracle CEP.

[0119] In one embodiment, an event adapter (202, 204) provides event data to input channels (206, 208). The input channels (206, 208) are connected to a CQL processor (212) associated with one or more CQL queries that operate on the events provided by the input channels (206, 208). The CQL processor (212) is connected to an output channel (220) where query results are written.

[0120] In some embodiments, an assembly file can be provided for the event handling application, describing the various components of the application, how these components are connected, and the types of events handled by the application. Separate configuration files can be provided to specify persistent queries or logic for selecting events. In some embodiments, the Spring® XML framework can be used to assemble information within the event handling application. As will be described in more detail below, this approach allows the application to be easily integrated with existing Spring® beans and other lightweight programming frameworks based on dependency injection mechanisms. For example, the assembly file can be a custom extension of the Spring® framework context XML configuration file so that the event server can fully leverage Spring®'s Inversion of Control (IoC) container as a whole, allowing users to seamlessly use Spring® beans in the assembly of the EPN.

[0121] Figure 3 This is a simplified block diagram 300 illustrating components of an application processing engine according to an embodiment of the present disclosure. In some embodiments, the application processing engine 314 may be configured to receive information identifying an event processing application (e.g., 304) from a user of user device 302 and generate a common runtime application model based on that information. As described above, the event processing application 304 may be generated by a user of user device 302 using an application design user interface 306 provided by a client application (e.g., 104) in the user device.

[0122] In some examples, the information identifying the application may include information describing the various components of the application (e.g., adapters, processors, streams, or event beans). This information may include, for example, configuration information, query information, and other types of information. Configuration information may include, for example, information describing the various components of the event handling application, how the components are connected together, and the types of events processed by the application. For example, configuration information may include information describing the network of components (i.e., EPN 200) that the event handling application is described as. Query information may include information specifying persistent queries or logic used by the application to select events. Other types of information may include plain old Java objects (POJOs) and business rules defined in the application.

[0123] In some examples, information identifying the application can be specified in a configuration file. For instance, each component in EPN 200 of the event handling application 304 can have an associated configuration file. In other examples, the information in application 304 can be represented in a single configuration file that includes information for all components in the application. In one implementation, the configuration file can be expressed as a regular XML document, the structure of which is defined using a standard XML schema based on a configuration schema defined by a common runtime model.

[0124] In some examples, various configuration files, such as EPN configuration files, query configuration files, and other files, can be used to specify information identifying the application. Examples of EPN and query configuration files for an event processing application are shown below. In the example shown, the event processing application is an order event processing application 304, which is configured to receive and process a continuous stream of events, where each event represents an order for an item sold by the company. Each order in the order event stream can include attributes related to the item, such as an order identifier, order status, and order quantity. An example of an EPN configuration file 308 for event processing application 304 is shown below. EPN configuration file 308 includes a series of sub-elements, each containing configuration information for the event processing components within the event processing application.

[0125] EPN configuration file

[0126]

[0127] The following shows an example of a query configuration file 310 for event handling application 304. Query configuration file 310 specifies one or more persistent queries or logic for selecting events in the event handling application.

[0128] Query configuration file

[0129]

[0130] In some embodiments, application processing engine 314 can be configured to generate a "common application runtime model" for an application (e.g., 304) based on configuration information, query information, and other information specified in EPN configuration file 308, query configuration file 310, and other files 312. This "common application runtime model" can then be transformed by target DAG generator 320 into one or more generic representations of the application for execution in different physical execution (runtime) environments supported by different target event stream processing engines 226, 228, and 230. The generation of the "common application runtime model" enables the execution of the generic representation of the application in different physical execution (runtime) environments without requiring application developers (e.g., users) to rewrite application code to adapt to the specific physical execution (runtime) environment of the target engine before the application is executed in the target engine. The common application runtime model is independent of the physical execution environment. This platform-independent abstraction allows the system to easily generate DAGs and code for physical execution environments.

[0131] In some embodiments, the "common application runtime model" generated by the application processing engine 314 may include representing the application as a set of one or more event beans with associated configuration information. Additional details of how the application processing engine 314 may represent the "common application runtime model" as a set of event beans are described in detail below.

[0132] In some embodiments, the application processing engine 314 includes a common application runtime model generator 316, a common application runtime model optimizer 318, and a target DAG generator 320. The common application runtime model generator 316 is configured to generate a "common application runtime model" for the application based on information identifying the application specified in the EPN profile 308, query profile 310, and other files 312. In some embodiments, generating the "common application runtime model" by the common application runtime model generator 316 involves loading the EPN profile 308, query profile 310, and other files 312 into memory using an EPN loader implemented using the Spring® application framework. The result is a set of Spring® beans connected via Spring® inversion of control injection. The common application runtime model generator 316 is then configured to parse the EPN profile 308 using an XML parser such as JAXB (Java Architecture for XML Binding) and set the parsed profile to each associated bean in the EPN network. In some examples, each resolved configuration block or file will have an identifier so that the block or file can locate the event bean and set the configuration block or file to that event bean. Therefore, in an embodiment, generating a "common application runtime model" for the application includes representing the application as a set of one or more configuration blocks, where each configuration block represents a Spring® event bean with associated metadata. Figure 4 The image shows an example of a representation of the "common application runtime model". In some examples, the "common application runtime model" maintains metadata about the components of the event handling application and does not contain any processing logic.

[0133] Figure 4 An example representation of a "public application runtime model" 400 generated by a public application model generator 316 according to an embodiment of this disclosure is depicted. In the embodiment, the "public application runtime model" 400 may be generated based on information identifying the application (e.g., from EPN profile 308, query profile 310, and other files 312), and the "public application runtime model" 400 may be represented as a set of one or more configuration blocks, wherein each configuration block represents a Spring® event bean with associated metadata. In the embodiment, configuration blocks 402, 404, 406, 408, and 410 in the public runtime application model 400 may include the following information:

[0134]

[0135] For example, in configuration block 402, socket inbound represents the "socket adapter" in the EPN; in configuration block 404, channel-1 represents the "orders" channel in the EPN; in configuration block 406, CQL processor represents the "order processor" in the EPN; in configuration block 408, channel-2 represents the "output channel" in the EPN; and in configuration block 410, socket outbound 410 represents the "output adapter" in the EPN.

[0136] As described above, representing an event processing application as a "common application runtime model" 400 enables the application to execute in different target event stream processing engines (e.g., 326, 328, or 330) without requiring the user to rewrite the application's code to fit the target engine's specific physical execution (runtime) environment before the application is executed in the target engine.

[0137] Return to Figure 3 In some embodiments, the common application runtime model generated by the common application runtime model generator 316 can also be optimized by the common application runtime model optimizer 318. Optimization of the common application runtime model (e.g., 400) may include, for example, combining multiple phases within a configuration block into a single phase (e.g., combining multiple continuous queries into a single optimized continuous query) or splitting a single phase into multiple phases for parallel processing (e.g., splitting a continuous query into multiple continuous queries that can perform map and reduce operations). For example, coherent queries without repartitioning can be combined into a single CQL phase using a single CQL process. For example, the two phases "select * from orders where orderStatus='open'" and "select count(*) from orders group by orderId" can be combined into a single CQL phase using "select count(*) from orders group by orderId where orderStatus="open". The optimizer can also divide a single phase into multiple phases to maximize scalability. For instance, the fully-stateful query phase "select count(*) from orders" can be partitioned into two phases: "select count(*) from order group by orderId" and "select sum(*) from counts". This allows the global count to be processed using partial counts from partitions, and then the partial counts can be added to the global count.

[0138] In some embodiments, the target DAG generator 320 may be configured to transform an optimized common application runtime model into one or more generic representations of an application that can be executed by one of the target event stream processing engines (326, 328, or 330). For example, the target DAG generator 320 may be configured to transform the optimized common application runtime model into runtime DAG 322 or the target representation 324 of the application, depending on the target event stream processing engine in which the application will execute. For example, if the target event stream processing engine (e.g., 326) is an Oracle Event Processor (OEP) managed by Oracle®, then the target DAG generator 320 may be configured to transform configuration blocks in the common application runtime model (or the optimized common application runtime model) into the target representation 324 of the application that will be executed by the target event stream processing engine 326. In some embodiments, the transformation may include a copy of an object of the common application runtime model with appropriate beans having actual processing logic by the target DAG generator 320. For example, the metadata of a CQL processor bean in the public application runtime model can be copied to a new instance of the event handling component (e.g., a CQL processor) that runs a CQL engine to process input events using a given query (e.g., copied from the CQL processor in the public runtime application model).

[0139] For example, if the target event stream processing engine (e.g., 328) is a distributed event processing platform managed by Spark® Distributed Systems, then the target DAG generator 320 can be configured to transform objects (configuration blocks) in a common application runtime model (or an optimized common application runtime model) into runtime DAGs 322 representing the application's components. An example of a runtime DAG 322 of components generated by the target DAG generator is shown in... Figure 5 As shown in the diagram. Then, the runtime DAG322 of the component is transformed into the target application (Spark® CQL application) by the target event stream processing engine 328. An example of a target application generated by the target event stream processing engine 328 for calculating the number of orders grouped by order status is shown below:

[0140] Example of the target application

[0141]

[0142]

[0143]

[0144] Figure 5 This is an example of a runtime directed acyclic graph (DAG) 500 of components generated by a target DAG generator according to embodiments of the present disclosure. In some embodiments, and as described above, when the execution (runtime) environment of the target application is a distributed event processing system (e.g., Spark® Distributed Event Processing System), the target DAG generator 320 may be configured to transform objects (402, 404, 406, 408, and 410) in a common application runtime model (e.g., 400) into a runtime DAG 500 representing components of that application.

[0145] In this embodiment, the runtime DAG of the components includes the following components: SocketText 502, Map-1 504, Map-2 506, PartitionBy 508, CQL 510, Map-3 512, and Print 514. Figure 4 (As shown) SocketInbound 402 is transformed into SocketText 502, Map-1 504, and Map 506. The SocketText 502 component includes line 7 of the example of the target application shown above, which loads a string from the socket. The Map-1 504 component includes line 8 of the example of the target application, which transforms the string into comma-separated values. The Map-2 506 component includes line 9 of the example of the target application, which transforms comma-separated values ​​into tuples. CQL processor 406 and Channel-1 404 are transformed into PartitionBy 508 and CQL 510. The PartitionBy 508 component includes line 10 of the example of the target application, which creates group-based partitions according to the standards in CQL. The CQL 510 component includes line 11 of the example of the target application, which is the main CQL processing stage. Channel-2 408 and StdoutOutbound 410 are transformed into Map-3 512 and Print 51. The Map-3 512 part includes line 12 from the example of the target application, which converts the tuple into a comma-separated string, and the Print 514 part includes line 13 from the example of the target application, which prints the output string to the stdout console.

[0146] Figure 6 A simplified high-level diagram of an event processing system 600 that can be incorporated into embodiments of this disclosure is depicted. In embodiments, the event processing system 600 may represent an Oracle Event Processor (OEP) system managed by Oracle®. The event processing system 600 may include one or more event sources (604, 606, 608), an Event Processing Service (EPS) 602 (also referred to as CQ Service 602) configured to provide an environment for processing event streams, and one or more event sinks (610, 612). The event sources generate event streams received by the EPS 602. The EPS 602 may receive one or more event streams from one or more event sources.

[0147] For example, such as Figure 6 As shown, EPS 602 receives a first input event stream 614 from event source 604, a second input event stream 616 from event source 606, and a third event stream 618 from event source 608. One or more event processing applications (614, 616, and 618) can be deployed on EPS 602 and executed by EPS 602. The event processing applications executed by EPS 602 can be configured to listen to one or more input event streams and process the events received via one or more event streams based on processing logic that selects one or more events from the input event streams as noteworthy events. The noteworthy events can then be sent to one or more event sinks (610, 612) as one or more output event streams. For example, in Figure 6 In this embodiment, EPS 602 outputs a first output event stream 620 to an event sink 610 and a second output event stream 622 to an event sink 612. In some embodiments, the event source, event handling application, and event sink are decoupled from each other, allowing any one of these components to be added or removed without affecting the others.

[0148] In one embodiment, EPS 602 can be implemented as a Java server that includes a lightweight Java application container with shared services, such as an Equinox OSGi-based container. In some embodiments, EPS 602 can support ultra-high throughput and microsecond latency for event processing, for example, by using JRockit RealTime. EPS 602 can also provide a development platform (e.g., a complete real-time end-to-end Java event-driven architecture (EDA) development platform) that includes tools for developing event processing applications (e.g., Oracle CEP Visualizer and Oracle CEPIDE).

[0149] An event handling application is configured to listen to one or more input event streams, execute logic (e.g., a query) to select one or more noteworthy events from the one or more input event streams, and output the selected noteworthy events to one or more event sources via one or more output event streams. Figure 6 An in-depth analysis of 614 for such an event handling application is provided. For example... Figure 6 As shown, event handling application 614 is configured to listen to input event stream 618, execute a continuous query 630 including logic for selecting one or more noteworthy events from input event 618, and output the selected noteworthy events to event sink 612 via output event stream 622. Examples of event sources include, but are not limited to, adapters (e.g., JMS, HTTP, and files), channels, processors, tables, caches, etc. Examples of event sinks include, but are not limited to, adapters (e.g., JMS, HTTP, and files), channels, processors, caches, etc.

[0150] Although Figure 6 The event handling application 614 is shown as listening to an input stream and outputting selected events via an output stream, but this is not limiting. In alternative embodiments, the event handling application may be configured to listen to multiple input streams received from one or more event sources, select events from the monitored streams, and output the selected events to one or more event sinks via one or more output event streams. The same query may be associated with more than one event sink and with event sinks of different types.

[0151] Due to its unbounded nature, the amount of data received via the event stream is typically very large. Therefore, storing or archiving all data for query purposes is generally impractical and undesirable. Event stream processing requires that events be processed in real-time as they are received by the EPS 602, without storing all received event data. Therefore, the EPS 602 provides a special query mechanism that enables event processing to be performed as events are received by the EPS 602, without storing all received events.

[0152] Event-driven applications are rule-driven, and these rules can be expressed as persistent queries used to process the input stream. Persistent queries can include instructions (e.g., logic) that identify the processing to be performed on received events, including which events to select as noteworthy events and output as the result of the query processing. Persistent queries can be persisted to a data repository and used to process both the input stream of events and the output stream that generates them. Persistent queries typically perform filtering and aggregation functions to discover and extract noteworthy events from the input event stream. Therefore, the number of outbound events in the output event stream is generally much smaller than the number of events in the input event stream from which these outbound events are selected.

[0153] Unlike SQL queries that run once on a finite dataset, a continuous query that has been registered by the application with EPS 602 for that event stream can be executed each time an event is received in a specific event stream. As part of the continuous query execution, EPS 602 evaluates the received events based on the instructions specified by the continuous query to determine whether to select one or more events as noteworthy events and output them as the result of the continuous query execution.

[0154] Continuous queries can be programmed using different languages. In some embodiments, continuous queries can be configured using CQL, provided by Oracle Corporation and used by Oracle's Complex Event Processing (CEP) product. Oracle's CQL is a declarative language that can be used to program queries that can be executed against a stream of events (called CQL queries). In some embodiments, CQL is based on SQL and has added constructs to support the processing of streaming event data.

[0155] It should be recognized that, Figure 6 The system 600 described in the text can have, except Figure 6 Other components besides those depicted. Additionally, Figure 6 The embodiments shown are merely one example of a system that can be incorporated into embodiments of this disclosure. In some other embodiments, system 600 may have more than Figure 6 The system may contain more or fewer components, may combine two or more components, or may have different component configurations or arrangements. System 600 can be of various types, including service provider computers, personal computers, portable devices (e.g., mobile phones or devices), workstations, network computers, mainframes, kiosks, servers, or any other data processing systems.

[0156] Figure 7 This is a simplified block diagram 700 illustrating the components of a distributed event processing system 710 according to an embodiment of the present disclosure. Figure 7 The embodiment shown is an example of a distributed event processing system that can be incorporated into embodiments of this disclosure. In some other embodiments, system 710 may have more than Figure 7 The system may contain more or fewer components, may combine two or more components, or may have different component configurations or arrangements. System 710 may be any type of computing device, such as, but not limited to, mobile computing devices, desktop computing devices, thin client computing devices and / or cloud computing devices, servers, or any other data processing system.

[0157] In some examples, the distributed event processing system 710 may consist of a pre-integrated and optimized combination of software resources, hardware resources, networking resources, and other resources. Hardware resources may include, but are not limited to, servers, data storage devices, printers, etc. Software resources may include, but are not limited to, computing programs, applications (e.g., cloud-based applications, enterprise applications, or any other applications), computer program products (e.g., software), services (e.g., cloud-based services), etc. Data resources may include, but are not limited to, any accessible data objects, such as files (e.g., networked files or directory information), databases, etc.

[0158] In some embodiments, the distributed event processing system 710 may include a receiver 704 and a cluster of compute nodes 708. The receiver 704 may be configured to receive a continuous stream of input events 702 and discretize (divide) the event stream into one or more batches of events 706 for a specific duration (e.g., X seconds) for subsequent processing by the cluster of compute nodes 708 in the distributed event processing system 710. Each batch of events is referred to herein as a Dstream. In some examples, each Dstream is internally represented by the receiver 704 as a Resilient Distributed Dataset (RDD), which is a snapshot of all input data (event) streams ingested during a specified time period (i.e., within an event batch). Thus, in some embodiments, the input data stream 702 is represented as a sequence of Dstreams, where each Dstream is internally represented as an RDD, and each RDD includes events (tuples) received during a specific batch interval. In some examples, each RDD represents a set of elements of an immutable partition, which may be stored in a cache and executed in parallel within the distributed event processing system.

[0159] In some embodiments, compute node cluster 704 can be configured to partition the data contained in each RDD across the compute node cluster and perform operations on the data in parallel against a set of queries defined in the application, providing the processing results to users of the distributed event processing system. Therefore, compute node cluster 708 can be configured to distribute the processing of event data in the RDD across compute node cluster 708 and provide users with results related to the application performed on the event data quickly and in real time. In embodiments, the Apache® Spark Streaming framework can be used to configure the distributed event processing system 710 to perform distributed and real-time processing of continuous data streams and the deployment of event processing applications.

[0160] Figure 8 This is an example flowchart of process 800 according to an embodiment of the present disclosure. Process 800 describes a set of operations for processing an event-handling application to generate a common application runtime model of the application. In some embodiments, process 800 may be... Figure 3 The process 800 is executed by one or more components (e.g., 316, 318, and 320) in the application processing engine (314) described herein. The process 800 begins at 802 by receiving information identifying the application. This information may include, for example, information describing various components of the application (e.g., adapters, processors, streams, or event beans), such as configuration information, query information, and other types of information. As described above, this information may be... Figure 3 The configuration files (308, 310, and 312) described in the document express this.

[0161] At 804, the process includes generating a "common application runtime model" for the application based on information identifying the application. In an embodiment, generating a "common application runtime model" for the application may include representing the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata. Configuration blocks may include, for example, information about... Figure 4 The described inbound socket event bean, outbound socket event bean, Continuous Query Language (CQL) processor event bean, or one or more channel event beans.

[0162] At 806, the process includes transforming the application's "common application runtime model" into a first general representation of the application. The first general representation of the application may be configured to execute in a first target event processing system among multiple target event processing systems. In one example, transforming the application's "common application runtime model" into the first general representation of the application may include transforming configuration blocks in the common application runtime model into runtime DAGs of the application's components. Figure 5 An example of a runtime DAG for a component generated for an application is shown in the figure.

[0163] In some embodiments, at 808, the process includes sending a first general representation of the application to a first target event processing system for execution by the first target event processing system. In one example, the first target event processing system is a distributed event processing system.

[0164] Figure 9 This is an example flowchart of process 900 according to another embodiment of the present disclosure, which describes a set of operations for processing an event-handling application to generate a common application runtime model of the application. In some embodiments, process 900 may be... Figure 3 One or more components (e.g., 316, 318, and 320) in the application processing engine (314) described herein are executed. Process 900 begins at 902 by receiving information identifying the application. As described above, this information may include configuration information, query information, and other types of information describing the various components of the application.

[0165] At 904, the process includes generating a "common application runtime model" for the application based on information identifying the application. In an embodiment, generating a "common application runtime model" for the application may include representing the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata. Configuration blocks may include, for example, information about... Figure 4 The described inbound socket event bean, outbound socket event bean, Continuous Query Language (CQL) processor event bean, or one or more channel event beans.

[0166] At 906, the process includes transforming the application's "common application runtime model" into a second general representation of the application. This second general representation of the application can be configured to execute in a second target event handling system across multiple target event handling systems. In some examples, the second target event handling system may differ from the first target event handling system. In one example, transforming the application's "common application runtime model" into the application's second general representation may include transforming configuration blocks in the common application runtime model into a target representation of the application.

[0167] In some embodiments, at 908, the process includes sending a second general representation of the application to a second target event processing system for execution by the second target event processing system. In one example, the second target event processing system is an Oracle® Event Processor (OEP) system.

[0168] Serialization and deserialization of event data

[0169] In some embodiments, the disclosed distributed event processing system can be configured to perform serialization and deserialization of event data received via a continuous event stream. Serialization and deserialization of event data enables the transformation of complex data objects in memory into bit sequences that can be transmitted to computation nodes in the distributed event processing system. This serialization and deserialization allows processing nodes in the distributed event processing system to efficiently store and represent data before it is processed by the system. Furthermore, serialization and deserialization reduce latency in exchanging input and output events between processing nodes in the distributed event processing system and improve the overall performance of the system.

[0170] The above techniques can be implemented in a variety of ways and in many contexts. See below. Figure 1-10 Several example implementations and contexts are provided, and these diagrams illustrate additional details of how the disclosed distributed event processing system can perform operations related to the deployment, processing, and execution of event processing applications.

[0171] Figure 10 This is a simplified block diagram 1000 illustrating components of a distributed event processing system according to an embodiment of the present disclosure. The distributed event processing system 1000 can be coupled with... Figure 1 The distributed event processing system 110 described herein is the same as or similar to that described in the document. Figure 10 The embodiment shown is an example of a distributed event processing system that can be incorporated into embodiments of this disclosure. In other embodiments, the distributed event processing engine may have more... Figure 10 The components shown may be more or fewer, two or more components may be combined, or they may have different configurations or arrangements. These components may be implemented using hardware, firmware, software, or a combination thereof. In some embodiments, the software may be stored in memory (e.g., a non-transitory computer-readable medium), a memory device, or some other physical memory, and may be executed by one or more processing units (e.g., one or more processors, one or more processor cores, one or more GPUs, etc.). Therefore, Figure 10 The embodiments shown are examples of a distributed event processing engine for implementing the system of the embodiments and are not intended to be limiting.

[0172] In some embodiments, the distributed event processing system 1002 may include a receiver 1004, an application deployment module 1008, and a cluster of computing nodes 1012. The receiver 1006 may be capable of... Figure 2 As shown (e.g., from event sources 204, 206, or 208), a continuous stream of input data 1004 is received, which is then divided into one or more batches of events 1010, referred to herein as Dstreams. As described above, each Dstream (i.e., an event batch) comprises all the input data (event) streams ingested during a specified time period and can be represented internally by receiver 1006 as an RDD object, which is an immutable collection of partitioned elements that can be executed in parallel across a cluster of compute nodes 1012 in the distributed event processing system 1002.

[0173] Application deployment module 1006 can be configured to deploy applications (e.g., event processing applications) for processing and execution by compute nodes in compute node cluster 1012. The applications described herein can refer to computer programs (e.g., user-built) in a distributed event processing system. For example, an application could include a word counting application that counts the number of references to a specific word within a set of input text. Such an application can be constructed using, for example, one or more persistent queries that read the set of text and count the number of times each word appears in each text. The input text can contain, for example, short messages received from a stream of an online application such as Facebook® or Twitter®. As described above, persistent queries can be configured using the CQL language. For example, to specify a word counting task / operation to be performed in a word counting streaming application, a user could write a CQL query in the form: SELECT count FROM location GROUP BY word. This query can collect all sentences from a specified location, group the unique words from those sentences into different groups, and then count the number of words in each group.

[0174] In some embodiments, the application deployment module 1008 may be configured to receive application identification information from a user of the distributed event processing system. For example, the application deployment module 1008 may be configured to receive application identification information via an application design user interface within the application deployment module 1008. The application identification information may include a set of one or more persistent queries defined in the application. The application identification information may also include application parameters associated with the application. Application parameters may include, for example, deployment type parameters that specify the deployment type of the application on the node cluster 1012 (e.g., "cluster type"). Additional application parameters may include parameters related to the application's runtime configuration (e.g., the number of executors to use, parallelism parameters, memory size, high availability parameters, etc.).

[0175] Upon receiving application-related information, in some embodiments, the application deployment module 1008 may then be configured to send instructions to the compute node cluster 1012 to deploy the application on the compute nodes in the cluster. In some examples, the compute node cluster 1012 may be configured to deploy the application to a master compute node 1014 on the compute node cluster 1012. The master compute node 1014 may be configured to store the application's "application context." The "application context" may include, for example, application content such as the application's topology, scheduling information, application parameters, etc.

[0176] In some embodiments, the master compute node 1014 may be referred to as a "driver," or the application master device that runs / executes the application. A driver can be defined as a process that runs the application's main() function and creates an "application context" for the application. The driver may be responsible for driving the application and requesting resources from resource manager 1016. Resource manager 1016 may be a service that acquires resources for compute nodes on compute node cluster 1012 to execute the application. To run / execute the application on the cluster, master compute node 1014 connects to resource manager 1016, which then allocates resources for the application. Once connected, master compute node 1014 acquires one or more executors on one or more compute nodes (also referred to as worker nodes 1018) in the cluster. An executor is a process that performs computations and stores data for the application. Master compute node 1014 sends application code (e.g., defined by a JAR file) to the executor. Based on transformations and actions defined in the application, master compute node 1014 may send task 1020 to the executor.

[0177] In some embodiments, the master compute node 1014 may include a DAG generator 1022, a DAG scheduler 1024, a task scheduler 1026, and application context information 1028. As described above, the application context information 1028 may include information about the application, such as the application's topology, scheduling information, application parameters, etc. The DAG generator 1022 may be configured to define and / or create a directed acyclic graph (DAG) of RDD objects based on the RDD objects it receives from the receiver. In some examples, the DAG generator 1022 may represent the DAG of the RDD objects as an RDD genealogy graph of all RDDs it has received over a certain time interval. Each RDD object in the RDD genealogy graph maintains pointers to one or more parents and metadata related to the type of relationship between the RDD object and its parents. The RDD genealogy graph also identifies the DAG of transformations to be performed for each RDD object. When the master compute node 1014 is requested to run a job in the application, the DAG generator 1022 performs the transformed DAG. Transformations identify one or more operations to be performed on an RDD object to transform the data within the RDD object from one form to another. For example, these operations can be defined as part of the application generation process performed by the application deployment module 1008. When an operation is applied to an RDD object, a new RDD object with the transformed data is obtained. Examples of operations performed on an RDD object can include, for example, map, filter, flatMap, reduce, union, groupByKey, distinct, join, collect, count, etc. The DAG involving transformations in the CQL language may be referred to as a CQL transformation in this paper.

[0178] DAG scheduler 1024 is configured to generate a physical execution plan based on an RDD genealogy graph generated by a DAG generator. In one embodiment, DAG scheduler 1024 generates the physical execution plan by splitting the RDD genealogy graph into multiple stages, where each stage is identified based on the transformations that need to be performed on the data in each RDD object. For example, if the transformations to be performed on the RDD object include map transformations and reduce transformations, then map transformations can be grouped together into a single stage, and reduce transformations can be grouped together into another stage. DAG scheduler 1024 then submits the stages to task scheduler 1026.

[0179] Task scheduler 1026 divides the application (job) into multiple stages. Each stage consists of one or more tasks. In one embodiment, the number of tasks in a particular stage is determined based on the number of partitions of the input data in the RDD object. For example, and as described above, DAG scheduler 1024 can schedule all map operations to a single stage. These stages are then passed to task scheduler 1026, and task scheduler 1026 initiates the tasks via resource manager. The tasks are then executed by executor node 1018. Task scheduler 1026 identifies the node in compute node cluster 1012 that will perform the operations defined in the application (job) for each RDD object (i.e., each batch of events being processed).

[0180] In some embodiments, when executor node 1018 receives an RDD object, if it needs to send the RDD object to other executor (worker) nodes in the compute node cluster, then executor node 1018 performs serialization and deserialization of the data in the RDD object. As described above, processing of the data in the RDD object may involve executing one or more continuous queries defined in the application. In an embodiment, the executor node (e.g., 1018) identified by task scheduler 1026 may invoke a CQL engine (such as CQL processor 230) to perform processing of the data in the RDD object and return the processing results to master compute node 1014. The following section discusses... Figure 11 This section discusses in detail how executor node 1018 can perform serialization and deserialization of data in RDD objects before processing them.

[0181] Figure 11 A high-level data flow is described according to embodiments of the present disclosure for performing serialization and deserialization of data in an RDD object. In some examples, the process can be performed by a node in a compute node cluster (e.g., a master node or executor node) when it receives the RDD object from a receiver for processing. Figure 10 One or more operations are involved. In one set of operations, receiver 1102 receives the input data stream and divides the input data stream into one or more batches of events (tuples) for a specific duration (e.g., X seconds). In one embodiment, and as per [reference to...] Figure 10 As described, each batch of events (tuples) can be internally represented as an RDD object by the receiver. The DAG generator 1104 receives the RDD object and creates the DAG of the RDD object 1106. As described above... Figure 10 As described, in some examples, the DAG 1106 of the RDD objects includes CQL transformations 1108 to be performed on each RDD object. When requesting the master compute node (e.g., as...) Figure 10 As shown in 1014, when a job is running in the application, the DAG generator 1104 performs a CQL transformation 1108 to process a set of input tuples represented by RDD objects. In some examples, the set of input tuples to be processed can be obtained from a parent transformation of the CQL transformation 1108. The sub-transformation of the CQL transformation 1108 then calls the specific operation to be performed for that set of input tuples, as shown in the CQL transformation 1108.

[0182] In some examples, CQL transform 1108 invokes batch serializer process 1112 for the set of input tuples in the RDD object to perform serialization of the data in the RDD object. In embodiments, batch serializer process 1112 may be executed by a node (e.g., an executor node) in a compute node cluster that is processing the RDD object. As described above, the data in the RDD object represents a batch of input tuples (events) received via an event stream. Batch serializer process 1112 serializes the data in the RDD object, and the serialized blocks of the result from the batch serializer process are sent over the network to CQL engine 1116 to process the set of input tuples. In some embodiments, the node processing the RDD object may invoke CQL engine 1116 to process the set of input tuples in the RDD object. For example, CQL engine 1116 may be an event processing engine deployed on an executor node (e.g., Figure 6 (630 as described in the document). The CQL engine 1116 can be configured to receive the set of input tuples, process the set of input tuples based on the processing logic (operations / transformations) defined in the application, and generate a set of output tuples as a result of the processing.

[0183] In some embodiments, CQL engine 1116 may invoke batch deserializer process 1114 for the serialized data blocks received from batch serializer process 1112 before processing the data in the RDD object. This is because the serialized blocks are in a binary or wired format suitable for transmission over a network and need to be deserialized into Java objects so that CQL engine 1116 can process them. Therefore, the result of the deserialization process is a set of input tuples in a form that can be processed by CQL engine 1116. CQL engine 1116 processes this set of input tuples and generates a set of output tuples based on the processing. In some examples, CQL engine 1116 invokes another batch serializer process 1118 to serialize this set of output tuples, and the result of the serialization is a serialized block of output tuples. When the serialized set of output tuples is received by the DAG generator, CQL transform 1108 invokes another batch deserializer process 1120 for the received serialized block of output tuples. The result of batch deserializer process 1120 is a set of deserialized output tuples. CQL transform 1108 returns this set of output tuples to a sub-transformation in the CQL transform to perform the next set of processing operations on the data in the RDD object. In some embodiments, the set of output tuples 1110 is then sent to the user of the distributed event processing system.

[0184] In some embodiments, the batch serializer process and batch deserializer process described above can be executed by software modules or instructions executed by nodes (e.g., executor nodes) in a node cluster of a distributed event processing system that are processing RDD objects. Additional details regarding the operations performed by the batch serializer process and batch deserializer process will be provided below. Figure 12-15 Let's discuss this in detail.

[0185] Figure 12 This is an example flowchart of process 1200 according to an embodiment of the present disclosure, which describes a set of operations by which data included in a batch of events can be serialized. In some embodiments, process 1200 may be executed by a batch serializer process (1112) in a distributed event processing system. As described above, in some embodiments, when the master computing node (e.g., such as...) Figure 10 When 1014 (as shown) is requested to run a job / operation defined in the application, the batch serializer process can be invoked by the master compute node. As described above, the master compute node identifies nodes (e.g., executor node 1018) in the compute node cluster 1012 of the distributed computing system 1002 to process a batch of events for the job / operation defined in the application and generate a set of output events as a result of the processing. In some embodiments, processing the batch of events may include serializing the data in the batch of events. Figure 12 The process describes a technique that allows data from a batch of events to be serialized. Figure 12 The specific series of processing steps described herein are not intended to be limiting. Alternative embodiments may be found in a variety of arrangements and combinations, and may have more... Figure 12 The steps shown may include more or fewer steps.

[0186] In some embodiments, whenever via ( Figure 10 When task 1020 (as shown) receives a batch of events, Figure 12 The processing described herein can be executed by nodes in the computing node cluster 1012 of the distributed computing system 1002. The process begins at 1202, and at 1204 receives a batch of events from the CQL transformation 1108. In some examples, each event in the batch may be referred to as a tuple, and the batch of events may be referred to as a batch of input tuples or a set of input tuples. As mentioned above, each event received via the event stream conforms to a pattern associated with the event stream, and this pattern identifies one or more attributes of each event received via the event stream.

[0187] For example, a persistent event stream can represent product-related information associated with products sold by a company, where each event in the stream can represent an order for an item. Such a persistent event stream can include attributes such as an order identifier associated with the item, order status, and order quantity. The schema used for this input stream can be represented as S(timestamp, ...). <orderid> ， <orderstatus> ， <orderamount>Therefore, each event received via this stream will be identified by a timestamp and three attributes. In some embodiments, one or more attributes of an event may be represented as one or more columns in a set of input tuples (a batch of events); thus, in some examples, an attribute may refer to a column that stores the data values ​​of the tuples (events) in a set of input tuples.

[0188] At 1206, the process includes identifying an attribute (e.g., a first attribute) of an event in the batch of events. At 1208, the process includes identifying the data type of the attribute. For example, according to the order processing flow example above, the processes at 1206 and 1208 could identify that the attribute corresponds to the "orderId" attribute of the event and that the data type of the attribute is a numeric data type. At 1210, the process includes determining whether the data type of the attribute is a numeric data type. If the identified data type of the attribute is a numeric data type, then in some embodiments, the process proceeds to 1212 to determine a first type of data compression to be performed on the data value represented by the attribute. For example, at 1212, the process may include determining that a numeric compression technique (e.g., base compression, precision reduction compression, or precision reduction value index) is applied to the data value represented by the attribute. At 1214, as a result of applying the numeric compression technique to the data value stored by the attribute, the process includes generating a set of serialized data values ​​for the attribute. Figure 13A , Figure 13B , Figure 13C and Figure 13D The process of generating a set of serialized data values ​​for a numerical attribute of an event is discussed. At 1216, the process includes storing the set of serialized data values ​​represented by the attribute.

[0189] In some embodiments, at 1218, the process includes determining whether there are additional attributes for an event that needs to be processed. If there are additional attributes to be processed, the process loops back to 1206 to identify the next attribute (e.g., a second attribute) for an event in the event batch, and performs the processes at 1208-516 for that next attribute.

[0190] In some embodiments, at 1210, if the data type of the identified attribute is not determined to be a numeric data type, then in some embodiments, the process proceeds to 1220 to determine a second type of data compression to be performed on the data value represented by the attribute. For example, continuing with the order processing flow example above, the processes at 1206 and 1208 may identify that a second attribute of the event corresponds to the "orderStatus" attribute and that the data type of this attribute is a non-numeric data type. In this case, the process continues to 1220 to determine a second type of data compression to be performed on the data value stored by the attribute. In embodiments, the second type of data compression may differ from the first type of data compression. For example, the process at 1220 may determine to apply a non-numeric compression technique (e.g., value index compression) to the data value stored by the attribute. At 1214, the process includes generating a set of serialized data values ​​represented by the attribute as a result of applying the non-numeric compression technique to the data values ​​stored by the attribute. Figure 14 The process of generating a set of serialized data values ​​for a non-numeric attribute of an event is discussed. At 1216, the process includes storing the set of serialized data values ​​represented by the attribute.

[0191] In some embodiments, the process may continue to 1218 to determine if there are any additional attributes for the event to be identified and processed. If more attributes exist, the process loops back to 1206 to identify a third attribute for the event in the batch. The process at 1208 may then include identifying the data type of the third attribute, and the process at 1210 may include determining a third type of data compression to be performed on the data value stored by the third attribute based on the data type of the third attribute. For example, continuing the order processing event flow example above, a third type of data compression could be performed on the data value stored by the "orderAmount" attribute based on the data type of the attribute. In some examples, the process ends at 1222 when all attributes of the event have been identified and processed.

[0192] Figure 13A This is an example flowchart of process 1300 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of serialized data values ​​for a numerical attribute of an event. In the embodiment, process 1300 describes a process by... Figure 12 Additional details of the operations performed in procedure 1214 are provided below. In some examples, procedure 1300 begins at 1302 by storing the current buffer offset into the current column number of the column whose data value is being processed (e.g., the first attribute). At 1304, the procedure includes obtaining the data type (column type) of the attribute. For example, in this example, the data type of the attribute can be determined to be a numeric attribute. At 1306, the procedure includes scanning a set of input tuples to obtain the minimum, maximum, and unique value set represented by the attribute. At 1308, the procedure includes calculating the number of bits required to store the data value represented by the attribute within the range (maximum - minimum). At 1309, the procedure includes determining whether the number of bits required is greater than half the number of bits of the attribute's data type and whether the size of the unique value set is less than the number of input tuples / value_index_threshold, where value_index_threshold is configurable. In the example, value_index_threshold can be configured to the value 11 as the default. If so, the procedure continues to 1350 to perform a precision reduction index value compression technique. At 1310, the process includes determining whether the number of bits required is less than the number of bits in the column's original data type. The check at 1310 ensures that the size of the serialized block does not increase compared to the original block. This is because if the number of bits required to cover the value range is greater than the number of bits required by the original data, the resulting block created using value indexing techniques might be larger than the original block size.

[0193] If the required number of bits is less than the number of bits in the column's original data type, the process continues to step 1312 to determine if the size of the unique value set is less than the number of input tuples / 2. This determination is performed to ensure a sufficiently high compression ratio. If there are too many unique values, then in some examples, the value itself will be used instead of the value and its index. If it is determined that the required number of bits is less than the column's original data type and the size of the unique value set is less than the number of input tuples / 2, then the processes described in steps 1314-1326 are performed.

[0194] For example, at 1314, the process includes storing the first type of data compression to be performed on the data values ​​represented by the attribute as a precision-reduced value index type data compression. Precision reduction techniques reduce the number of bits representing the values ​​from the values ​​used by looking up the range of values. The number of bits required will depend on the range of values. At 1316, the process includes storing the minimum data value of the attribute. At 1318, the process includes storing the number of bits for each minimum value. At 1320, the process includes performing the operations at 1322 and 1324 for each data value of the column (e.g., the attribute) whose data values ​​are currently being processed. For example, at 1322, the process includes obtaining an index from the set of unique data values. At 1324, the process includes storing the index into a buffer. After all the data values ​​of the column have been processed, at 1326, the process includes storing (unique value - minimum value) for each unique value in the set of tuples. These values ​​represent the actual value indexed from the index stored at step 1324.

[0195] In some embodiments, if the process at 1312 determines that the size of the unique value set is not less than the number of input tuples / 2, then in some embodiments, the following is performed. Figure 13B The procedures 1332-1338 described herein. In some embodiments, if the procedure at 1310 determines that the required number of bits is not less than the original data type of the column, then in one embodiment, execution is performed. Figure 13C The process described in 1342-1346. In some embodiments, the process ends at 1328 by returning a set of serialized data values ​​(i.e., a serialized data block) to the CQL engine for processing the tuple (i.e., the batch of events) received via the event stream.

[0196] Figure 13B This is an example flowchart of process 1350 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using precision reduction compression techniques. In the embodiment, process 1350 describes a process by... Figure 13A Additional details of the operations performed in procedure 1330. In some examples, procedure 1350 begins at 1332 by storing the type of data compression to be performed on the data values ​​represented by the attribute as precision reduction compression. At 1334, the procedure includes storing the minimum data value of the attribute. At 1336, the procedure includes storing the number of bits for each data value of the attribute. At 1338, the procedure includes performing a bit copy (value - minimum value) on only the required bits for each data value of the column. For example, a set of input values ​​(10, 11, 12) will be stored, where bit 00 is used for value 0, which is the result of (10 - 10 (minimum value)), 01 is used for value 1, which is the result of (11 - 10), and 02 is used for value 2, which is the result of (12 - 10). The sequence of bit values ​​00, 01, and 02 can be stored in byte (8 bits) 00010200 and stored as a hexadecimal value of 154.

[0197] Figure 13C This is an example flowchart of process 1360 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using conventional compression techniques. In the embodiment, process 1360 describes a process by... Figure 13A Additional details regarding the operations performed in process 1340. In some examples, process 1360 begins at 1342 by storing the first type of data compression to be performed on the data values ​​represented by the attributes as a general compression type. At 1344, the process includes compressing the array of column values ​​using standard compression techniques such as zip or gzip. At 1346, the process includes storing the compressed bytes of the data values ​​represented by the attributes.

[0198] Figure 13D This is an example flowchart of process 1370 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of serialized data values ​​for the numerical attributes of an event using a precision-reduced value index compression technique. In the embodiment, process 1370 describes a process by... Figure 13A Additional details of the operations performed in procedure 1350. In some examples, procedure 1370 begins at 1372 by storing the type of data compression to be performed on the data values ​​represented by the attribute as precision reduction index value compression. At 1374, a set of differences (e.g., value-minimum) is calculated. At 1376, the procedure includes scanning all values ​​in the set of differences to obtain an enumerated set of values. At 1378, the procedure includes calculating an index set for each data value represented by the set of differences. At 1380, the procedure includes calculating the minimum and maximum values ​​from the index set. At 1382, the procedure includes storing the minimum data value of the attribute. At 1384, the procedure includes storing the number of bits for each data value of the index values. At 1386, the procedure includes performing bit copying for each data value of the column, only for the required bits. At 1388, the procedure includes storing the set of enumerated differences.

[0199] Figure 14 This is an example flowchart of process 1400 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of serialized data values ​​for a non-numeric attribute of an event. In the embodiment, process 1400 describes a process performed when it is determined that a second type of data compression should be performed on data values ​​represented by attributes (e.g., non-numeric attributes). Figure 12 Additional details of the operations performed in process 1214. Process 1400 begins at 1402 by storing the current buffer offset into the current column number of the column whose data value is being processed (e.g., an attribute). At 1404, the data type (column type) of the attribute is obtained. For example, in this case, the data type of the attribute is determined to be a non-numeric attribute. At 1406, the process includes storing the type of data compression to be performed on the data value represented by the attribute as value-indexed compression. In this case, all possible values ​​within the input batch are enumerated, and positional indexing is used instead of copying the value multiple times.

[0200] At 1408, the process includes scanning all input tuples to obtain an enumerated set of values ​​for the column. At 1410, the process includes calculating an index set for each data value represented by the column. At 1412, the process includes performing the operations described below in 1414-1416 for each data value stored in the column. At 1414, the process includes obtaining the index from the enumerated set of values. At 1416, the process includes storing the index into a buffer. At 1418, the process includes storing the enumerated set of values. At 1420, the process ends by returning the set of serialized data values ​​for the attribute (i.e., the serialized data block) to the CQL engine to process a set of tuples received via the event stream (i.e., a batch of events).

[0201] Figure 15 This is an example of a method for serializing event stream data based on the data type of the attributes of events in the event stream, according to embodiments of this disclosure. In the example described below, the event stream represents product-related information associated with products sold by a company. Each event in the event stream may represent an order for an item and includes attributes related to the item, such as an order identifier, order status, and order quantity. The schema used for such an order event stream can be represented as S(timestamp, <orderid> ， <orderstatus> ， <orderamount>Therefore, each event received via this stream can be identified by a timestamp and three attributes. As an example, a batch of events received via an order event stream may include the following events and associated timestamps:

[0202] …

[0203] (timestamp_N, 10, "open", 100)

[0204] (timestamp_N+1,11,"open",5000)

[0205] (timestamp_N+2,10,"processing",100)

[0206] (timestamp_N+3,10,"shipped",100)

[0207] (timestamp_N+4, 11, "processing", 5000)

[0208] (timestamp_N+5,10,"closed",100)

[0209] (timestamp_N+6,11,"shipped",5000)

[0210] (timestamp_N+7,11,"closed",5000)

[0211] As described above, in some embodiments, one or more attributes of an event can be represented as one or more columns of a set of input tuples representing a batch of events. Therefore, in some examples, an attribute can refer to a column storing the data values ​​of the tuples (events) in a set of input tuples. An example of a set of input tuples corresponding to a batch of events received via an order event stream can be shown in Table-1 below:

[0212] Table 1

[0213] In some embodiments, events in a batch of events are serialized by identifying the data type of each attribute of the event and determining, based on the data type of the attribute, a specific type of compression technique to be applied to the data value represented by each attribute. For example, a first compression technique may be applied to a first attribute of the event (e.g., an order ID attribute) based on determining that a first attribute is a numeric attribute; a second compression technique may be applied to a second attribute of the event (e.g., an order status attribute) based on determining that a second attribute is a non-numeric attribute; and a third compression technique may be applied to a third attribute of the event (e.g., an order quantity attribute) based on determining that a third attribute is a numeric attribute. In some examples, the first type of compression technique, the second type of compression technique, and the third type of compression technique may be different from each other.

[0214] In embodiments, columnar storage can be used to store attributes (columns) that have the same data type (such that the values ​​in the column are of the same data type). In some embodiments, radix compression or precision reduction compression techniques can be used to compress values ​​stored in columns of numeric types. Precision reduction reduces the number of bits representing the values ​​used by looking up the range of values. The number of bits required will depend on the range of values. Radix compression uses the minimum value as the radix and stores the difference from the radix for other values. For example, the set of input values ​​representing the "order ID" for each event in an event batch (10, 11, 10, 10, 11, 10, 11, 11) can be compressed to (10, 1), 01001011 (binary), or 0x4B (hexadecimal), which represents the values ​​(0, 1, 0, 0, 1, 0, 1, 1), thus reducing the number of bits used from 32 to 2 because the minimum value is 10 and the range is 2. For another example, precision reduction and value indexing techniques can be used to compress the set of input values ​​representing "order quantity". In this case, the set of input values ​​representing the order quantity (100, 5000, 100, 100, 5000, 100, 5000, 5000) can be compressed to (100, 2, 0x10, 0x4F) and (0, 4900) using precision reduction and value indexing techniques. The input set can be represented as (0, 4900, 0, 0, 4900, 0, 4900, 4900), with a base of 100. The values ​​of the result set are binary 00010000, 01001111, hexadecimal 10 and 4F, which represent (0, 1, 0, 0, 1, 0, 1, 1) with indices to the base table (0, 4900) (e.g., 0 points to 0 and then to 100 because the base is 100, and 1 points to 4900 and then to 5000 because the base is 100).

[0215] In some embodiments, the "value index compression" technique can be used to process the values ​​of columns storing non-numeric values ​​such as string values. In this case, we enumerate all possible values ​​in a batch of input and use the index of position instead of copying the value multiple times. For example, if the value of the "order status" attribute (column) is (open, open, processing, shipping, processing, closed, shipping, closed), then the corresponding enumerated unique value will be (open, processing, shipping, closed). When the column values ​​are stored sequentially in a linear buffer, the index of each value will be (0, 5, 17, 25) because the buffer will have open / 0 processing / 0 shipped / 0 closed / 0, where / 0 indicates the end of the string marker. In the case of a linear buffer of values, the final compressed result is (0, 0, 5, 17, 5, 25, 17, 25).

[0216] Figure 16 This is an example flowchart of process 1600 according to an embodiment of the present disclosure, which describes a set of operations through which data included in a batch of events can be deserialized. In some embodiments, process 1600 may be executed by a batch deserializer process (420) in a distributed event processing system. As described above, in some embodiments, when a request is made to the master computing node (e.g., such as...) Figure 3 As shown in 314), when running a job / operation defined in the application, the batch deserializer process can be invoked by the master compute node. As described above, the master compute node identifies nodes (e.g., executor node 318) in the compute node cluster 312 of the distributed computing system 302 to process a batch of events for the job / operation defined in the application and generate a set of output events as a result of the processing. In some embodiments, processing the batch of events may include serialization and subsequent deserialization of the data in the batch of events. Figure 16 The process describes a technique that allows data from a batch of events to be deserialized. Figure 16 The specific series of processing steps described herein are not intended to be limiting. Alternative embodiments may be found in a variety of arrangements and combinations, and may have more... Figure 16 The steps shown may include more or fewer steps.

[0217] In some embodiments, process 1600 begins at 1602 by receiving a set of serialized data values ​​corresponding to one or more attributes of events in a batch of events (a set of input tuples). At 1604, the process includes processing the set of serialized data values ​​corresponding to one or more attributes of events in the batch of events to generate a set of output events. In some examples, the process at 1604 may include generating a set of deserialized data values ​​corresponding to the attribute based on the set of serialized data values ​​at 1606, and processing the set of deserialized data values ​​corresponding to the attribute for a set of one or more continuous queries at 1608 to generate a first set of output events. At 1610, the process includes sending the set of output events to a user of the distributed event processing system.

[0218] Regarding the following Figure 17-18 The process of generating a set of deserialized data values ​​corresponding to the attributes of events in a batch of events is discussed in detail

[1604] . Specifically, Figure 17 This describes the process by which a set of deserialized data values ​​corresponding to one or more attributes of events in a batch of events can be generated. Figure 18 This describes the process by which a set of deserialized data values ​​corresponding to the non-numeric properties of an event can be generated. Figure 19-21 This describes the process by which a set of deserialized data values ​​corresponding to the numerical properties of an event can be generated.

[0219] Figure 17 This is an example flowchart of process 1700 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of deserialized data values ​​from one or more attributes of events in a batch of events. In the embodiment, process 1700 describes Figure 16 Additional details of the operation of the procedure in 1604. In some examples, procedure 1700 begins at 1702 by creating an array of tuples. At 1704, the procedure includes identifying the first column (first attribute) of the event. At 1706, the procedure includes obtaining the buffer offset of the current column number of the column whose data value is being processed (e.g., the first attribute). At 1708, the procedure includes reading the compression type of the attribute. For example, this involves reading the data compression type performed by the batch serializer process to serialize the data value of the attribute. At 1710, the procedure includes determining whether the type of data compression applied to the attribute is value-indexed compression. If the compression type applied to the attribute is value-indexed compression, then the procedure then continues to 1724 to execute. Figure 18 The process described in the text.

[0220] At 1712, the process includes determining whether the type of data compression applied to the attribute is precision reduction compression. If the compression type applied to the attribute is precision reduction compression, then the process continues to 1726 to perform. Figure 19 The process described in the text.

[0221] In some embodiments, at 1714, the process includes determining whether the type of data compression applied to the attribute is precision reduced value index compression. If the compression type applied to the attribute is precision reduced value index compression, then the process continues to 1728 to perform. Figure 20 The process described in the text.

[0222] In some embodiments, if the process determines that the compression type applied to the attribute is neither value index compression, nor precision reduction compression, nor precision reduction value index compression, then the process continues to 1716 to determine that the compression type applied to the attribute is general compression. At 1716, the process includes performing... Figure 21 The process described in the text.

[0223] At 1718, the process includes determining whether there are any additional attributes to process. If there are, the process loops back to 1704 to identify and process the next attribute for the event. If there are no attributes to process, in some embodiments, the process returns the array of tuples to the CQL engine for further processing at 1720. In some embodiments, the process ends at 1722.

[0224] Figure 18 This is an example flowchart of process 1800 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of deserialized data values ​​corresponding to numeric or non-numeric attributes of events in a batch of events using value index compression. In an embodiment, process 1810 describes... Figure 17 Additional details of process 1724, and when applying data compression types to attributes (e.g., in...). Figure 17 When the value index (at position 1710 in the data) is determined to be compressed, process 1810 is executed. As an example, process 1800 can be executed for the "order status" property (a non-numeric property) in the order event stream.

[0225] An example of using value index compression techniques is in Figure 8 As shown in column 2, the compressed values ​​have two data sets: (0, 0, 5, 17, 5, 25, 17, 25) and ('Open', 'Processing', 'Shipping', 'Closed'). The second set is called `value_arrays` because it contains the actual values. The first set is called `index_values` because it includes the indexes to the actual values ​​stored in `value_arrays`. `value_index` refers to each of the individual indexes in `index_values`.

[0226] In some embodiments, process 1800 begins at 1802 by reading the index values ​​into index_values. At 1804, the process includes reading the array of values ​​into value_array. At 1806, the process includes performing operations at 1808, 1810, and 1812 for each data value corresponding to an attribute in a set of input tuples. For example, at 1808, the process includes obtaining the index from index_values[value_index]. At 1810, the process includes obtaining the value from value_array[index]. At 1812, the process includes setting the value into the tuple column tuple[value_index].

[0227] Figure 19 This is an example flowchart of process 1900 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of deserialized data values ​​corresponding to numerical attributes of events in a batch of events using precision reduction compression techniques. In the embodiment, process 1900 describes Figure 17 Additional details regarding process 1726, and when applying data compression types to attributes (e.g., in...). Figure 17 At position 1712 in the data, process 1900 was executed during "precision reduction" compression. As an example, process 1900 can be executed for the "Order ID" or "Order Quantity" attribute (a numeric attribute) in the order event stream, and the compressed result will be displayed in... Figure 8 Column 1 is shown as (10, 2, 0x4B). In one example of the set of operations described below, the item "Minimum" has a value of 17, which is the minimum value in the range of values, and the item "Number of Bits" has a value of 2, which is the number of bits representing the range of values.

[0228] In some embodiments, process 1900 begins at 1902 by reading the minimum value into base_value. At 1904, the process includes reading the number of bits. At 1906, the process includes performing the procedures in 1908 and 1910 for each data value corresponding to an attribute in a set of input tuples. For example, at 1908, the process includes reading the value bits into value_bits. At 1910, the process includes setting base_value + value_bits into the tuple column tuple[value_index].

[0229] Figure 20 This is an example flowchart of process 2000 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of deserialized data values ​​corresponding to numerical attributes of events in a batch of events using a precision reduction value index. In the embodiment, process 2000 describes Figure 17 Additional details regarding process 1728, and when applying data compression types to attributes (e.g., in...). Figure 17 At position 1712 in the data, the "Precision Reduction Value Index" is identified, and process 2000 is executed during compression. As an example, process 2000 can be executed for the "Order ID" or "Order Quantity" attribute (numerical attribute) in the order event stream. Figure 8 The example results for the "Order Quantity Attribute" are shown in column 3, which are (100, 2, 0x10, 0x4F) and (0, 4900).

[0230] In the set of operations described below, the term "base value" refers to the value 170 for the base value of the column value, the term "number of bits" refers to the value 2 for the number of bits of the index value, the term "index_values" refers to the value (0x10, 0x4F) used as the index value, and the term "value_array" refers to (0, 4900) representing the set of differences.

[0231] In some embodiments, process 2000 begins at 2002 by reading the minimum value into base_value. At 2004, the process includes reading the number of bits. At 2006, the process includes reading the index value into index_values. At 2008, the process includes reading the array of values ​​into value_array. At 2010, the process includes setting value_array[j] to value_array[j] + base_value for each value from j ≠ 0 to value_array.length. At 2012, the process includes performing the procedures in 2014, 2016, and 2018 for each data value corresponding to an attribute in a set of input tuples. For example, at 2014, the process includes obtaining the index from index_values[value_index]. At 2016, the process includes obtaining the value from value_array[index]. At 2018, the process includes setting that value into the tuple column tuple[value_index].

[0232] Figure 21 This is an example flowchart of process 2100 according to an embodiment of the present disclosure, which describes a set of operations for generating a set of deserialized data values ​​corresponding to numerical or non-numerical attributes of events in a batch of events. In the embodiment, process 2100 describes Figure 17 Additional details regarding process 1716, and when applying data compression types to attributes (e.g., in...). Figure 17 When the "general compression" technique is determined at point 1716 in the data stream, process 2100 is executed. As an example, process 2100 can be executed for the "Order ID" or "Order Quantity" attribute (a numeric attribute) in the order event stream. In the set of operations described below, the item "value" refers to the uncompressed value.

[0233] In some embodiments, process 2100 begins at 2102 by decompressing the block into an array of values. At 2104, the process includes setting the values ​​in the array of values ​​to tuple columns corresponding to attributes in a set of input tuples for each data value.

[0234] Scheduling and managing multiple CEP engines in a micro-batch-based event processing system

[0235] In recent years, data stream management systems (DSMs) have been developed that can continuously execute queries on potentially unbounded real-time data streams. Among the newer DSMs, these systems employ micro-batch-based stream processing to provide a combination of batch and stream processing from a single framework. An example of such a system is the Spark® Streaming application running on the Spark® platform.

[0236] Micro-batch stream processing has some drawbacks due to the nature of its generally complex system design involving stateful processing. One such drawback is the inability to perform "type matching" operations. Type matching is an important desired feature that stream processing systems should support, and it requires highly stateful processing to run state machines to detect patterns from unbounded streams of events.

[0237] To support fully stateful query processing, the disclosed techniques add a CQL query engine to micro-batch stream processing. Since there is more than one CQL engine in the cluster, issues related to scheduling, tracing, and maintaining locality must be addressed.

[0238] Figure 22 This is an example system or architecture in which the scheduling process in a CQL engine tracer can be implemented. In one embodiment, and as follows: Figure 22 As shown, a CQL engine tracker 2202 component is disclosed in the driver (main) 2206, which can communicate remotely between the CQL engine 2212 and the CQL Resilient Distributed Dataset (RDD) 2218. For startup and scheduling, the CQL engine tracker 2202 uses a two-step scheduling strategy to differentiate between different system environments. To maximize locality, in one embodiment, the CQL engine tracker 2202 uses the following affinity algorithm.

[0239] All CQL engines 2212, 2214, and 2216 are started from driver 2206 by CQL engine tracker 2202. No association is set between the CQL engine and the preferred location.

[0240] The first CQLRDD 2218 does not have preferred location information.

[0241] Scheduler 2204 will attempt to use the preferred location of the parent RDD to co-locate to the host where the parent RDD resides.

[0242] The first run of CQLRDD 2218 associates CQL engine 2212 with the same host 2208.

[0243] The next CQLRDD 2220 will set the preferred location information based on the association information set in step 4.

[0244] Scheduler 2204 will attempt to run CQLRDD to the preferred location where it is set.

[0245] The disclosed techniques enable fully stateful CQL engines 2212, 2214, and 2216 in micro-batch streaming processing, thereby maintaining locality between the CQL engine and CQLRDD, as well as a multi-step scheduling algorithm for starting and restarting the CQL engine. Furthermore, the disclosed local affinity algorithm provides maximum performance compared to other event-based streaming systems.

[0246] In some embodiments, the disclosed CQL engine tracker 2202 is responsible for scheduling, tracking, and restarting CQL engines in the cluster. The CQL engines run as long-running tasks within the cluster and can be started as regular streaming jobs. The CQL engine tracker 2202 does not return except in the event of a failure.

[0247] In some embodiments, the following tracking information may be maintained for the CQL engine tracker 2202.

[0248] state: CQLEngineState - INACTIVE, SCHEDULED, ACTIVE

[0249] This changed from INACTIVE -> SCHEDULED -> ACTIVE -> INACTIVE throughout the entire lifecycle of CQL engines 2212, 2214, and 2216.

[0250] scheduleLocation:TaskLocation

[0251] Initial scheduling position

[0252] runningExecutor:ExecutorCacheTaskLocation

[0253] The CQL engine actually runs at executor position 2208.

[0254] name: String

[0255] The names of CQL engines 2212, 2214, and 2216

[0256] endpoint: RpcEndpointRef

[0257] Remote Process Call (RPC) endpoints for CQL engines 2212, 2214, and 2216 for remote access.

[0258] errorInfo: CQLEngineErrorInfo

[0259] Last known error message

[0260] In the embodiments, the startup process of CQL engines 2212, 2214, and 2216 can be described as follows:

[0261] Determine the number of CQL engines 2212, 2214, and 2216 to be activated.

[0262] Obtain the list of executors 2208, 2210, and 2212.

[0263] Run the polling scheduler to schedule CQL engines 2212, 2214, and 2216 to executor lists 2208, 2210, and 2212.

[0264] Task scheduler 2204 initiates the actual long-running task.

[0265] The newly launched CQL engine calls a "register" RPC call to the CQL engine tracker (e.g., CQL engine tracker 2202).

[0266] In some embodiments, the locality affinity algorithm of the CQL engine can be described by the following process:

[0267] All CQL engines are started from the driver by the CQL engine tracker. There is no association between the CQL engine and the preferred location.

[0268] The first CQLRDD does not have preferred location information.

[0269] The scheduler will attempt to use the preferred location of the parent RDD to locate the host where the parent RDD resides.

[0270] The first run of CQLRDD associates the CQL engine with the same host.

[0271] The next CQLRDD will set the preferred location information based on the association information set from step 4.

[0272] The scheduler will attempt to run CQLRDD to the preferred location it has set.

[0273] In this embodiment, the CQL engine restart scheduling process can be described as follows:

[0274] Handle two scenarios (rejection, crash).

[0275] Rejected - If the scheduled location is different from the actual location (failed to start according to the schedule).

[0276] To obtain the scheduled executor, use either the old scheduled executor (minus the inactive executor) or the new scheduled executor with `schedulePolicy.rescheduleCQLEngine`.

[0277] Select the executor that is still alive in the list of scheduled locations.

[0278] Start the CQL engine using the scheduled executor.

[0279] The following flowchart illustrates the data flow of the above architecture:

[0280] The CQL engine tracer 2202 in driver 2206 initiates a long-running task for each CQL engine. The CQL engine tracer 2202 exposes its RPCEndpoint to the long-running task.

[0281] Task scheduler 2204 executes long-running tasks to executors 2208, 2210, and 2212 in the cluster.

[0282] As part of a long-running task, the CQL engine runs from executors 2208, 2210, and 2212.

[0283] The CQL engine registers itself with the CQL engine tracker 2202 in the driver 2206 of the CQL lendpoint that has the CQL engine.

[0284] As part of the streaming DAG, a CQLRDD will exist to handle CEP processing. The CQLRDD is processed by either the local CQL engine or a remote CQL engine by consulting the CQL engine tracker. The remote CQL engine is then called via RPC.

[0285] Start the CQL engine

[0286] The CQL engine runs as a long-running task within the cluster. It is started as a regular job by the CQL engine tracer, but it never returns and remains running except in the event of a failure or crash. The CQL engine tracer starts the CQL engine in the cluster according to the following algorithm:

[0287] Determine the number of CQL engines to be activated

[0288] Get the list of executors

[0289] Run the round-robin scheduler to schedule the CQL engine to the list of executors.

[0290] The task tracker initiates the actual long-running task.

[0291] The newly launched CQL engine calls the "register" RPC call to the CQL engine tracer.

[0292] Long-running tasks only return to control if the CQL engine crashes or other malfunctions occur.

[0293] In step #1, the number of CQL engines to start in the cluster is determined. The default number of CQL engines in the cluster is the same as the number of executors in the cluster. Therefore, one CQL engine runs from each executor. This maximum number of CQL engines can be configured.

[0294] In step #2, a list of executor information (executor host and executor ID) is retrieved from the cluster.

[0295] In step #3, the polling scheduler assigns the executor to the CQL engine.

[0296] At step #4, a long-running task is started for each CQL engine. The task scheduler uses the scheduled executor information (executor host and ID) to start the CQL engine in the scheduled executor.

[0297] At step #5, the newly started CQL engine makes a "register" RPC call to the CQL engine tracer. This step initiates the tracing process from the CQL engine tracer, as shown below.

[0298] At step #6, a failure or crash of the CQL engine triggers a recovery process from the CQL engine tracker, as shown below.

[0299] Tracking CQL engine

[0300] In some embodiments, the CQL engine tracer may maintain the following tracing information for each CQL engine.

[0301]

[0302]

[0303] The 'state' parameter maintains the state of the CQL engine. It changes throughout the CQL engine's lifecycle from INACTIVE -> SCHEDULED -> ACTIVE -> INACTIVE. INACTIVE is the initial state of the CQL engine before it is tracked by the CQL engine tracker. SCHEDULED is the state when the CQL engine is scheduled to execute in the executor. ACTIVE is the state when the CQL engine actually runs from the executor.

[0304] 'scheduledLocation' maintains the scheduled location for executing the CQL engine.

[0305] 'runningExecutor' keeps track of the executor location where the CQL engine actually runs.

[0306] 'name' is the name of the CQL engine, which serves as an identifier.

[0307] 'endpoint' is the RPCEndpoint with which to communicate.

[0308] 'errorInfo' is the last known error message in the CQL engine.

[0309] CQL engine restoration

[0310] Long-running tasks only return control to the CQL engine tracer in the event of a CQL engine crash or other failure. The CQL engine tracer uses the following CQL engine restart scheduler to restart the CQL engine. The restart scheduler is invoked in two cases: Rejected and Crashed.

[0311] A crash occurs when the running CQL engine crashes or a long-running task returns due to any failure. A rejection occurs when the scheduled location differs from the actual location (e.g., unable to start from the scheduled executor and from a different executor via the task scheduler). This can occur due to resource issues from the cluster.

[0312] In this embodiment, the CQL engine restart scheduling process can be described as follows:

[0313] Obtain a list of candidate executors using the old scheduled executors in the cluster (minus inactive executors) and the new executors.

[0314] Select the still-living executors from the list of candidate executors.

[0315] The scheduled executor starts a long-running task, which in turn starts the CQL engine.

[0316] Locality Affinity Algorithm

[0317] To support horizontal scalability, the input dataset is partitioned and processed using parallelized distributed data processing. The CQL engine can process multiple partitions using an affinity or association between the queryId and partitionId. To optimize the transformation with minimal network traffic for sending data between executors, this affinity needs to be created while maximizing locality. In one embodiment, to maximize locality, the CQL engine tracker uses the following affinity algorithm.

[0318] All CQL engines are started from the driver by the CQL engine tracker. There is no association between the CQL engine and the preferred location.

[0319] The first CQLRDD does not have preferred location information.

[0320] The Spark® scheduler will attempt to use the preferred location of the parent RDD to co-locate to the executor where the parent RDD resides.

[0321] CQLRDD calls the "getCQLEngine" RPC to the CQL engine tracer.

[0322] The first calculation of a CQLRDD partition will associate (partitionId, queryId) with the CQL engine and the same executor of the CQLRDD.

[0323] The mapping of (partitionId, queryId) to the preferred location in the CQL engine is maintained in the CQL engine tracker.

[0324] The CQL engine returns a CQLRDD, and the RDD is processed by that CQL engine.

[0325] The next CQLRDD will set the preferred location information based on the association information set in step 5.

[0326] The Spark® scheduler will attempt to run the CQLRDD at its preferred location.

[0327] CQLRDD calls the "getCQLEngine" RPC to the CQL engine tracer, and (partitionId, queryId) should already be the same executor.

[0328] Figure 23 This is an example flowchart of process 2300 according to an embodiment of the present disclosure, which describes a set of operations for scheduling and managing multiple CEP engines within a micro-batch-based event processing system. In the embodiment, process 2300 describes... Figure 22 Additional details of the operations described herein. In some examples, process 2300 begins at 2302 by starting a first CQL engine in a cluster of CQL engines. The first CQL engine, as well as any additional CQL engines, can be started using the CQL engine tracing engine. At 2304, the CQL engine tracing engine can also schedule the first CQL engine to process a continuous stream of input events related to the application. At 2306, the CQL engine tracing engine can also track the first CQL engine to be scheduled for execution. At 2308, the CQL engine tracing engine can also execute the first CQL engine to process the continuous stream of input events to generate a set of output events related to the application.

[0329] Automatic data partitioning and parallelism using Group By and Object ID fields.

[0330] In recent years, data stream management systems (DSMs) have been developed that can continuously execute queries on potentially unbounded real-time data streams. For example, a typical DSMS can receive one or more data streams, register queries against those streams, and continuously execute queries as new data appears in the stream. Because this type of continuous query runs for a long time, a DSMS can provide clients with a continuous stream of updated results.

[0331] A typical application in DSMS is designed as a "topology" of the shape of a directed acyclic graph (DAG) that is manipulated or transformed. This topology acts as a pipeline for data transformation.

[0332] Most stream processing systems, including Apache Storm, Spark Streaming, and Flink, provide application programming interfaces (APIs) for application developers to build topologies using different programming languages ​​such as Java, Scala, or Clojure.

[0333] APIs are helpful for programmers to build stream processing applications, but due to the complexity of the code generation layer, they are relatively complex for code generation systems such as Stream Analytics, which generates stream processing applications for users.

[0334] Figure 24 Example architecture 2400 shows that the input pipeline for data transformation can be fed into a pipeline analyzer 2402 and classified by a stage classification module 2404. In some examples, the Stream Analytics code generation layer is responsible for automatically determining the parallelism in the data transformation pipeline by analyzing the pipeline stages. The data transformation pipeline consists of stages, each performing a specific transformation according to its definition. The aggregator stage computes real-time aggregations on the incoming streaming data. If the pipeline stages can be processed on a cluster of nodes, the data transformation pipeline processing can be optimized.

[0335] To compute stages on a cluster of nodes, it is desirable to automatically determine the parallelism characteristics of stage operations and then create a DAG of transformations, where the computation of transformations can be performed on the set of cluster nodes by maximizing parallelism.

[0336] In one embodiment, a data flow management system (DSMS) is constructed that analyzes a user-designed data transformation pipeline, derives partitioning criteria for each stage, and generates an optimized DAG of the transformations, where each stage can run on a set of cluster nodes.

[0337] In some embodiments, the following stages may be included in a pipeline designed by the Stream Analytics platform.

[0338] 1) Query

[0339] 2) Business Rules

[0340] 3) Space

[0341] 4) Type

[0342] A pipeline can consist of one or more stages of the types described above.

[0343] An example of a sample pipeline is shown below:

[0344] Input -> Query -> Query -> Space -> Output

[0345] In some examples, users create pipelines to implement desired business logic. When designing a pipeline, users can select each stage and configure its properties. In some examples, these configuration properties become stage metadata.

[0346] If the stage type is query, then the exposed technology determines the automatic data partitioning of the stage. To transform a user-created pipeline into a DAG that is transformed at runtime, the Stream Analytics platform can perform the following steps:

[0347] 1) Traverse the pipeline from source to sink.

[0348] 2) For each stage

[0349] i) Determine the phase type

[0350] ii) If the stage type is "query", can the platform tag compute transformations for this stage in a distributed manner?

[0351] a) Identify the CQL query associated with the query phase.

[0352] b) Parse the CQL query into a token.

[0353] c) Perform semantic analysis of the parsed query using CQL.

[0354] d) Use various rules to determine query categories

[0355] These rules categorize continuous queries into the following types:

[0356] Stateless, semi-stateful, fully stateful

[0357] e) If the query is stateless, then mark the stage as to be partitioned without any partitioning attributes (standard) 2406. In this way, the stage will depend on the partitioning standard of the parent stage.

[0358] f) If the query is stateful, then mark the phase as unpartitioned 2408. In this way, the phase is executed only on a single cluster of nodes.

[0359] g) If the query is semi-stateful, then mark the stage as to be partitioned using partition attributes 2410. The partition attributes will be obtained from the results of step 2.ii.d. In this way, the stage computation can be partitioned on automatically determined partition attributes.

[0360] 3) For each stage, generate the transformations in the DAG of the data transformation pipeline.

[0361] i) If a stage is marked as to be partitioned without any partitioning attribute 2406, then a transformation is generated for this stage in the DAG without any repartitioning transformation. The number of partitions in a stage will be determined by the number of partitions from the previous stage.

[0362] ii) If a stage is marked as to be partitioned using partition attribute 2410, then a transformation is generated for this stage in the DAG using a repartition transformation. The inputs to the repartition transformation will be the partition attribute and the number of partitions. The repartition transformation will repartition the incoming event stream using the new partitioning criteria.

[0363] iii) If a stage is marked as unpartitioned 2408, then a transformation is generated for that stage in the DAG using a repartition transformation followed by a stage transformation. The inputs to the repartition transformation will be the partition attribute and the number of partitions that will be 1. The repartition transformation will repartition the partitioned / unpartitioned stream into single partitions.

[0364] In some pipelines, if the system does not have enough metadata or if it cannot determine the partition from query analysis, then the system marks the object ID as partition attribute 2412.

[0365] If the stage type is spatial and DSMS is processing a geolocation event stream of moving objects, where each object has a unique identity, then the Stream Analytics platform marks the object ID as the partition attribute for that stage and generates a transformation for that stage in the DAG using partition transformation followed by stage transformation.

[0366] The embodiments of the disclosed technology provide the following features:

[0367] The execution pipeline stages undergo metadata scanning, and the stages are categorized based on CQL query classification.

[0368] Partition attributes are automatically determined by semantic analysis of query execution based on a continuous query language.

[0369] The transformed DAG is generated by using partitioning.

[0370] Existing technologies involve explicitly defining the parallelism characteristics of pipeline stages for users using pipeline data transformation systems. If not specified, the system may process pipeline stages without fully utilizing computational resources.

[0371] The disclosed technology automatically determines data partitioning criteria by analyzing the stages of a pipelined stream processing system. This significantly reduces the complexity of designing data processing pipelines for the Stream Analytics platform.

[0372] Figure 25 This is an example flowchart of process 2500, which describes a set of operations for data partitioning and parallelism in a distributed event processing system. In the embodiment, process 2500 describes... Figure 24 Additional details of the operations described herein. In some examples, process 2500 begins at 2502 by determining the phase of a Continuous Query Language (CQL) query processed by the event processing system. At 2504, the system can be configured to determine the phase type associated with that phase. In some examples, process 2500 may continue at 2506 by determining the transformations to be computed for the phase, at least in part, based on the phase type. Process 2500 may also determine the classification of the CQL query at 2508, at least in part, based on multiple rules. In some examples, process 2500 may include marking the phase as a partitioned phase or an unpartitioned phase at 2510 by applying partitioning criteria to the phase. Furthermore, in some examples, process 2500 may generate transformations in the directed acyclic graph (DAG) of the phase's data transformation pipeline at 2512, at least in part, based on the phase's partitioning criteria. At 2514, process 2500 may determine the partitioning of the phase, at least in part, based on this transformation. Process 2500 may also process the CQL query, at least in part, based on the determined partitioning.

[0373] Explanatory System

[0374] Figures 26-28 An example environment for implementing aspects of this disclosure is shown according to various embodiments. Figure 26 A simplified diagram of a distributed system 2600 for implementing embodiments of the present disclosure is depicted. In the illustrated embodiment, the distributed system 2600 includes one or more client computing devices 2602, 2604, 2606, and 2608, which are configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., via one or more networks 2610. A server 2612 may be communicatively coupled to remote client computing devices 2602, 2604, 2606, and 2608 via network 2610.

[0375] In various embodiments, server 2612 may be adapted to run one or more services or software applications, such as services and applications providing event processing services. In some embodiments, server 2612 may also provide other services, or the software applications may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based or cloud services or under a Software as a Service (SaaS) model to users of client computing devices 2602, 2604, 2606, and / or 2608. Users operating client computing devices 2602, 2604, 2606, and / or 2608 may then interact with server 2612 using one or more client applications to utilize the services provided by these components.

[0376] exist Figure 26 In the depicted configuration, software components 2618, 2620, and 2622 of system 2600 are shown as implemented on server 2612. In other embodiments, one or more components of system 2600 and / or the services provided by these components may also be implemented by one or more of client computing devices 2602, 2604, 2606, and / or 2608. A user operating the client computing device can then utilize one or more client applications to use the services provided by these components. These components can be implemented in hardware, firmware, software, or a combination thereof. It should be understood that various different system configurations are possible and may differ from the distributed system 2600. Therefore, Figure 26 The embodiments shown are examples of distributed systems for implementing the system of the embodiments, and are not limiting.

[0377] Client computing devices 2602, 2604, 2606, and / or 2608 may include various types of computing systems. For example, client devices may include portable handheld devices (e.g., iPhone®, cellular phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 26, and Palm OS. The devices may support various applications (such as various internet-related applications, email, and short message service (SMS) applications) and may use various other communication protocols. Client computing devices may also include general-purpose personal computers, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. Client computing devices may be workstation computers running any of the various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS). Client computing devices may also include electronic devices capable of providing one or more network 2610 communications (such as thin client computers, internet-enabled gaming systems (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices) and / or personal messaging devices).

[0378] Although Figure 26 The distributed system 2600 is shown with four client computing devices, but can support any number of client computing devices. Other devices (such as devices with sensors) can interact with the server 2612.

[0379] The network(s) 2610 in the distributed system 2600 can be any type of network familiar to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk, etc. By way of example only, the network(s) 2610 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any IEEE 2602.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0380] Server 2612 may consist of one or more general-purpose computers, special-purpose server computers (including, as examples, PC (personal computer) servers, UNIX® servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 2612 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization. One or more flexible pools of logical storage devices may be virtualized to maintain virtual storage devices for the server. The virtual network may be controlled by server 2612 using software-defined networking. In various embodiments, server 2612 may be adapted to run one or more services or software applications described in the foregoing disclosure. For example, server 2612 may correspond to a server according to embodiments of this disclosure for performing processing as described above.

[0381] Server 2612 can run any operating system discussed above, as well as any commercial server operating system. Server 2612 can also run various additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA® servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from Oracle, Microsoft, Sybase, IBM, etc.

[0382] In some implementations, server 2612 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 2602, 2604, 2606, and 2608. As an example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party information sources and continuous data streams, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. Server 2612 may also include one or more applications that display data feeds and / or real-time events via one or more display devices of client computing devices 2602, 2604, 2606, and 2608.

[0383] The distributed system 2600 may also include one or more databases 2614 and 2616. These databases may provide mechanisms for storing information, such as event information and other information used by embodiments of this disclosure. Databases 2614 and 2616 may reside in various locations. For example, one or more of databases 2614 and 2616 may reside on non-transient storage media local to server 2612 (and / or within server 2612). Alternatively, databases 2614 and 2616 may be located remotely from server 2612 and communicate with server 2612 via a network-based or dedicated connection. In one set of embodiments, databases 2614 and 2616 may reside in a storage area network (SAN). Similarly, any necessary files for performing functions belonging to server 2612 may be appropriately stored locally on server 2612 and / or remotely. In one set of embodiments, databases 2614 and 2616 may include relational databases, such as those provided by Oracle, adapted to store, update, and retrieve data in response to commands in SQL format.

[0384] The systems depicted in some of the accompanying figures can be provided in various configurations. In some embodiments, the system can be configured as a distributed system, wherein one or more components of the system are distributed across one or more networks in one or more cloud infrastructure systems.

[0385] A cloud infrastructure system is a collection of one or more server computing devices, network devices, and / or storage devices. These resources can be partitioned by a cloud service provider and allocated to its customers in some way. For example, cloud service providers (such as Oracle Corporation in Redwood Shores, California) can provide various types of cloud services, including but not limited to one or more services offered under the Software as a Service (SaaS) category, services offered under the Platform as a Service (PaaS) category, services offered under the Infrastructure as a Service (IaaS) category, or other categories of services that include hybrid services. Examples of SaaS services include, but are not limited to, the ability to build and deliver a suite of on-demand applications (such as Oracle Fusion applications). SaaS services enable customers to utilize applications running on a cloud infrastructure system without requiring customers to purchase software for the applications. Examples of PaaS services include, but are not limited to, services that enable organizations (such as Oracle) to integrate existing applications on a shared, common architecture, and the ability to build new applications that fully utilize shared services provided by the platform (such as Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), etc.). IaaS services typically help manage and control the underlying computing resources, such as storage, networking, and other basic computing resources, for customers to utilize the services provided by SaaS and PaaS platforms.

[0386] Figure 27 This is a simplified block diagram of one or more components of a system environment 2700 according to an embodiment of the present disclosure, through which services provided by one or more components of the embodiment system can be provided as cloud services. In the illustrated embodiment, system environment 2700 includes one or more client computing devices 2704, 2706, and 2708 that can be used by a user to interact with a cloud infrastructure system 2702 providing cloud services. The client computing devices can be configured to operate client applications, such as web browsers, proprietary client applications (e.g., Oracle Forms), or some other application, which can be used by the user of the client computing devices to interact with the cloud infrastructure system 2702 to use the services provided by the cloud infrastructure system 2702.

[0387] It should be recognized that the cloud infrastructure system 2702 depicted in the figures may have other components besides those depicted. Furthermore, the embodiment shown in the figures is merely one example of a cloud infrastructure system that can be incorporated into embodiments of the present invention. In some other embodiments, the cloud infrastructure system 2702 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations or arrangements.

[0388] Client computing devices 2704, 2706, and 2708 can be devices similar to those described above for 502, 504, 506, and 508.

[0389] Although the example system environment 2700 is shown with three client computing devices, any number of client computing devices can be supported. Other devices, such as those with sensors, can interact with the cloud infrastructure system 2702.

[0390] One or more networks 2710 can facilitate data communication and exchange between clients 2704, 2706, and 2708 and cloud infrastructure system 2702. Each network can be any type of network familiar to those skilled in the art that supports data communication using any of a variety of commercially available protocols, including those described above for one or more networks 2710.

[0391] The cloud infrastructure system 2702 may include one or more computers and / or servers, which may include those computers and / or servers described above for server 2712.

[0392] In some embodiments, services provided by a cloud infrastructure system may include a variety of services available on demand to users of the cloud infrastructure system, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, etc. Services provided by a cloud infrastructure system can be dynamically scaled to meet the needs of users of the cloud infrastructure system. A specific instantiation of a service provided by a cloud infrastructure system is referred to herein as a "service instance." Generally, any service available to users from a cloud service provider's system via a communication network (such as the Internet) is referred to as a "cloud service." Typically, in a public cloud environment, the servers and systems that constitute the cloud service provider's system are different from the customer's own local servers and systems. For example, a cloud service provider's system may host applications, and users may subscribe to and use applications on demand via a communication network such as the Internet.

[0393] In some examples, services within a computer network cloud infrastructure may include protected computer network access to storage devices, hosted databases, hosted web servers, software applications, or other services provided to users by the cloud provider, or as otherwise known in the art. For example, services may include password-protected access to remote storage devices in the cloud via the Internet. As another example, services may include web-based hosted relational databases and scripting language middleware engines for private use by networked developers. As yet another example, services may include access to email software applications hosted on a cloud provider's website.

[0394] In some embodiments, cloud infrastructure system 2702 may include a suite of application, middleware, and database service products delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such a cloud infrastructure system is the Oracle public cloud provided by this assignee.

[0395] In various embodiments, cloud infrastructure system 2702 can be adapted to automatically provision, manage, and track customer subscriptions to services provided by cloud infrastructure system 2702. Cloud infrastructure system 2702 can provide cloud services via different deployment models. For example, services can be provided based on a public cloud model, where cloud infrastructure system 2702 is owned by an organization selling cloud services (e.g., owned by Oracle), and the services are available to the general public or businesses in different industries. As another example, services can be provided based on a private cloud model, where cloud infrastructure system 2702 operates only for a single organization and can provide services to one or more entities within that organization. Cloud services can also be provided based on a community cloud model, where cloud infrastructure system 2702 and the services provided by cloud infrastructure system 2702 are shared by several organizations in the relevant community. Cloud services can also be provided based on a hybrid cloud model, which is a combination of two or more different models.

[0396] In some embodiments, the services provided by the cloud infrastructure system 2702 may include one or more services offered under the Software as a Service (SaaS) category, Platform as a Service (PaaS) category, Infrastructure as a Service (IaaS) category, or other service categories that include hybrid services. A customer may subscribe to one or more services provided by the cloud infrastructure system 2702 via a subscription order. The cloud infrastructure system 2702 then performs processing to provide the services in the customer's subscription order.

[0397] In some embodiments, the services provided by the cloud infrastructure system 2702 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform may be configured to provide cloud services that fall into the SaaS category. For example, the SaaS platform may provide the ability to build and deliver on-demand application suites on an integrated development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure used to provide SaaS services. By utilizing the services provided by the SaaS platform, customers can leverage applications running on the cloud infrastructure system. Customers can obtain application services without having to purchase separate licenses and support. A variety of different SaaS services may be provided. Examples include, but are not limited to, services providing solutions for sales performance management, enterprise integration, and business agility for large organizations.

[0398] In some embodiments, platform services may be provided by a cloud infrastructure system via a PaaS platform. The PaaS platform may be configured to provide cloud services that fall into the PaaS category. Examples of platform services may include, but are not limited to, services that enable organizations (such as Oracle) to integrate existing applications on a shared, public architecture and to leverage the shared services provided by the platform to build new applications. The PaaS platform can manage and control the underlying software and infrastructure used to provide PaaS services. Customers can access PaaS services provided by the cloud infrastructure system without having to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), etc.

[0399] By leveraging services provided by a PaaS platform, customers can employ programming languages ​​and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware Service), and Java cloud services. In one embodiment, the database cloud service may support a shared services deployment model that enables organizations to aggregate database resources and provide database-as-a-service to customers in the form of a database cloud. Within the cloud infrastructure system, the middleware cloud service provides customers with a platform for developing and deploying various business applications, and the Java cloud service provides customers with a platform for deploying Java applications.

[0400] Various infrastructure services can be provided by IaaS platforms within cloud infrastructure systems. Infrastructure services facilitate the management and control of underlying computing resources, such as storage devices, networks, and other basic computing resources, so that customers can utilize services provided by SaaS and PaaS platforms.

[0401] In some embodiments, the cloud infrastructure system 2702 may also include infrastructure resources 2730 for providing resources to customers of the cloud infrastructure system for delivering various services. In one embodiment, infrastructure resources 2730 may include a combination of pre-integrated and optimized hardware, such as servers, storage devices, and networking resources, to perform services provided by PaaS and SaaS platforms.

[0402] In some embodiments, resources in cloud infrastructure system 2702 can be shared by multiple users and dynamically reallocated as needed. Furthermore, resources can be allocated to users in different time zones. For example, cloud infrastructure system 2730 can enable a first group of users in a first time zone to utilize the resources of the cloud infrastructure system for a specified number of hours, and then enable the same resources to be reallocated to another group of users located in a different time zone, thereby maximizing resource utilization.

[0403] In some embodiments, multiple internal shared services 2732 may be provided, shared by different components or modules of the cloud infrastructure system 2702 and services provided by the cloud infrastructure system 2702. These internal shared services may include, but are not limited to: security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, cloud-enabled services, email services, notification services, file transfer services, etc.

[0404] In some embodiments, cloud infrastructure system 2702 can provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) within the cloud infrastructure system. In one embodiment, cloud management functionality may include the ability to provision, manage, and track customer subscriptions received by cloud infrastructure system 2702.

[0405] In one embodiment, as illustrated in the figure, cloud management functionality may be provided by one or more modules, such as an order management module 2720, an order orchestration module 2722, an order supply module 2724, an order management and monitoring module 2726, and an identity management module 2728. These modules may include or be provided using one or more computers and / or servers, which may be general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0406] In exemplary operation 2734, a customer using a client device (such as client device 2704, 2706, or 2708) can interact with cloud infrastructure system 2702 by requesting one or more services provided by cloud infrastructure system 2702 and placing an order to subscribe to one or more services provided by cloud infrastructure system 2702. In some embodiments, the customer can access cloud user interfaces (UIs) (cloud UI 2712, cloud UI 2714, and / or cloud UI 2716) and place subscription orders via these UIs. Order information received by cloud infrastructure system 2702 in response to a customer placing an order may include information identifying the customer and the one or more services provided by cloud infrastructure system 2702 that the customer wishes to subscribe to.

[0407] After a customer places an order, the order information is received via cloud UI 2712, 2714 and / or 2716.

[0408] At operation 2736, the order is stored in order database 2718. Order database 2718 can be one of several databases operated by cloud infrastructure system 2718 and operating together with other system components.

[0409] At operation 2738, the order information is forwarded to the order management module 2720. In some cases, the order management module 2720 can be configured to perform order-related billing and accounting functions, such as verifying the order and booking the order after verification.

[0410] At operation 2740, order information is transmitted to order orchestration module 2722. Order orchestration module 2722 can use the order information to orchestrate services and supply resources for customer orders. In some cases, order orchestration module 2722 can use the services of order supply module 2724 to orchestrate resource supply to support subscribed services.

[0411] In some embodiments, the order orchestration module 2722 enables the management of business processes associated with each order and applies business logic to determine whether an order should be made available for provisioning. At operation 2742, upon receiving a new subscription order, the order orchestration module 2722 sends a request to the order provisioning module 2724 to allocate resources and configure those resources required to fulfill the subscription order. The order provisioning module 2724 enables the allocation of resources for the services ordered by the customer. The order provisioning module 2724 provides an abstraction layer between the cloud services provided by the cloud infrastructure system 2700 and the physical implementation layer for providing the resources used to provide the requested services. Therefore, the order orchestration module 2722 can be isolated from implementation details such as whether services and resources are actually provided on demand or pre-provided and allocated / assigned only upon request.

[0412] At operation 2744, once services and resources are supplied, a notification of the supplied services can be sent to customers on client devices 2704, 2706, and / or 2708 via the order provisioning module 2724 of the cloud infrastructure system 2702. At operation 2746, the order management and monitoring module 2726 can manage and track customer subscription orders. In some cases, the order management and monitoring module 2726 can be configured to collect service usage statistics from subscription orders, such as storage usage, data transfer volume, number of users, system uptime, and system downtime.

[0413] In some embodiments, the cloud infrastructure system 2700 may include an identity management module 2728. The identity management module 2728 may be configured to provide identity services, such as access management and authorization services within the cloud infrastructure system 2700. In some embodiments, the identity management module 2728 may control information about customers who wish to utilize services provided by the cloud infrastructure system 2702. Such information may include information authenticating the identities of these customers and information describing what actions these customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). The identity management module 2728 may also include management of descriptive information about each customer and how and by whom this descriptive information is accessed and modified.

[0414] Figure 28 An exemplary computer system 2800 that can be used to implement embodiments of the present disclosure is illustrated. In some embodiments, the computer system 2800 can be used to implement any of the various servers and computer systems described above. Figure 28 As shown, the computer system 2800 includes various subsystems, including a processing unit 2804 that communicates with multiple peripheral subsystems via a bus subsystem 2802. These peripheral subsystems may include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. The storage subsystem 2818 may include a tangible computer-readable storage medium 2822 and system memory 2810.

[0415] Bus subsystem 2802 provides a mechanism for enabling various components and subsystems of computer system 2800 to communicate with each other as desired. While bus subsystem 2802 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2802 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses utilizing any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard, and so on.

[0416] The processing subsystem 2804 controls the operation of the computer system 2800 and may include one or more processing units 2832, 2834, etc. The processing unit may include one or more processors, including single-core or multi-core processors, one or more cores of a processor, or a combination thereof. In some embodiments, the processing subsystem 2804 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 2804 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0417] In some embodiments, the processing unit in the processing subsystem 2804 can execute instructions stored in system memory 2810 or on computer-readable storage medium 2822. In various embodiments, the processing unit can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in system memory 2810 and / or on computer-readable storage medium 2822, potentially including one or more storage devices. With appropriate programming, the processing subsystem 2804 can provide the various functions described above for dynamically modifying documents (e.g., web pages) in response to usage patterns.

[0418] In some embodiments, a processing acceleration unit 2806 may be provided for performing customized processing or for offloading some processing performed by the processing subsystem 2804 in order to accelerate the overall processing performed by the computer system 2800.

[0419] I / O subsystem 2808 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 2800. Generally, the term "input device" is used to include all possible types of devices and mechanisms for inputting information to computer system 2800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, dial wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices such as the Microsoft Kinect® motion sensor, the Microsoft Xbox® 360 game controller, and devices that provide an interface for receiving input using gesture and spoken commands, enabling users to control and interact with the input device. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., "blinking" when taking pictures and / or making menu selections) and translates the eye gesture into input to an input device (e.g., Google Glass®). In addition, user interface input devices may include voice recognition sensing devices that enable users to interact with voice recognition systems (such as the Siri® navigator) via voice commands.

[0420] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointers, gamepads and graphics tablets, and audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission computed tomography, and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0421] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those utilizing liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from computer system 2800 to a user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0422] Storage subsystem 2818 provides a repository or data storage for storing information used by computer system 2800. Storage subsystem 2818 provides a tangible, non-transient, computer-readable storage medium for storing basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the above-described functionality when executed by processing subsystem 2804 can be stored in storage subsystem 2818. The software can be executed by one or more processing units of processing subsystem 2804. Storage subsystem 2818 may also provide a repository for storing data used according to this disclosure.

[0423] The storage subsystem 2818 may include one or more non-transient memory devices, including volatile and non-volatile memory devices. For example... Figure 28 As shown, the storage subsystem 2818 includes system memory 2810 and computer-readable storage medium 2822. System memory 2810 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some implementations, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within computer system 2800, such as during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 2804. In some implementations, system memory 2810 may include multiple different types of memories, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0424] As an example rather than a limitation, such as in Figure 28 As illustrated, system memory 2810 can store application programs 2812, which may include client applications, web browsers, middleware applications, relational database management systems (RDBMS), program data 2814, and operating system 2816. As an example, operating system 2816 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 26 OS, and Palm® OS.

[0425] Computer-readable storage medium 2822 may store programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that enables the processor to provide the aforementioned functionality when executed by processing subsystem 2804 may be stored in storage subsystem 2818. As an example, computer-readable storage medium 2822 may include non-volatile memory, such as hard disk drives, disk drives, optical disc drives such as CD ROMs, DVDs, Blu-ray® discs, or other optical media. Computer-readable storage medium 2822 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc. Computer-readable storage medium 2822 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs), and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Computer-readable medium 2822 can provide storage for computer-readable instructions, data structures, program modules and other data for computer system 2800.

[0426] In some embodiments, the storage subsystem 2800 may also include a computer-readable storage medium reader 2820, which may be further connected to a computer-readable storage medium 2822. Optionally, together with and in combination with the system memory 2810, the computer-readable storage medium 2822 may comprehensively represent remote, local, fixed and / or removable storage devices plus storage media for storing computer-readable information.

[0427] In some embodiments, computer system 2800 may provide support for executing one or more virtual machines. Computer system 2800 may execute programs such as hypervisors to facilitate the configuration and management of the virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, kernel), I / O, and networking resources. Each virtual machine typically runs its own operating system, which may be the same as or different from the operating systems executed by other virtual machines by computer system 2800. Accordingly, multiple operating systems may potentially run concurrently by computer system 2800. Each virtual machine generally runs independently of other virtual machines.

[0428] The communication subsystem 2824 provides interfaces to other computer systems and networks. The communication subsystem 2824 serves as an interface for receiving and sending data to other systems within the computer system 2800. For example, the communication subsystem 2824 enables the computer system 2800 to establish communication channels to one or more client devices via the Internet, for receiving and sending information to and from the client devices. Furthermore, the communication subsystem 2824 can be used to send notifications of successful login or password re-entry from the privileged account manager to the requesting user.

[0429] The communication subsystem 2824 may support both wired and / or wireless communication protocols. For example, in some embodiments, the communication subsystem 1724 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks, such as cellular telephone technology, advanced data network technologies (e.g., 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.11 family of standards), or other mobile communication technologies, or any combination thereof). In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 2824 may provide a wired network connection (e.g., Ethernet).

[0430] The communication subsystem 2824 can receive and send data in various forms. For example, in some embodiments, the communication subsystem 2824 can receive input communications in the form of structured and / or unstructured data feeds 2826, event streams 2828, event updates 2830, etc. For example, the communication subsystem 1724 can be configured to receive (or send) data feeds 2826 in real time from users of social media networks and / or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0431] In some embodiments, the communication subsystem 2824 may be configured to receive data that may be inherently continuous or unbounded without a definite end, in the form of a continuous data stream, wherein the continuous data stream may include an event stream 2828 of real-time events and / or event updates 2830. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0432] The communication subsystem 2824 can also be configured to output structured and / or unstructured data feeds 2826, event streams 2828, event updates 2830, etc. to one or more databases, wherein the one or more databases can communicate with one or more streaming data source computers coupled to the computer system 2800.

[0433] The computer system 2800 can be of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), personal computers, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0434] Due to the constantly changing nature of computers and networks, Figure 28 The description of the computer system 2800 shown in the illustration is intended only as a concrete example. It has more than Figure 28 Many other configurations of the system with more or fewer components are possible. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0435] While specific embodiments of this disclosure have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments of this disclosure are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although embodiments of this disclosure have been described using a specific series of transactions and steps, it will be clear to those skilled in the art that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0436] Furthermore, while embodiments of this disclosure have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments of this disclosure may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination. Accordingly, where a component or module is described as being configured to perform certain operations, such configuration may be implemented, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may utilize various technologies to communicate, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0437] Therefore, the specification and drawings should be considered in an illustrative rather than restrictive sense. However, it will be clear that additions, omissions, deletions, and other modifications and changes may be made therein without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, these embodiments are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims. Modifications and changes include any related combinations of the disclosed features.< / orderamount> < / orderstatus> < / orderid> < / orderamount> < / orderstatus> < / orderid> < / stockprice> < / attributes>

Claims

1. A method for managing continuous query language (CQL) engines, comprising: starting, using a CQL engine tracking engine, a first CQL engine in a cluster of CQL engines; scheduling, using the CQL engine tracking engine, the first CQL engine to process a batch of continuous input event streams related to an application; tracking, using the CQL engine tracking engine, the first CQL engine to be scheduled for execution; and executing, using the CQL engine tracking engine, the first CQL engine to process the batch of continuous input event streams to generate a set of output events related to the application.

2. The method of claim 1, wherein, a state of the first CQL engine comprises at least one of an inactive state, a scheduled state, or an active state.

3. The method of claim 1, further comprising determining a total number of CQL engines to start in the cluster.

4. The method of claim 3, wherein, the first CQL engine is one of the CQL engines in the cluster.

5. The method of claim 3, further comprising providing a long running task to a task scheduler for each of the CQL engines in the cluster.

6. The method of claim 5, further comprising instructing the task scheduler to execute each long running task to an executor, wherein each of the executors implements each of the CQL engines in the cluster.

7. The method of claim 1, wherein, the CQL engine tracking engine is implemented by a driver and the first CQL engine is implemented by an executor.

8. A system, comprising: a memory configured to store computer executable instructions; and a processor configured to access the memory and execute the computer executable instructions to: start, using a CQL engine tracking engine, a first CQL engine in a cluster of CQL engines; schedule, using the CQL engine tracking engine, the first CQL engine to process a batch of continuous input event streams related to an application; track, using the CQL engine tracking engine, the first CQL engine to be scheduled for execution; and execute, using the CQL engine tracking engine, the first CQL engine to process the batch of continuous input event streams to generate a set of output events related to the application.

9. The system of claim 8, wherein, a state of the first CQL engine comprises at least one of an inactive state, a scheduled state, or an active state.

10. The system of claim 8, wherein, the computer executable instructions are further executed to determine a total number of CQL engines to start in the cluster.

11. The system of claim 8, wherein, the first CQL engine is one of the CQL engines in the cluster.

12. The system of claim 10, wherein, the computer executable instructions are further executed to provide a long running task to a task scheduler for each of the CQL engines in the cluster.

13. The system of claim 12, wherein, the computer executable instructions are further executed to instruct the task scheduler to execute each long running task to an executor, wherein each of the executors implements each of the CQL engines in the cluster.

14. The system of claim 8, wherein, the CQL engine tracking engine is implemented by a driver and the first CQL engine is implemented by an executor.

15. A computer-readable medium storing computer executable code that, when executed by a processor, causes the processor to perform operations comprising: tracking, using a CQL engine tracker, a first CQL engine in a cluster of CQL engines; scheduling, using the CQL engine tracker, the first CQL engine to process a batch of continuous input event streams related to an application; tracking, using the CQL engine tracker, the first CQL engine to be scheduled for execution; and executing, using the CQL engine tracker, the first CQL engine to process the batch of continuous input event streams to generate a set of output events related to the application.

16. The computer readable medium of claim 15, wherein, a state of the first CQL engine comprises at least one of an inactive state, a scheduled state, or an active state.

17. The computer readable medium of claim 15, wherein, the operations further comprise determining a total number of CQL engines to start.

18. The computer readable medium of claim 17, wherein, the operations further comprise providing, for each of the CQL engines in the cluster, a long running task to a task scheduler.

19. The computer readable medium of claim 18, wherein, the operations further comprise instructing the task scheduler to execute each long running task to an executor, wherein each of the executors implements each of the CQL engines in the cluster.

20. The computer readable medium of claim 15, wherein, the CQL engine tracker is implemented by a driver and the first CQL engine is implemented by an executor.

Images