Data serialization in distributed event processing systems

Abstract

This application relates to data serialization in distributed event processing systems. A distributed event processing system is disclosed that receives a batch of events via a continuous data stream and performs serialization of data in the batch of events. In certain embodiments, the system identifies a first data type of a first attribute of each event in the batch of events and determines a first type of data compression to be performed on a data value represented by the first attribute. The system determines the first type of data compression to be performed on the data value represented by the first attribute based on the first data type of the first attribute. The system then generates a first set of serialized data values for the first attribute. The system processes the first set of serialized data values against a set of one or more continuous queries to generate a first set of output events.

Description

[0001] This application is a divisional application of Inventive Patent Application 201780054700.0, filed September 12, 2017, entitled "Data Serialization in Distributed Event Processing Systems." BACKGROUND

[0002] In traditional database systems, data is typically stored in the form of tables in one or more databases. Data management languages such as Structured Query Language (SQL) are then used to query and manipulate the stored data. For example, SQL queries can be defined and executed to identify relevant data from data stored in a database. Thus, SQL queries are performed on a finite set of data stored in a database. Additionally, when a SQL query is executed, it is performed once on the finite set of data and produces a finite static result. Thus, databases are best suited for running queries on finite stored sets of data.

[0003] However, many modern applications and systems generate data in the form of continuous data or event streams rather than finite sets of data. Examples of such applications include, but are not limited to, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. These applications have given rise to a need for new types of applications that can process data streams. For example, a temperature sensor can be configured to emit temperature readings.

[0004] Managing and processing data for these types of event stream-based applications involves building data management and query capabilities with a strong temporal focus. Different types of query mechanisms are needed, including long-running queries over continuous unbounded sets of data. While some vendors now offer product suites oriented toward event stream processing, these products still lack the processing flexibility needed to address today's event processing needs. SUMMARY

[0005] Techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for serializing event data received via a continuous stream of events. In one embodiment, a distributed event processing system is disclosed that is configured to receive a batch of events via a continuous stream of data and perform serialization of data in the batch of events. The distributed event processing system can be configured to process the event data for a set of continuous queries to generate a set of output events. The distributed event processing system can also be configured to send the set of output events to a user of the distributed event processing system.

[0006] In one embodiment, a distributed event processing system including a plurality of computing devices is disclosed. A computing device of the plurality of computing devices is configured to receive a batch of events from an event stream. The event stream has an associated schema. The schema identifies one or more attributes of each event received via the event stream. The computing device is included in a cluster of computing nodes in the distributed event processing system, and the cluster of computing nodes includes at least a subset of the plurality of computing devices in the distributed event processing system.

[0007] In certain embodiments, the computing device is configured to identify a first data type of a first attribute of the one or more attributes of each event in the batch of events, and determine a first type of data compression to be performed on data values represented by the first attribute. In certain examples, the first type of data compression is determined based on the first data type of the first attribute. The computing device is then configured to generate a first set of serialized data values for the first attribute and store the first set of serialized data values represented by the first attribute.

[0008] In certain embodiments, the computing device is further configured to process the first set of serialized data values corresponding to the first attribute for a set of one or more continuous queries to generate a first set of output events. The computing device is then configured to send the first set of output events to a user.

[0009] In some embodiments, the computing device is configured to generate a first set of deserialized data values corresponding to the first attribute based on the first set of serialized data values, and process the first set of deserialized data values corresponding to the first attribute for a set of one or more continuous queries to generate a first set of output events.

[0010] In some embodiments, the computing device is configured to generate a first set of deserialized data values corresponding to the first attribute by identifying a first type of data compression to be performed on data values represented by the first attribute and deserializing the first set of serialized data values represented by the first attribute according to the first type of data compression.

[0011] In some embodiments, the computing device is configured to identify a second data type of a second attribute of each event in the batch of events. In one example, the second data type is different from the first data type. The computing device is configured to determine, for the events in the batch of events, a second type of data compression to perform on data values represented by the second attribute. In an example, the second type of data compression is different from the first type of data compression. The computing device is further configured to generate, based on determining the second type of data compression, a second set of serialized data values represented by the second attribute, and store the second set of serialized data values represented by the second attribute. In some examples, the second type of data compression is determined based on the second data type of the second attribute.

[0012] In some embodiments, the computing device is configured to generate, based on the second set of serialized data values, a second set of deserialized data values corresponding to the second attribute, and process the second set of deserialized data values corresponding to the second attribute for the set of one or more continuous queries to generate the first set of output events.

[0013] In some examples, the first type of data compression or the second type of data compression includes at least one of a base compression technique, a value index compression technique, or a precision reduction and value index compression technique.

[0014] In certain embodiments, the computing device is configured to process the first set of serialized data values corresponding to the first attribute to generate the first set of output events by identifying, based on the set of one or more continuous queries, a set of one or more operations to perform on each event in the batch of events, representing the set of one or more operations as a directed acyclic graph (DAG) of transformed continuous query language (CQL) resilient distributed datasets (RDDs), and processing the first set of serialized data values corresponding to the first attribute for the DAG of transformed CQL RDDs.

[0015] In some examples, the first data type of the first attribute is a numeric data type and the second data type of the second attribute is a non-numeric data type.

[0016] The above and below described techniques can be implemented in many ways and in many contexts. Several example implementations and contexts are provided with reference to the following drawings, as described in more detail below. However, the following implementations and contexts are only a small sample of the many. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 Aspects of an example event processing system architecture that provides an environment in which event processing applications can be processed for different execution environments are depicted in accordance with embodiments of the present disclosure.

[0018] Figure 2 is a graphical representation of an event processing network (EPN) for an event processing application according to embodiments of the present disclosure.

[0019] Figure 3 is a simplified block diagram illustrating components of an application processing engine according to embodiments of the present disclosure.

[0020] Figure 4 is an example of a representation of a "common application runtime model" generated by a common application model generator according to embodiments of the present disclosure.

[0021] Figure 5 is an example of a runtime directed acyclic graph (DAG) of components generated by a DAG generator according to embodiments of the present disclosure.

[0022] Figure 6 is a simplified high-level diagram of an event processing system that can incorporate embodiments of the present disclosure.

[0023] Figure 7 is a block diagram illustrating components of a distributed event processing system according to embodiments of the present disclosure.

[0024] Figure 8 is an example flow diagram of a process describing a set of operations for processing an event processing application to generate a common application runtime model of the application according to one embodiment of the present disclosure.

[0025] Figure 9 is an example flow diagram of a process describing a set of operations for processing an event processing application to generate a common application runtime model of the application according to another embodiment of the present disclosure.

[0026] Figure 10 is a simplified block diagram illustrating components of a distributed event processing system according to embodiments of the present disclosure.

[0027] Figure 11 is a high-level dataflow of a process for performing serialization and de-serialization of data in a Resilient Distributed Dataset (RDD) object according to embodiments of the present disclosure.

[0028] Figure 12 is an example flow diagram of a process describing a set of operations by which data included in a batch of events can be serialized according to embodiments of the present disclosure.

[0029] Figure 13A is an example flow diagram of a process describing a set of operations for generating a collection of serialized data values for a numeric attribute of an event according to embodiments of the present disclosure.

[0030] Figure 13Bis an example flow diagram describing a process for a set of operations for generating a serialized collection of data values for a numeric attribute of an event using precision reduction compression techniques, according to embodiments of the present disclosure.

[0031] Figure 13C is an example flow diagram describing a process for a set of operations for generating a serialized collection of data values for a numeric attribute of an event using regular compression techniques, according to embodiments of the present disclosure.

[0032] Figure 13D is an example flow diagram describing a process for a set of operations for generating a serialized collection of data values for a numeric attribute of an event using precision reduction value index compression techniques, according to embodiments of the present disclosure.

[0033] Figure 14 is an example flow diagram describing a process for a set of operations for generating a serialized collection of data values for a non-numeric attribute of an event, according to embodiments of the present disclosure.

[0034] Figure 15 is an example of a manner in which event stream data can be serialized based on determining a data type of an attribute of an event in an event stream, according to embodiments of the present disclosure.

[0035] Figure 16 is an example flow diagram describing a process for a set of operations by which data included in a batch of events can be de-serialized, according to embodiments of the present disclosure.

[0036] Figure 17 is an example flow diagram describing a process for a set of operations for generating a collection of de-serialized data values for one or more attributes of an event in a batch of events, according to embodiments of the present disclosure.

[0037] Figure 18 is an example flow diagram describing a process for a set of operations for generating a collection of de-serialized data values corresponding to numeric or non-numeric attributes of an event in a batch of events using value index compression, according to embodiments of the present disclosure.

[0038] Figure 19 is an example flow diagram describing a process for a set of operations for generating a collection of de-serialized data values corresponding to numeric attributes of an event in a batch of events using precision reduction compression techniques, according to embodiments of the present disclosure.

[0039] Figure 20 is an example flow diagram describing a process for a set of operations for generating a collection of de-serialized data values corresponding to numeric attributes of an event in a batch of events, according to embodiments of the present disclosure.

[0040] Figure 21is an example flow diagram describing a process for scheduling and managing a plurality of CEP engines according to embodiments of the present disclosure.

[0041] Figure 22 is a simplified block diagram illustrating components of a distributed event processing system configured for scheduling and managing a plurality of CEP engines according to embodiments of the present disclosure.

[0042] Figure 23 is an example flow diagram describing a process for scheduling and managing a plurality of CEP engines according to embodiments of the present disclosure.

[0043] Figure 24 is a simplified block diagram illustrating components of a distributed event processing system configured for data partitioning and parallelism according to embodiments of the present disclosure.

[0044] Figure 25 is an example flow diagram describing a process for automatically partitioning and parallelizing data using query clauses and object IDs according to embodiments of the present disclosure.

[0045] Figure 26 depicts a simplified diagram of a distributed system for implementing embodiments of the present disclosure.

[0046] Figure 27 is a simplified block diagram of one or more components of a system environment through which services provided by one or more components of an embodiment system can be provided as a cloud service according to embodiments of the present disclosure.

[0047] Figure 28 illustrates an example computer system that can be used for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0048] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments can be practiced without specific details being presented herein. Furthermore, well-known features can be omitted or simplified in order not to obscure the embodiment being described.

[0049] Overview of Complex Event Processing (CEP)

[0050] Complex event processing (CEP) provides a modular platform for building applications based on event-driven architectures. At the heart of the CEP platform is a continuous query language (CQL), which allows applications to filter, query, and perform pattern matching operations on data streams using a declarative, SQL-like language. Developers can use CQL in conjunction with a lightweight Java programming model to write applications. Other platform modules include a feature-rich IDE, a management console, clustering, distributed caching, event repositories, and monitoring, among others.

[0051] As event-driven architectures and complex event processing have become prominent features of the enterprise computing landscape, more and more enterprises have begun to use CEP technology to build mission-critical applications. Today, mission-critical CEP applications can be found in many different industries. For example, CEP technology is used in the electric power industry to make facilities more efficient by allowing them to react immediately to changes in power demand. CEP technology is used in the credit card industry to detect potentially fraudulent transactions in real-time as they occur. The list of mission-critical CEP applications continues to grow. Building mission-critical applications using CEP technology leads to a need for making CEP applications highly available and fault-tolerant.

[0052] Today's information technology (IT) environment generates a continuous stream of data for everything from monitoring financial markets and network performance, to business process execution and tracking RFID-tagged assets. CEP provides a rich, declarative environment for developing event processing applications to improve the effectiveness of business operations. CEP can process multiple event streams in real-time to detect patterns and trends and provide the necessary visibility for enterprises to exploit emerging opportunities or mitigate developing risks.

[0053] A continuous data stream (also referred to as an event stream) can include a data stream or event stream that can be continuous or unbounded in nature, with no explicit end. Logically, an event stream or data stream can be a sequence of data elements (also referred to as events), each data element having an associated timestamp. A continuous event stream can be logically represented as a bag or set of elements (s, T), where "s" represents the data portion and "T" is in the time domain. The "s" portion is generally referred to as a tuple or event. Thus, an event stream can be a sequence of time-stamped tuples or events.

[0054] In some aspects, the timestamps associated with events in a stream can equal clock time. However, in other examples, the time associated with events in a stream of events can be defined by the application domain and can not correspond to clock time, but can instead be represented, for example, by a sequence number. Thus, the time information associated with events in a stream of events can be represented by a number, a timestamp, or any other information that represents a concept of time. For a system receiving a stream of input events, the events arrive at the system in order of increasing timestamp. There can be multiple events with the same timestamp.

[0055] In some examples, events in a stream of events can represent the occurrence of some mundane event (e.g., when a temperature sensor changes a value to a new value, when the price of a stock symbol changes), and the time information associated with the event can indicate when the mundane event represented by the data stream event occurred.

[0056] For events received via a stream of events, the time information associated with the events can be used to ensure that events in the stream of events arrive in order of increasing timestamp value. This can enable events received in a stream of events to be ordered based on the time information associated with the events. To enable such ordering, timestamps can be associated with events in a stream of events in a non-decreasing manner, such that later generated events have later timestamps than earlier generated events. As another example, if sequence numbers are used as time information, then the sequence number associated with a later generated event can be greater than the sequence number associated with an earlier generated event. In some examples, multiple events can be associated with the same timestamp or sequence number, for example, when the mundane event represented by the data stream event occurs at the same time. Events belonging to the same stream of events can generally be processed in the order imposed on the events by the associated time information, such that earlier events are processed before later events.

[0057] The time information associated with events in a stream of events (e.g., timestamps) can be set by the source of the stream, or alternatively can be set by the system receiving the stream. For example, in certain embodiments, a heartbeat can be maintained on the system receiving the stream of events, and the time associated with events can be based on the time at which the events arrive at the system as measured by the heartbeat. It is possible for two events in a stream of events to have the same time information. Note that while the timestamp ordering requirement is specific to one stream of events, events of different streams can be arbitrarily interleaved.

[0058] The event stream has an associated schema "S" that includes temporal information and a set of one or more named attributes. All events belonging to a particular event stream conform to the schema associated with that particular event stream. Thus, for an event stream (s, T), the event stream can have a schema "S" as (<time_stamp>, <attribute(s)>), where <attributes>The data portion represents the schema of the mode and can include one or more attributes. For example, a schema for a stock ticker event stream can include attributes <stock symbol> and <stockprice>(<stock price>). Each event received via this stream will have a timestamp and two attributes. For example, a stock ticker event stream might receive the following events and associated timestamps:

[0059] …

[0060] (<timestamp_N, <NVDA, 4>)

[0061] (<timestamp_N+1, <ORCL, 62>)

[0062] (<timestamp_N+2, <PCAR, 38>)

[0063] (<timestamp_N+3, <SPOT, 53>)

[0064] (<timestamp_N+4, <PDCO, 44>)

[0065] (<timestamp_N+5, <PTEN, 50>)

[0066] …

[0067] In the above stream, for stream element (<timestamp_N+1, <ORCL, 62>), the event is <ORCL, 62> with attributes "stock_symbol" and "stock_value". The timestamp associated with the stream element is "timestamp_N+1". Thus, a continuous event stream is a stream of events, each event having the same series of attributes.

[0068] As described above, a stream can be a primary source of data on which CQL queries can act. A stream S can be a bag of elements (s, T) where "s" is in the schema of S and "T" is in the time domain. Further, a stream element can be a tuple-time stamp pair, which can be represented as a sequence of timestamped tuple insertions. In other words, a stream can be a sequence of timestamped tuples. In some cases, there can be more than one tuple with the same timestamp. Further, the tuples of an input stream can be requested so as to arrive at the system in timestamp increasing order. Alternatively, a relation (also referred to as a "time- varying relation" and not to be confused with "relational data," which can include data from a relational database) can be a mapping from the time domain to a bag of tuples of schema R. In some examples, a relation can be an unordered, time-varying bag of tuples (i.e., an instantaneous relation). In some cases, at each time instance, a relation can be a bounded set. It can also be represented as a sequence of timestamped tuples, which can include insertions, deletions, and / or updates to capture the changing state of the relation. Similar to a stream, a relation can have a fixed schema to which each tuple of the relation can conform. Additionally, as used herein, a continuous query can generally be capable of processing (i.e., querying against) data of a stream and / or a relation. Further, a relation can reference data of a stream.

[0069] Event processing applications

[0070] In IT environments, both the volume and speed of raw infrastructure and business events are growing exponentially. Whether it is streaming stock data for financial services, streaming satellite data for the military, or real-time vehicle location data for transportation and logistics businesses, companies in multiple industries must process large volumes of complex data in real time. In addition, the explosion of mobile devices and ubiquitous high-speed connectivity has increased the explosion of mobile data. At the same time, the demand for business process agility and execution is also growing. These two trends are putting pressure on organizations to increase their ability to support event-driven architecture implementation patterns. Real-time event processing requires both the infrastructure and the application development environment to perform on the event processing requirements. These requirements often include the need to scale from everyday use cases to extremely high speed data and event throughput (and possibly latency measured in microseconds rather than seconds). In addition, event processing applications must often detect complex patterns in the streams of these events.

[0071] The Oracle Stream Analytics (Oracle Stream Analytics) platform is oriented towards numerous industries and functional areas. Some use cases are as follows:

[0072] Telecommunications: the ability to perform distributed denial of service attack detection and real-time call detail record (CDR) record monitoring.

[0073] Financial Services: Ability to exploit arbitrage opportunities that exist in millisecond or microsecond windows. Ability to perform real-time risk analysis, monitoring and reporting for financial securities trading and calculation of foreign exchange prices.

[0074] Transportation: Ability to create passenger alerts and detect baggage locations in the event of flight variances due to local or destination city weather, ground crew operations, airport security, etc.

[0075] Public Sector / Military: Ability to detect dispersed geographic enemy information, abstract it, and interpret high probability of enemy attacks. Ability to alert the most appropriate resources to respond to emergencies.

[0076] Insurance: Ability to learn and detect potentially fraudulent claims.

[0077] IT Systems: Ability to detect failed applications or servers in real-time and trigger corrective measures.

[0078] Supply Chain and Logistics: Ability to track goods in real-time and detect and report potential delays in arrival.

[0079] Real-time streaming and event processing analytics

[0080] With the explosion of data from an increasing number of connected devices, there is a large amount of dynamically changing data; data is not only moving within an organization, but also outside the firewall. High velocity data brings high value, especially for business processes that are uncertain. However, some of this data loses its operational value in a short time. Big data makes processing time for actionable insights a luxury. Fast data, on the other hand, requires extracting maximum value from highly dynamic and strategic data. It requires faster processing and facilitates taking timely action on data generated as close to real-time as possible. The Oracle Stream Analytics platform provides fast data and responsiveness. Oracle Edge Analytics pushes processing to the network edge in real-time, correlates, filters, and analyzes data to gain actionable insights.

[0081] The Oracle Stream Analytics platform provides the ability to join incoming streaming events with persistent data, providing context-aware filtering, correlation, aggregation, and pattern matching. It provides lightweight, out-of-the-box adapters for common event sources. It also provides an easy-to-use adapter framework for custom adapter development. With this platform, organizations can identify and anticipate opportunities and threats represented by seemingly unrelated events. Its incremental processing paradigm can process events using minimal resources, providing extremely low-latency processing. It also allows it to create very timely alerts and immediately detect missing or delayed events, such as the following:

[0082] Correlated events: If event A occurs, then event B almost always follows within 2 seconds of event A.

[0083] Missing or out-of-sequence events: Events A, B, C should occur in sequence. C is seen immediately after A, without B.

[0084] Causal events: The weight of manufactured items slowly trends lower or readings fall outside an acceptable range. This signals a potential problem or future maintenance need.

[0085] In addition to real-time event sources, the Oracle Stream Analytics platform design environment and runtime execution also support standard-based continuous query execution across both event streams and persistent data stores, such as databases and high-performance data grids. This enables the platform to act as the intelligent core of systems that need to answer questions in a few microseconds or a few minutes to discern patterns and trends that would otherwise be missed. Event processing use cases require the speed of in-memory processing and the mathematical accuracy and reliability of standard database SQL. This platform queries listen to incoming event streams and continuously execute registered queries in memory for each event, while utilizing advanced automated algorithms for query optimization. While based on an in-memory execution model, however, the platform takes full advantage of standard ANSI SQL syntax for query development, ensuring the accuracy and extensibility of query construction. The platform is fully compliant with the ANSI SQL'99 standard and is one of the first products available that supports ANSI SQL review for extensions to standard SQL to enable real-time continuous query pattern matching. The CQL engine optimizes the execution of in-process queries, allowing developers to focus more on business logic than optimization.

[0086] The Oracle Stream Analytics platform allows both SQL and Java code to be combined to provide robust event processing applications. Taking full advantage of standard industry terminology to describe event sources, processors, and event outputs or sinks, the platform provides a metadata-driven approach to defining and manipulating events within the application. Its developers use a visual directed graph canvas and palette for application design to quickly outline the flow of events and processing across both events and data sources. By drag-and-drop modeling and configuration wizards, developers can then enter appropriate metadata definitions to connect the design to the implementation. Where necessary or preferred, developers can enter custom Java code development or directly use The framework encodes high-level concepts into its applications.

[0087] Event-driven applications are often characterized by the need to provide low and deterministic latency in handling extremely high rates of streaming input data. The foundation of the Oracle Stream Analytics platform is a lightweight Java container based on the backplane. It incorporates proven components from the WebLogic JEE application server, such as security, logging, and work management algorithms, but leverages these services in the real-time event processing environment. The integrated real-time kernel provides unique services to optimize thread and memory management supported by the JMX framework, enabling interaction with the container for performance and configuration. Web 2.0 rich Internet applications can communicate with the platform using HTTP publish and subscribe services, which enable them to subscribe to application channels and push events to clients. The platform is small footprint, lightweight Java-based container that provides faster time-to-production and reduces overall cost of ownership.

[0088] The Oracle Stream Analytics platform has the ability to handle millions of events per second with sub-millisecond processing latency on standard commodity hardware (or optimally using Oracle Exalogic and its combination with other engineering systems). This is achieved through a complete "top-down" layered solution that not only has a design focus on high performance event processing use cases, but also has a tight integration with enterprise-class real-time processing infrastructure components. The performance-oriented server clustered platform architecture focuses on reliability, fault tolerance, and extremely high flexibility through tight integration with Oracle Coherence technology, and enables enterprises to scale mission-critical applications across data grids predictably, ensuring continuous data availability and transactional integrity.

[0089] In addition, this platform allows deterministic processing, which means that the same event can be fed to multiple servers or to the same server at different rates, achieving the same result each time. This has incredible advantages compared to systems that rely only on the system clock of the server that is running.

[0090] The above and below described techniques can be implemented in a variety of ways and in many contexts. Several example implementations and contexts are provided with reference to the following drawings, as described in more detail below. However, the following implementations and contexts are only a small sample of the many.

[0091] Distributed event processing

[0092] In some cases, users of an enterprise can desire to quickly identify and respond to important events occurring within the enterprise so that they can take action immediately upon identifying the events. For example, a user can wish to identify important events related to sales orders that have exceeded a threshold within the enterprise. In this case, the user can submit one or more queries to a data store / data warehouse and wish to view the query results within seconds, rather than minutes or hours, so that the user can take action immediately in the event that an anomaly is detected. When acting within seconds or minutes is of significant importance, an enterprise can use real-time data processing and data analysis to process streams of events in real-time for more reactive decision making and immediate action.

[0093] According to embodiments of the present disclosure, a distributed event processing system is disclosed that can use a combination of CEP and distributed event stream processing to process or query very large amounts of data relatively quickly and in real-time. The distributed event processing system can perform real-time processing of data streams by executing queries (e.g., CQL queries) against continuously received data streams (e.g., live feeds). The distributed event processing system can receive one or more continuous data streams, register a continuous query against the data streams, and continuously execute the query as new data appears in the streams. Since this type of continuous query is long-running, the distributed event processing system can provide a continuous stream of results to a user.

[0094] In certain embodiments, the disclosed distributed event processing system can be configured to deploy and execute applications (e.g., event processing applications) by distributing execution of the applications across a cluster of machines within the system. The event processing applications described herein can include a set of rules that can be expressed in the form of a continuous query for processing an input stream. The continuous query can include instructions (e.g., logic) that identify processing to be performed on received events, including what events are to be selected as noteworthy events and output as a result of the query processing. The continuous query can generally perform filtering and aggregation functions to discover and extract noteworthy events from an input event stream. The application can be configured to listen to one or more input event streams, execute the logic (e.g., query) to select one or more noteworthy events from the one or more input event streams, and output the selected noteworthy events via one or more output event streams.

[0095] For example, an event processing application can include a word count application that counts the number of references to a particular word within a collection of input texts. Such an application can include, for example, a continuous query that reads the collection of texts and counts the number of times each word appears in each text. The input texts can contain, for example, short messages received in a stream from an online application, such as a social networking site. or As described above, the continuous query can be configured using the CQL language. For example, to specify a word count task / operation to be performed in the word count stream application, a user can write a CQL query that can take the form, such as: FROM location GROUP BY word SELECT count. Such a query can gather all sentences from a specified location, group the unique words from these sentences into different groups, and then count the number of words in each group.

[0096] By distributing execution of the application across a cluster of machines, the disclosed distributed event processing system can be configured to provide results related to execution of the application to a user quickly and in real-time. The distributed event processing system can be configured to partition data related to the application to separate computing nodes, and each computing node can be maintained as a separate file on a separate computing machine. Each such machine can be configured to execute the query in the application in parallel with other machines with respect to the data maintained on that machine.

[0097] Efficient DAG generation for distributed event processing systems

[0098] In certain embodiments of the present disclosure, an application processing engine for processing information related to an application (e.g., an event processing application) is disclosed. The application processing engine is configured to receive information identifying an event processing application. In certain examples, the event processing application is expressed as an event processing network (EPN) of components, and the information identifying the event processing application information includes information related to various components (e.g., adapters, processors, flows, or event beans) of the event processing application. For example, the information identifying the event processing application can include configuration information, query information, and other types of information related to the application.

[0099] In certain embodiments, the application processing engine can be configured to process the information identifying the application and generate a "common application runtime model" of the application. As described herein, the "common application runtime model" of the application is a representation of the application as a set of one or more configuration blocks, where each configuration block represents a processing stage having associated metadata describing the application. The application processing engine can be configured to convert the "common application runtime model" of the application into one or more generic representations of the application. The application processing engine can then be configured to cause the one or more generic representations of the application to be executed in different execution (runtime) environments supported by different target event processing systems.

[0100] The generation of the "common application runtime model" according to embodiments of the present disclosure enables the execution of generic representations of the application in different physical execution (runtime) environments without requiring the developer (e.g., user) of the application to re-write the application code to adapt to the specific physical execution (runtime) environment of the target engine before executing the application in the target engine.

[0101] The above-described techniques can be implemented in a number of ways and in a number of contexts. Reference is made to the following detailed description, which sets forth various Figures 1-9 Several example implementations and contexts are provided that describe additional details of the manner in which the disclosed distributed event processing system can perform operations related to the deployment, processing, and execution of event processing applications.

[0102] Figure 1 Aspects of an example event processing system architecture 100 that provides an environment in which event processing applications can be processed for different execution environments according to embodiments of the present disclosure are depicted. In embodiments, the architecture (event processing system) 100 includes an application processing engine 110 communicatively connected to one or more user devices 102 via a network 108.

[0103] The network 108 can facilitate data communications and exchanges between the user device 102 and the application processing engine. The network 108 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially-available protocols, including without limitation TCP / IP, SNA, IPX, AppleTalk, and the like. Merely by way of example, the network 108 can be a local area network (LAN), such as Ethernet, Token-Ring network and / or the like; a wide area network; a virtual network, including without limitation a virtual private network (VPN); the Internet; an intranet; an extranet; a public switched telephone network (PSTN); an infra-red network; a wireless network (e.g., a network operating under any of the IEEE 802.1X suite of protocols, the Bluetooth protocol known in the art, and / or any other wireless protocol); and / or any combination of these and / or other networks.

[0104] The user device 102 can be a general purpose personal computer including, by way of example, a personal computer and / or laptop computer running a variety of versions of Microsoft Windows and / or Apple Macintosh operating systems; a cellular telephone or PDA running software such as the Microsoft Windows Mobile and being Internet, e-mail, SMS, Blackberry, or other communication protocol enabled; a workstation computer running any of a variety of commercially-available UNIX or UNIX-like operating systems; or any other computing device. In other embodiments, the user device 102 can be any other electronic device capable of communicating over a network (e.g., the network 108), such as a thin-client computer, an Internet-enabled gaming system, and / or a personal messaging device. Although the example system environment 100 is shown with one user device, in other embodiments any number of users and / or client computing devices can be supported.

[0105] In certain embodiments, the application processing engine 110 can be configured to process event processing applications for execution in different runtime environments. In certain examples, the event processing applications can be generated by a user of the user device 102. For example, the user 102 can use the client application 104 (e.g., a browser) in the user device to build an application (e.g., an event processing application) using the application design user interface 106 provided by the client application 104. As described above, the event processing application can include a set of rules (expressed, for example, in the form of a continuous query) for processing an input stream of data from an event source. The event source can include various data sources, such as a monitoring device, a financial services company, or a motor vehicle. Using this data, the event processing application can identify and respond to patterns, look for special events and alert other applications, or do some other work based on rapidly changing data that requires immediate action.

[0106] The application processing engine 110 can include one or more computers and / or servers, which can be general purpose computers, special purpose server computers (including, for example, PC servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other appropriate arrangement and / or combination. The computing devices that make up the application processing engine 110 can run any operating system or various additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, Java servers, database servers, etc. Example database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM, etc.

[0107] In certain embodiments, the application processing engine 110 can be configured to receive an application (e.g., an event processing application) from the user device 102 as described above and process the information in the application to generate a "common application runtime model" of the application. As described above, the "common application runtime model" of an application is a representation of the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata describing that application. The application processing engine 110 can be configured to convert the "common application runtime model" of the application into one or more generic representations of the application. In certain embodiments, the application processing engine 110 can then be configured to cause these more generic representations of the application to be executed in different execution (runtime) environments supported by different target event processing systems.

[0108] In certain embodiments, the application processing engine 110 can include a common application runtime model generator 112, a common application runtime model optimizer 114, and a target DAG generator 116. These components can be implemented in hardware, firmware, software, or a combination thereof. The common application runtime model generator 112 can be configured to generate a "common application runtime model" for an application based on information associated with the application. The common application runtime model optimizer 114 can be configured to optimize the "common application runtime model" in order to generate an optimized common application runtime model for the application. The target DAG generator 116 can be configured to convert the optimized common application runtime model into one or more generic representations of the application that can be executed by one of the target event stream processing engines (systems). The operations performed by the components 112, 114, and 116 of the application processing engine 110 will be discussed in more detail below with respect to Figure 2 FIGS. 1-3.

[0109] In some embodiments, the target event processing engine 118 can be configured to receive a common application runtime model from the application processing engine 110 and transform the information in the common application runtime model into a platform-specific implementation of the application (i.e., the target event processing application 120), which can be executed in a runtime (execution) environment provided by the target event stream processing engine 118. The target event stream processing engine 118 can then be configured to continuously execute the target event processing application 120 as new data appears in the stream, providing a continuous stream of results to the user. The target event stream processing engine 118 can perform real-time processing of the data stream by executing one or more operations (e.g., CQL queries) defined in the target event processing application 120 for the continuously received data stream (e.g., live feeds). For example, the target event stream processing engine 118 can receive one or more continuous data streams, register the target event processing application 120 for the data streams, and continuously execute one or more queries defined in the target event processing application 120 as new data appears in the stream. Because this type of continuous query runs for a long time, the target event stream processing engine can provide a continuous stream of results to the user. Additional operations performed by the target event stream processing engine 118 regarding Figure 3 Let's discuss this in detail.

[0110] In some embodiments, each target event stream processing engine 118 may represent a specific physical execution environment for executing a target event processing application. For example, a first target event stream processing engine may include a first event streaming platform configured to execute the target event processing application in a first physical execution (runtime) environment, a second target event stream processing engine may include a second event streaming platform configured to execute the application in a second physical execution (runtime) environment, a third target event stream processing engine may include a third event streaming platform configured to execute the application in a third physical execution (runtime) environment, and so on. The first, second, and third event streaming platforms may be different from each other. For example, the first event streaming platform may represent a specific physical execution environment for executing a target event processing application. The managed Oracle Event Processor (OEP) system. The second event streaming platform can represent the first type of distributed event processing platform (such as those managed by Oracle Event Processors). System management (framework), and the third event streaming platform can represent a third type of distributed event processing platform (such as by...). System management frame).

[0111] Figure 2 is a graphical representation of an event processing network (EPN) 200 for an event processing application according to embodiments of the present disclosure. In certain examples, an event processing application can be expressed as a network of components. Such a network of components is often referred to as an event processing network (EPN) 200. The EPN 200 is a conceptual model for expressing event-based interactions and event processing specifications among components of an event processing application. Components of an event processing application can include adapters, streams, processors, business logic plain old Java objects (POJOs), and beans. Each component in the EPN 200 has a role in processing data received via event streams. As noted above, the event processing network (EPN) 200 can include information describing these various components, how the components are connected together, the types of events processed by the application, a continuous query or logic used to select events for use by the application, business rules defined in the application, and the like.

[0112] In certain embodiments, a user of the user device 102 can use an application design user interface (e.g., 106) provided by a client application (e.g., 104) in the user device to generate an EPN 200 for an event processing application. In other embodiments, the user can provide information identifying the application via the application design user interface. Such information can include, for example, one or more continuous queries defined in the application, application parameters specifying a deployment type for the application, a runtime configuration for the application (e.g., a number of executors to use, parallelism parameters, a size of memory, high availability parameters), and the like, and the client application can construct and / or generate the EPN 200 for the event processing application based on such information.

[0113] In certain embodiments, and as shown in Figure 2 The EPN 200 for an event processing application can be composed of the following component types:

[0114] (1) One or more adapters (202, 204) that interface directly to input and output streams and relation sources and sinks. Adapters are configured to understand input and output stream protocols and are responsible for converting event data into a normalized form that can be queried by application processors. Adapters can forward normalized event data to channels or output streams and relation sinks. Event adapters can be defined for various data sources and sinks. In the embodiment shown in Figure 2 The adapters include a StreamOrRelationSourcel (stream or relation source 1) adapter 202 and a StreamOrRelationSource2 (stream or relation source 2) adapter 204.

[0115] (2) One or more channels (206, 208, 210) that act as event processing endpoints. Among other things, the channels are responsible for queuing event data until an event processing agent can act on the event data.

[0116] (3) One or more application processors (or event processing agents) 212 configured to consume the normalized event data from the channels, use queries to process it to select noteworthy events, and forward (or replicate) the selected noteworthy events to an output channel 210.

[0117] (4) One or more beans 214, 216, and 218 configured to listen to the output channel 220 and triggered by the insertion of new events into the output channel 220. In some embodiments, this user code is a simple plain old Java object (POJO). The user application can use a set of external services (such as JMS, Web services, and file writers) to forward the generated events to external event sinks.

[0118] (5) The event beans 214, 216, and 218 can be registered to listen to the output channel 220 and triggered by the insertion of new events into the output channel. In some embodiments, this user code can use the Oracle CEP event bean API so that the bean can be managed by Oracle CEP.

[0119] In one embodiment, the event adapters (202, 204) provide event data to input channels (206, 208). The input channels (206, 208) are connected to CQL processors (212) associated with one or more CQL queries that operate on the events provided by the input channels (206, 208). The CQL processors (212) are connected to an output channel (220) to which the results of the queries are written.

[0120] In some embodiments, an event processing application can be provided with an assembly file that describes the various components of the event processing application, how the components are connected together, and the types of events that are processed by the application. A separate configuration file can be provided that specifies the continuous queries or logic used to select events. In certain embodiments, an XML framework can be used to assemble the information in the event processing application. As will be described in greater detail below, this approach enables the application to easily integrate with existing beans and other lightweight programming frameworks based on a dependency injection mechanism. For example, the assembly file can be a custom extension of the framework context XML configuration file so that the event server can take full advantage of the capabilities of the framework. ​Inversion of Control (IoC) containers, thereby allowing users to seamlessly use beans.

[0121] Figure 3 is a simplified block diagram 300 illustrating components of an application processing engine according to embodiments of the present disclosure. In certain embodiments, the application processing engine 314 can be configured to receive information identifying an event processing application (e.g., 304) from a user of a user device 302 and generate a common runtime application model based on the information. As described above, the event processing application 304 can be generated by the user of the user device 302 using an application design user interface 306 provided by a client application (e.g., 104) in the user device.

[0122] In some examples, the information identifying the application can include information describing various components (e.g., adapters, processors, streams, or event beans) of the application. Such information can include, for example, configuration information, query information, and other types of information. Configuration information can include, for example, information describing various components of the event processing application, how the components are connected together, and the types of events processed by the application. For example, the configuration information can include information describing the event processing application as a network of components (i.e., the EPN 200). Query information can include information specifying a continuous query or logic used to select events by the application. Other types of information can include plain old Java objects (POJOs) and business rules defined in the application.

[0123] In certain examples, the information identifying the application can be specified in a configuration file. For example, each component in the EPN 200 of the event processing application 304 can have an associated configuration file. In other examples, the information in the application 304 can be represented in a single configuration file that includes information for all components in the application. In one implementation, the configuration file can be expressed as a regular XML document whose structure is defined using a standard XML schema based on a configuration schema defined by the common runtime model.

[0124] In some examples, various configuration files such as EPN configuration files, query configuration files, and other files can be used to specify information identifying an application. Examples of an EPN configuration file and a query configuration file for an event processing application are shown below. In the illustrated example, the event processing application is an order event processing application 304 that is configured to receive and process a continuous stream of events, where each event represents an order for an item sold by a company. Each order in the order event stream can include attributes related to the item such as an order identifier, an order status, and an order amount. An example of an EPN configuration file 308 for the event processing application 304 is shown below. The EPN configuration file 308 includes a series of sub-elements, where each sub-element includes configuration information for an event processing component in the event processing application.

[0125] EPN configuration file

[0126] <wlevs:event-type-repository>

[0127] <wlevs:event-type type-name="OrderEvent">

[0128] <wlevs:properties>

[0129] <wlevs:property name="orderId" type="int" / >

[0130] <wlevs:property name="status" type="char" / >

[0131] <wlevs:property name="amount" type="int" / >

[0132] < / wlevs:properties>

[0133] < / wlevs:event-type>

[0134] < / wlevs:event-type-repository>

[0135] <wlevs:adapter id="socketAdapter" provider="socket" / ><wlevs:channel id="orders" event-type="OrderEvent">

[0136] <wlevs; listener ref="orderProcessor" / >

[0137] <wlevs; source ref="socketAdapter" / >

[0138] < / wlevs: channel>

[0139] <wlevs; processor id="orderProcessor" / >

[0140] <wlevs; channel id="outputChannel" event-type="OrderEvent">

[0141] <wlevs; listener ref="outputAdapter" / >

[0142] <wlevs; source ref="orderProcessor" / >

[0143] < / wlevs: channel>

[0144] <wlevs; adapter id="outputAdapter" provider="stdout" / >

[0145] The following shows an example of a query configuration file 310 for the event processing application 304. The query configuration file 310 specifies one or more continuous queries or logic for selecting events in the event processing application.

[0146] Query configuration file

[0147] <processor>

[0148] <name>orderProcessor< / name>

[0149] <rules>

[0150] <query id="helloworldRule"><![CDATA[

[0151] select status,count(*)from orders group by status

[0152] ]]>

[0153]

[0154] < / rules>

[0155] < / processor>

[0156] In certain embodiments, the application processing engine 314 can be configured to generate a "common application runtime model" for an application (e.g., 304) based on configuration information, query information, and other information specified in the EPN configuration file 308, the query configuration file 310, and other files 312. This "common application runtime model" can then be converted by the target DAG generator 320 into one or more generic representations of the application for execution in different physical execution (runtime) environments supported by different target event stream processing engines 226, 228, and 230. The generation of the "common application runtime model" enables the generic representations of the application to be executed in different physical execution (runtime) environments without the developer (e.g., user) of the application having to rewrite the application code to accommodate the particular physical execution (runtime) environment of the target engine before the application can be executed in the target engine. The common application runtime model is independent of the physical execution environment. The platform-independent abstraction allows the system to easily generate DAGs and code for the physical execution environment.

[0157] In certain embodiments, the generation of the "common application runtime model" by the application processing engine 314 can include representing the application as a set of one or more event beans with associated configuration information. Additional details of the manner in which the application processing engine 314 can represent the "common application runtime model" as a set of event beans are described in detail below.

[0158] In certain embodiments, the application processing engine 314 includes a common application runtime model generator 316, a common application runtime model optimizer 318, and a target DAG generator 320. The common application runtime model generator 316 is configured to generate a "common application runtime model" for an application based on information identifying the application specified in the EPN configuration file 308, the query configuration file 310, and other files 312. In certain embodiments, the generation of the "common application runtime model" by the common application runtime model generator 316 involves using a The EPN loader implemented by the application framework loads the EPN configuration file 308, the query configuration file 310, and other files 312 into memory. The result is a set of The inversion of control injection connects a set of bean. The common application runtime model generator 316 is then configured to parse the EPN configuration file 308 using an XML parser such as JAXB (Java Architecture for XML Binding) and set the parsed configuration file to each associated bean in the EPN network. In some examples, each parsed configuration block or file will have an identifier so that the block or file can find the event bean and set the configuration block or file to the event bean. Thus, in embodiments, generating a "common application runtime model" for an application includes representing the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata event bean. Figure 4 An example of a representation of a "common application runtime model" is shown in FIG. 4. In some examples, a "common application runtime model" maintains metadata about the components of an event processing application and does not have any processing logic in it.

[0159] Figure 4 An example of a representation of a "common application runtime model" 400 generated by the common application model generator 316 according to embodiments of the disclosure is depicted. In embodiments, the "common application runtime model" 400 can be generated based on information identifying the application (e.g., from the EPN configuration file 308, the query configuration file 310, and other files 312) and can be represented as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata event bean. In embodiments, the configuration blocks 402, 404, 406, 408, and 410 in the common runtime application model 400 can include the following information:

[0160] SocKetlnbound 402

[0161] host="localhost"

[0162] port=9999

[0163] Channel-1 404

[0164] tableName="orders"

[0165] eventType="OrderEvent"

[0166] relation=false

[0167] CQL Processor 406

[0168] rule="select status, count(*) from orders group by status"

[0169] Channel-2 408

[0170] eventType="OrderEvent"

[0171] StdoutOutbound 410

[0172] <no additional info>

[0173] For example, in configuration block 402, socket inbound represents a "socketAdapter" adapter in the EPN, in configuration block 404, channel-1 represents an "orders" channel in the EPN, in configuration block 406, CQL processor represents an "orderProcessor" processor in the EPN, in configuration block 408, channel-2 represents an "otutputChannel" channel in the EPN, and in configuration block 410, socket outbound 410 represents an "outputAdapter" adapter in the EPN.

[0174] As noted above, representing an event processing application as the "common application runtime model" 400 enables the application to execute in different target event stream processing engines (e.g., 326, 328, or 330) without requiring the user to re-write the code of the application to accommodate the particular physical execution (runtime) environment of the target engine before the application executes in the target engine.

[0175] Returning to Figure 3 discussed above, the common application runtime model generated by the common application runtime model generator 316 can also be optimized by a common application runtime model optimizer 318 in certain embodiments. Optimization of the common application runtime model (e.g., 400) can include, for example, combining multiple stages within a configuration block into a single stage (e.g., combining multiple continuous queries into a single optimized continuous query) or splitting a single stage into multiple stages for parallel processing (e.g., splitting a continuous query into multiple continuous queries that can perform map and reduce operations). For example, consecutive queries without repartitioning can be combined into a single CQL stage with a single CQL process. For example, the two stages "select * from orders where orderStatus = 'open'" and "select count(*) from orders group by orderId" can be combined into one CQL stage with "select count(*) from orders group by orderId where orderStatus = "open". The optimizer can also split a single stage into multiple stages in order to maximize scalability. For example, a fully-stateful query stage "select count(*) from orders" can be split into two stages with partitioning, "select count(*) from order group by orderId" and "select sum(*) from counts". This uses a global count that can process events using partial counts with partitioning, which can then be added to the global count.

[0176] In certain embodiments, the target DAG generator 320 can be configured to convert the optimized common application runtime model into one or more generic representations of the application that can be executed by one of the target event stream processing engines (326, 328, or 330). For example, the target DAG generator 320 can be configured to convert the optimized common application runtime model into a runtime DAG 322 or a target representation 324 of the application depending on the target event stream processing engine in which the application will be executed. For example, if the target event stream processing engine (e.g., 326) is a CQL engine, the target DAG generator 320 can be configured to convert the optimized common application runtime model into a runtime DAG 322 that can be executed by the CQL engine. If the target event stream processing engine (e.g., 328) is a SQL engine, the target DAG generator 320 can be configured to convert the optimized common application runtime model into a target representation 324 of the application that can be executed by the SQL engine. If the target event stream processing engine (e.g., 328) is an Oracle Event Processor (OEP) managed by the distributed system, then the target DAG generator 320 can be configured to convert the configuration blocks in the common application runtime model (or the optimized common application runtime model) into a target representation 324 of the application to be executed by the target event stream processing engine 326. In certain embodiments, the conversion can include a replication of the objects of the common application runtime model with appropriate beans having actual processing logic by the target DAG generator 320. For example, the metadata of a CQL processor bean in the common application runtime model can be replicated to a new instance of the event processing component (e.g., CQL processor) that runs a CQL engine to process input events using a given query (e.g., replicated from the CQL processor in the common runtime application model).

[0177] For example, if the target event stream processing engine (e.g., 328) is an Oracle Event Processor (OEP) managed by the distributed system, If the target event stream processing engine (e.g., 328) is an Oracle Event Processor (OEP) managed by the distributed system, then the target DAG generator 320 can be configured to convert the configuration blocks in the common application runtime model (or the optimized common application runtime model) into a target representation 324 of the application to be executed by the target event stream processing engine 326. In certain embodiments, the conversion can include a replication of the objects of the common application runtime model with appropriate beans having actual processing logic by the target DAG generator 320. For example, the metadata of a CQL processor bean in the common application runtime model can be replicated to a new instance of the event processing component (e.g., CQL processor) that runs a CQL engine to process input events using a given query (e.g., replicated from the CQL processor in the common runtime application model). Figure 5

[0178] Examples of target applications

[0179] Setup

[0180] 1. val sparkConf = new SparkConf

[0181] 2. val sc = new SparkContext(sparkConf)

[0182] 3. val cc = new CQLContext(sc, Seconds(1))

[0183] Event Type, Stream Registration

[0184] ​​4. val orderEvent = EventType("orders", Attribute("orderId", INT), Attribute("status", CHAR), Attribute("amount", INT))

[0185] 5. cc.registerEventType(orderEvent)

[0186] 6. cc.registerStream(orderEvent)

[0187] Load Data

[0188] 7. val lines = cc.socketTextStream("localhost", 9999)

[0189] 8. val rows = lines.map(_.split(","))

[0190] 9. val kv_orders = rows.map(r => (r(1), EventUtil.createTupleValue(orderEvent, r(0).toInt, r(1), r(2).toInt))

[0191] Partition Data

[0192] 10. val orders = rorders.transform(rdd => rdd.partitionBy(new OrderPartitioner(numPartitions)).map { case (k, v) => v} )

[0194] CQL Processing

[0195] 11. val result = cc.cql(orders, "select status, count(*) from orders group by status")

[0196] Output

[0197] 12. val sresult = result.map(x => x.mkString(","))

[0198] 13. sresult.print

[0199] Figure 5 This is an example of a runtime directed acyclic graph (DAG) 500 of components generated by a target DAG generator according to embodiments of this disclosure. In some embodiments, and as described above, when the execution (runtime) environment of the target application is a distributed event processing system (e.g., In a distributed event processing system, the target DAG generator 320 can be configured to transform objects (402, 404, 406, 408, and 410) in a common application runtime model (e.g., 400) into runtime DAGs 500 representing the components of that application.

[0200] In this embodiment, the runtime DAG of the components includes the following components: SocketText 502, Map-1 504, Map-2 506, PartitionBy 508, CQL 510, Map-3 512, and Print 514. Figure 4 (As shown) SocketInbound 402 is converted to SocketText 502, Map-1504, and Map 506. The SocketText 502 component includes line 7 of the example of the target application shown above, which loads a string from the socket. The Map-1504 component includes line 8 of the example of the target application, which converts the string into comma-separated values. The Map-2506 component includes line 9 of the example of the target application, which converts the comma-separated values ​​into tuples. CQL processor 406 and Channel-1404 are converted to PartitionBy 508 and CQL 510. The PartitionBy 508 component includes line 10 of the example of the target application, which creates group-based partitions according to the standards in CQL. The CQL 510 component includes line 11 of the example of the target application, which is the main CQL processing stage. Channel-2 408 and StdoutOutbound 410 are converted to Map-3512 and Print 51. The Map-3 512 part includes line 12 from the example of the target application, which converts the tuple into a comma-separated string, and the Print 514 part includes line 13 from the example of the target application, which prints the output string to the stdout console.

[0201] Figure 6 A simplified high-level diagram of an event handling system 600 that can be incorporated into embodiments of the present disclosure is depicted. In embodiments, the event handling system 600 may represent a system composed of… Oracle Event Processor (OEP) system. The event processing system 600 can include one or more event sources (604, 606, 608), an event processing service (EPS) 602 (also referred to as a CQ service 602) configured to provide an environment for processing event streams, and one or more event sinks (610, 612). The event sources generate event streams that are received by the EPS 602. The EPS 602 can receive one or more event streams from one or more event sources.

[0202] For example, as shown in Figure 6 The EPS 602 receives a first input event stream 614 from the event source 604, a second input event stream 616 from the event source 606, and a third event stream 618 from the event source 608. One or more event processing applications (614, 616, and 618) can be deployed on and executed by the EPS 602. The event processing applications executed by the EPS 602 can be configured to listen to one or more input event streams, process events received via the one or more event streams based on processing logic that selects one or more events from the input event streams as noteworthy events. The noteworthy events can then be sent to one or more event sinks (610, 612) in the form of one or more output event streams. For example, in Figure 6 The EPS 602 outputs a first output event stream 620 to the event sink 610 and a second output event stream 622 to the event sink 612. In certain embodiments, the event sources, event processing applications, and event sinks are decoupled from each other such that any of these components can be added or removed without causing changes to the other components.

[0203] In one embodiment, the EPS 602 can be implemented as a Java server that includes a lightweight Java application container with shared services, such as an Equinox OSGi-based container. In some embodiments, the EPS 602 can support ultra-high throughput and microsecond latency for processing events, for example, by using JRockit Real Time. The EPS 602 can also provide a development platform (e.g., a complete real-time end-to-end Java event-driven architecture (EDA) development platform) that includes tools for developing event processing applications (e.g., Oracle CEP Visualizer and Oracle CEP IDE).

[0204] The event processing applications are configured to listen to one or more input event streams, execute logic (e.g., queries) for selecting one or more noteworthy events from the one or more input event streams, and output the selected noteworthy events to one or more event sinks via one or more output event streams. Figure 6 An in-depth analysis is provided for one such event processing application 614. As shown in Figure 6 The event processing application 614 is configured to listen to an input event stream 618, execute a continuous query 630 that includes logic for selecting one or more noteworthy events from the input events 618, and output the selected noteworthy events to an event sink 612 via an output event stream 622, as shown in

[0205] While the event processing application 614 in Figure 6 is shown as listening to one input stream and outputting selected events via one output stream, this is not limiting. In alternative embodiments, an event processing application can be configured to listen to multiple input streams received from one or more event sources, select events from the monitored streams, and output the selected events to one or more event sinks via one or more output event streams. The same query can be associated with more than one event sink and with different types of event sinks.

[0206] Due to its unbounded nature, the amount of data received via an event stream is generally very large. Therefore, it is generally impractical and undesirable to store or archive all the data for query purposes. The processing of an event stream requires that the events be processed in real-time as they are received at the EPS 602 without storing all the received event data. Thus, the EPS 602 provides a special query mechanism that enables the processing of events as they are received at the EPS 602 without storing all the received events.

[0207] Event-driven applications are rule-driven, and these rules can be expressed in the form of continuous queries for processing input streams. A continuous query can include instructions (e.g., logic) that identify the processing to be performed on received events, including what events are to be selected as noteworthy events and output as a result of the query processing. A continuous query can be persisted to a data store and used to process an input stream of events and generate an output stream of events. A continuous query typically performs filtering and aggregation functions to discover and extract noteworthy events from an input event stream. Thus, the number of outbound events in the output event stream is generally much lower than the number of events in the input event stream from which the outbound events are selected.

[0208] Unlike SQL queries that run once on a finite dataset, a continuous query that has been registered by the application with EPS 602 for that event stream can be executed each time an event is received in a specific event stream. As part of the continuous query execution, EPS 602 evaluates the received events based on the instructions specified by the continuous query to determine whether to select one or more events as noteworthy events and output them as the result of the continuous query execution.

[0209] Continuous queries can be programmed using different languages. In some embodiments, continuous queries can be configured using CQL, provided by Oracle Corporation and used by Oracle's Complex Event Processing (CEP) product. Oracle's CQL is a declarative language that can be used to program queries that can be executed against a stream of events (called CQL queries). In some embodiments, CQL is based on SQL and has added constructs to support the processing of streaming event data.

[0210] It should be recognized that, Figure 6 The system 600 described in the text can have, except Figure 6 Other components besides those depicted. Additionally, Figure 6 The embodiments shown are merely one example of a system that can be incorporated into embodiments of this disclosure. In some other embodiments, system 600 may have more than Figure 6 The system may contain more or fewer components, may combine two or more components, or may have different component configurations or arrangements. System 600 can be of various types, including service provider computers, personal computers, portable devices (e.g., mobile phones or devices), workstations, network computers, mainframes, kiosks, servers, or any other data processing systems.

[0211] Figure 7 This is a simplified block diagram 700 illustrating the components of a distributed event processing system 710 according to an embodiment of the present disclosure. Figure 7 The embodiment shown is an example of a distributed event processing system that can be incorporated into embodiments of this disclosure. In some other embodiments, system 710 may have more than Figure 7 The system may contain more or fewer components, may combine two or more components, or may have different component configurations or arrangements. System 710 may be any type of computing device, such as, but not limited to, mobile computing devices, desktop computing devices, thin client computing devices and / or cloud computing devices, servers, or any other data processing system.

[0212] In some examples, the distributed event processing system 710 can be comprised of a pre-integrated and optimized combination of software resources, hardware resources, networking resources, and other resources. Hardware resources can include, but are not limited to, servers, data storage devices, servers, printers, and the like. Software resources can include, but are not limited to, computing programs, applications (e.g., cloud-based applications, enterprise applications, or any other applications), computer program products (e.g., software), services (e.g., cloud-based services), and the like. Data resources can include, but are not limited to, any accessible data objects, such as files (e.g., networked files or directory information), databases, and the like.

[0213] In certain embodiments, the distributed event processing system 710 can include a receiver 704 and a cluster of compute nodes 708. The receiver 704 can be configured to receive a continuous stream of input events 702 and discretize (partition) the stream of events into one or more batches of events 706 of a particular duration (e.g., X seconds) for subsequent processing by the cluster of compute nodes 708 in the distributed event processing system 710. Each batch of events is referred to herein as a Dstream. In some examples, each Dstream is internally represented by the receiver 704 as a Resilient Distributed Dataset (RDD), which is a snapshot of all input data (events) stream ingested during a specified time period (i.e., in a batch of events). Thus, in some embodiments, the input data stream 702 is represented as a sequence of Dstreams, where each Dstream is internally represented as an RDD, and each RDD includes events (tuples) received during a particular batch interval. In certain examples, each RDD represents an immutable collection of partitioned elements that can be stored in cache memory and executed in parallel in the distributed event processing system.

[0214] In certain embodiments, the cluster of compute nodes 704 can be configured to partition the data contained in each RDD across the cluster of compute nodes and perform operations on the data in parallel for a set of queries defined in an application and provide the results of the processing to users of the distributed event processing system. Thus, the cluster of compute nodes 708 can be configured to distribute the processing of event data in the RDDs across the cluster of compute nodes 708 and provide results related to the execution of the application on the event data to users quickly and in real-time. In embodiments, the distributed event processing system 710 can be configured using the Spark Streaming framework to perform distributed and real-time processing of continuous data streams and deployment of event processing applications.

[0215] Figure 8 ​is an example flow diagram of a process 800 according to one embodiment of the disclosure, the process 800 describing a set of operations for processing an event processing application to generate a common application runtime model for the application. In certain embodiments, the process 800 can be performed by one or more components (e.g., 316, 318, and 320) of the application processing engine (314) described in Figure 3 The process 800 begins at 802 by receiving information identifying an application. Such information can include, for example, information describing various components (e.g., adapters, processors, streams, or event beans) of the application, such as configuration information, query information, and other types of information. As described above, such information can be expressed in configuration files (308, 310, and 312) as described in Figure 3

[0216] At 804, the process includes generating a "common application runtime model" for the application based on the information identifying the application. In embodiments, generating a "common application runtime model" for the application can include representing the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata. The configuration blocks can include an inbound socket event bean, an outbound socket event bean, a continuous query language (CQL) processor event bean, or one or more channel event beans as described with respect to Figure 4

[0217] At 806, the process includes converting the "common application runtime model" for the application into a first generic representation of the application. The first generic representation of the application can be configured to be executed in a first target event processing system of a plurality of target event processing systems. In one example, converting the "common application runtime model" for the application into a first generic representation of the application can include converting the configuration blocks in the common application runtime model into a runtime DAG of components for the application. Figure 5 An example of a runtime DAG of components generated for an application is shown in

[0218] In some embodiments, at 808, the process includes sending the first generic representation of the application to the first target event processing system for execution by the first target event processing system. In one example, the first target event processing system is a distributed event processing system.

[0219] Figure 9 is an example flow diagram of a process 900 according to another embodiment of the disclosure, the process 900 describing a set of operations for processing an event processing application to generate a common application runtime model for the application. In certain embodiments, the process 900 can be performed by one or more components (e.g., 316, 318, and 320) of the application processing engine (314) described in Figure 3 ​​One or more components (e.g., 316, 318, and 320) of the application processing engine (314) described above are executed. The process 900 begins by receiving information identifying an application at 902. As described above, such information can include configuration information, query information, and other types of information describing various components of the application.

[0220] At 904, the process includes generating a "common application runtime model" of the application based on the information identifying the application. In embodiments, generating a "common application runtime model" for the application can include representing the application as a set of one or more configuration blocks, where each configuration block represents an event bean with associated metadata. The configuration blocks can include, as described above, an inbound socket event bean, an outbound socket event bean, a continuous query language (CQL) processor event bean, or one or more channel event beans. Figure 4

[0221] At 906, the process includes converting the "common application runtime model" of the application into a second universal representation of the application. The second universal representation of the application can be configured to be executed in a second target event processing system of the plurality of target event processing systems. In some examples, the second target event processing system can be different from the first target event processing system. In one example, converting the "common application runtime model" of the application into the second universal representation of the application can include converting the configuration blocks in the common application runtime model into a target representation of the application.

[0222] In some embodiments, at 908, the process includes sending the second universal representation of the application to the second target event processing system for execution by the second target event processing system. In one example, the second target event processing system is an event processor (OEP) system.

[0223] Serialization and de-serialization of event data

[0224] In certain embodiments, the disclosed distributed event processing system can be configured to perform serialization and de-serialization of event data received via continuous event streams. Serialization and de-serialization of event data enables the conversion of complex data objects in memory into sequences of bits that can be communicated to computing nodes in the distributed event processing system. Serialization and de-serialization of event data enables processing nodes in the distributed event processing system to efficiently store and represent data prior to processing of the event data by the distributed event processing system. Moreover, serialization and de-serialization of event data reduces latency in exchanging input events and output events between processing nodes in the distributed event processing system and improves overall performance of the distributed event processing system.

[0225] ​The above techniques can be implemented in a variety of ways and in many contexts. See below. Figures 1-10 Several example implementations and contexts are provided, and these diagrams illustrate additional details of how the disclosed distributed event processing system can perform operations related to the deployment, processing, and execution of event processing applications.

[0226] Figure 10 This is a simplified block diagram 1000 illustrating components of a distributed event processing system according to an embodiment of the present disclosure. The distributed event processing system 1000 can be coupled with... Figure 1 The distributed event processing system 110 described herein is the same as or similar to that described in the document. Figure 10 The embodiment shown is an example of a distributed event processing system that can be incorporated into embodiments of this disclosure. In other embodiments, the distributed event processing engine may have more... Figure 10 The components shown may be more or fewer, two or more components may be combined, or they may have different component configurations or arrangements. These components may be implemented using hardware, firmware, software, or a combination thereof. In some embodiments, the software may be stored in memory (e.g., a non-transitory computer-readable medium), a memory device, or some other physical memory, and may be executed by one or more processing units (e.g., one or more processors, one or more processor cores, one or more GPUs, etc.). Therefore, Figure 10 The embodiments shown are examples of a distributed event processing engine for implementing the system of the embodiments and are not intended to be limiting.

[0227] In some embodiments, the distributed event processing system 1002 may include a receiver 1004, an application deployment module 1008, and a cluster of computing nodes 1012. The receiver 1006 may be capable of... Figure 2 As shown (e.g., from event sources 204, 206, or 208), a continuous stream of input data 1004 is received, which is then divided into one or more batches of events 1010, referred to herein as Dstreams. As described above, each Dstream (i.e., event batch) comprises all the input data (event) streams ingested during a specified time period and can be represented internally by receiver 1006 as an RDD object, which is an immutable collection of partitioned elements that can be executed in parallel across a cluster of compute nodes 1012 in the distributed event processing system 1002.

[0228] The application deployment module 1006 can be configured to deploy an application (e.g., an event processing application) for processing and execution by the computing nodes in the computing node cluster 1012. An application described herein can refer to a computer program (e.g., built by a user) of a distributed event processing system. For example, an application can include a word count application that counts the number of references to a particular word within a collection of input text. Such an application can be built using, for example, one or more continuous queries that read the collection of text and count the number of times each word appears in each text. The input text can contain, for example, short messages received in a stream from an online application such as a or ). As described above, continuous queries can be configured using the CQL language. For example, to specify a word count task / operation to be performed in a word count streaming application, a user can write a CQL query that can take the form: SELECT count FROM location GROUP BY word. Such a query can gather all sentences from a specified location, group the unique words from those sentences into different groups, and then count the number of words in each group.

[0229] In certain embodiments, the application deployment module 1008 can be configured to receive information identifying an application from a user of the distributed event processing system. For example, the application deployment module 1008 can be configured to receive information identifying an application via an application design user interface in the application deployment module 1008. The information identifying the application can include a set of one or more continuous queries defined in the application. The information identifying the application can also include application parameters associated with the application. The application parameters can include, for example, a deployment type parameter that specifies a deployment type (e.g., a "clustered mode") of the application on the node cluster 1012. Additional application parameters can include parameters related to the runtime configuration of the application (e.g., a number of executors to use, parallelism parameters, size of memory, high availability parameters, etc.).

[0230] After receiving the information related to the application, in certain embodiments, the application deployment module 1008 can then be configured to send instructions to the computing node cluster 1012 to deploy the application on the computing nodes in the cluster. In certain examples, the computing node cluster 1012 can be configured to deploy the application to a master computing node 1014 on the computing node cluster 1012. The master computing node 1014 can be configured to store an "application context" of the application. The "application context" can include, for example, the contents of the application such as the topology of the application, scheduling information, application parameters, etc.

[0231] In certain embodiments, the master compute node 1014 can be referred to as a "driver," or an application master that runs / executes an application. The driver can be defined as a process that runs the main() function of an application and creates an "application context" for the application. The driver can be responsible for driving the application and requesting resources from the resource manager 1016. The resource manager 1016 can be a service that gets resources for compute nodes on the cluster 1012 to execute an application. To run / execute an application on the cluster, the master compute node 1014 connects to the resource manager 1016, which then allocates resources for the application. Once connected, the master compute node 1014 gets one or more executors on one or more compute nodes (also referred to as worker nodes 1018) in the cluster. An executor is a process that runs computations for an application and stores data. The master compute node 1014 sends application code (e.g., defined by a JAR file) to the executors. Based on the transformations and actions defined in the application, the master compute node 1014 can send tasks 1020 to the executors.

[0232] In certain embodiments, the master compute node 1014 can include a DAG generator 1022, a DAG scheduler 1024, a task scheduler 1026, and application context information 1028. As described above, the application context information 1028 can include information about the application, such as the topology of the application, scheduling information, application parameters, etc. The DAG generator 1022 can be configured to define and / or create a directed acyclic graph (DAG) of the RDD objects based on the RDD objects it receives from the receivers. In some examples, the DAG generator 1022 can represent the DAG of the RDD objects as an RDD lineage graph of all the RDDs it has received during a certain time interval. Each RDD object in the RDD lineage graph maintains a pointer to one or more parents and metadata about the type of relationship that the RDD object has with its parents. The RDD lineage graph also identifies a DAG of the transformations to be performed for each RDD object. When the master compute node 1014 is requested to run a job in an application, the DAG generator 1022 executes the DAG of the transformations. A transformation can identify one or more operations to be performed on an RDD object to transform the data in the RDD object from one form to another. For example, the operations can be defined as part of the application generation process performed by the application deployment module 1008. When the operations are applied to the RDD object, a new RDD object is obtained with the transformed data. Examples of operations performed on an RDD object can include, for example, map, filter, flatMap, reduce, union, groupByKey, distinct, join, collect, count, etc. The DAG of transformations involving the CQL language can be referred to herein as CQL transformations.

[0233] The DAG scheduler 1024 is configured to generate a physical execution plan based on the RDD lineage graph generated by the DAG generator. In one embodiment, the DAG scheduler 1024 generates the physical execution plan by splitting the RDD lineage graph into a plurality of stages, where each stage is identified based on the transformations that need to be performed for the data in each RDD object. For example, if the transformations to be performed on an RDD object include a map transformation and a reduce transformation, then the map transformations can be grouped together into a single stage and the reduce transformations can be grouped together into another stage. The DAG scheduler 1024 then submits the stages to the task scheduler 1026.

[0234] The task scheduler 1026 divides the application (job) into multiple stages. Each stage consists of one or more tasks. In one embodiment, the number of tasks for a particular stage is determined based on the number of partitions of the input data in the RDD objects. For example, and as described above, the DAG scheduler 1024 can schedule all map operations to a single stage. The stages are then passed to the task scheduler 1026, and the task scheduler 1026 launches the tasks via the resource manager. The tasks are then executed by the executor nodes 1018. The task scheduler 1026 identifies the nodes in the cluster of compute nodes 1012 that will execute the operations defined in the application (job) for each RDD object (i.e., each batch of events being processed).

[0235] In certain embodiments, when an executor node 1018 receives an RDD object, the executor node 1018 performs serialization and de-serialization of the data in the RDD object if the RDD object needs to be sent to other executor (worker) nodes in the cluster of compute nodes. As described above, the processing of the data in the RDD object can involve executing one or more continuous queries defined in the application. In embodiments, the executor node (e.g., 1018) identified by the task scheduler 1026 can invoke a CQL engine (such as the CQL processor 230) to perform the processing of the data in the RDD object and return the results of the processing to the master compute node 1014. The manner in which the executor node 1018 can perform serialization and de-serialization of the data in the RDD object prior to its processing is discussed in detail below with respect to Figure 11

[0236] Figure 11 A high-level dataflow of a process for performing serialization and de-serialization of data in an RDD object is depicted in accordance with embodiments of the present disclosure. In certain examples, when a node (e.g., a master node or an executor node) in the cluster of compute nodes receives an RDD object from a receiver for processing, one or more operations in Figure 10 may be performed by the node. In one set of operations, the receiver 1102 receives an input data stream and divides the input data stream into one or more batches of events (tuples) for a particular duration of time (e.g., X seconds). In one embodiment, and as described with respect to Figure 10 each batch of events (tuples) can be internally represented by the receiver as an RDD object. The DAG generator 1104 receives the RDD objects and creates a DAG 1106 of the RDD objects. As described above in Figure 10 In certain examples, the DAG 1106 of the RDD objects includes CQL transformations 1108 to be performed for each RDD object. When a master compute node is requested (e.g., as described with respect to Figure 10 ​The DAG generator 1104 performs a CQL transformation 1108 to process a set of input tuples represented by an RDD object when the job is run in the application 1014) as shown in FIG. 11. In some examples, the set of input tuples to be processed can be obtained from a parent transformation of the CQL transformation 1108. Then, a child transformation of the CQL transformation 1108 invokes a specific operation to be performed on the set of input tuples as represented in the CQL transformation 1108.

[0237] In certain examples, the CQL transformation 1108 invokes a batch serializer process 1112 for the set of input tuples in the RDD object to perform serialization of data in the RDD object. In embodiments, the batch serializer process 1112 can be executed by a node (e.g., an executor node) in the cluster of computing nodes that is processing the RDD object. As described above, the data in the RDD object represents a batch of input tuples (events) received via the event stream. The batch serializer process 1112 serializes the data in the RDD object, and the serialized batch of results from the batch serializer process is sent over the network to a CQL engine 1116 to process the set of input tuples. In certain embodiments, the node that is processing the RDD object can invoke the CQL engine 1116 to process the set of input tuples in the RDD object. For example, the CQL engine 1116 can be an event processing engine deployed on the executor node (e.g., Apache Flink®) that is configured to process the set of input tuples based on the processing logic (operations / transformation) defined in the application. Figure 6 The CQL engine 1116 can be configured to receive the set of input tuples, process the set of input tuples based on the processing logic (operations / transformation) defined in the application, and generate a set of output tuples as a result of the processing, as described in 630) of FIG. 11. The CQL engine 1116 can be configured to receive the set of input tuples, process the set of input tuples based on the processing logic (operations / transformation) defined in the application, and generate a set of output tuples as a result of the processing, as described in 630) of FIG. 11.

[0238] In certain embodiments, the CQL engine 1116 can invoke the batch deserializer process 1114 on the serialized data chunk it receives from the batch serializer process 1112 for the data in the RDD object before processing the data. This is because the serialized chunk is in a binary or wire format that is suitable for transmission over a network, and needs to be deserialized into JAVA objects for the CQL engine 1116 to be able to process. Thus, the result of the deserialization process is a set of input tuples in a form that can be processed by the CQL engine 1116. The CQL engine 1116 processes the set of input tuples and generates a set of output tuples based on the processing. In certain examples, the CQL engine 1116 invokes another batch serializer process 1118 to serialize the set of output tuples, and the result of the serialization is a serialized chunk of output tuples. When the serialized set of output tuples is received by the DAG generator, the CQL transformation 1108 invokes another batch deserializer process 1120 on the serialized chunk of output tuples received. The result of the batch deserializer process 1120 is a set of deserialized output tuples. The CQL transformation 1108 returns the set of output tuples to a child transformation in the CQL transformation to perform the next set of processing operations on the data in the RDD object. In some embodiments, the set of output tuples 1110 is then sent to a user of the distributed event processing system.

[0239] In certain embodiments, the above-described batch serializer processes and batch deserializer processes can be performed by software modules or instructions executed by a node (e.g., an executor node) in a cluster of nodes in a distributed event processing system that is processing an RDD object. Additional details of the operations performed by the batch serializer processes and batch deserializer processes will be discussed in detail below with respect to Figures 12-15 FIGS. 12A and 12B.

[0240] Figure 12 is an example flowchart of a process 1200 that describes a set of operations by which data included in a batch of events can be serialized, in accordance with an embodiment of the present disclosure. In certain embodiments, the process 1200 can be performed by a batch serializer process (1112) in a distributed event processing system. As described above, in certain embodiments, the batch serializer process can be invoked by a master compute node (e.g., 1014 as shown in FIG. 11) when the master compute node is requested to run a job / operation defined in an application. As described above, the master compute node identifies a node (e.g., an executor node 1018) in a cluster of compute nodes 1012 in a distributed computing system 1002 to process a batch of events for a job / operation defined in an application and generate a set of output events as a result of the processing. In certain embodiments, processing the batch of events can include serialization of data in the batch of events. Figure 10 In certain embodiments, the batch serializer process can be invoked by the master compute node when the master compute node is requested to run a job / operation defined in an application. As described above, the master compute node identifies a node (e.g., an executor node 1018) in a cluster of compute nodes 1012 in a distributed computing system 1002 to process a batch of events for a job / operation defined in an application and generate a set of output events as a result of the processing. In certain embodiments, processing the batch of events can include serialization of data in the batch of events. Figure 12 The process described in FIG. 1 1 1 1 describes a technique by which data in a batch of events can be serialized. Figure 12 The particular series of processing steps depicted in FIG. 1 1 1 1 is not intended to be limiting. Alternative embodiments can have more or fewer steps than those shown in FIG. 1 1 1 1 in various arrangements and combinations. Figure 12

[0241] In certain embodiments, whenever a batch of events is received via the task 1020 (shown in FIG. 1 1 1 1), Figure 10 In certain embodiments, whenever a batch of events is received via the task 1020 (shown in FIG. 1 1 1 1), Figure 12 The process depicted in FIG. 1 1 1 1 can be performed by a node in the cluster of computing nodes 1012 in the distributed computing system 1002. The process begins at 1202, which receives a batch of events from the CQL transform 1 108 at 1204. In certain examples, each event in a batch of events can be referred to as a tuple, and the batch of events can be referred to as a batch of input tuples or a set of input tuples. As described above, each event received via the event stream conforms to a schema associated with the event stream, and the schema identifies one or more attributes of each event received via the event stream.

[0242] For example, a continuous event stream can represent product-related information related to products sold by a company, where each event in the event stream can represent an order for an item. Such a continuous event stream can include attributes such as an order identifier related to the item, an order status, and an order quantity. A schema for such an input stream can be represented as S(time stamp, order identifier, order status, order quantity). <orderid> ， <orderstatus> ， <orderamount>). Thus, each event received via this stream will be identified by a timestamp and three attributes. In certain embodiments, one or more attributes of an event can be represented as one or more columns in a set of input tuples (a batch of events), and thus, in some examples, an attribute can refer to a column storing a data value for a tuple (event) in a set of input tuples.

[0243] At 1206, the process includes identifying an attribute (e.g., a first attribute) of an event in the batch of events. At 1208, the process includes identifying a data type of the attribute. For example, according to the example of the order processing stream described above, the process at 1206 and 1208 can identify that the attribute corresponds to the "orderId” attribute of the event and that the data type of the attribute is a numeric data type. At 1210, the process includes determining whether the data type of the attribute is a numeric data type. If the identified data type of the attribute is a numeric data type, then in certain embodiments, the process proceeds to 1212 to determine a first type of data compression to perform on the data value represented by the attribute. For example, at 1212, the process can include determining to apply a numeric compression technique (e.g., base compression, precision reduction compression, or precision reduction value indexing) to the data value represented by the attribute. At 1214, as a result of applying the numeric compression technique to the data value stored by the attribute, the process includes generating a set of serialized data values for the attribute. In certain embodiments, the process at 1214 includes generating a serialized data value for each data value represented by the attribute. For example, if the attribute represents a set of data values {1, 2, 3, 4, 5}, then the process at 1214 can include generating a serialized data value for each data value in the set {1, 2, 3, 4, 5}. In some examples, the process at 1214 can include generating a serialized data value for each data value in the set {1, 2, 3, 4, 5} by applying a numeric compression technique to each data value in the set {1, 2, 3, 4, 5}. For example, if the numeric compression technique is base compression, then the process at 1214 can include generating a serialized data value for each data value in the set {1, 2, 3, 4, 5} by applying base compression to each data value in the set {1, 2, 3, 4, 5}. In some examples, the process at 1214 can include generating a serialized data value for each data value in the set {1, 2, 3, 4, 5} by applying precision reduction compression to each data value in the set {1, 2, 3, 4, 5}. In some examples, the process at 1214 can include generating a serialized data value for each data value in the set {1, 2, 3, 4, 5} by applying precision reduction value indexing to each data value in the set {1, 2, 3, 4, 5}. Figure 13A 、 Figure 13B 、 Figure 13C and Figure 13D The process of generating a set of serialized data values for a numeric attribute of an event is discussed in 1214. At 1216, the process includes storing the set of serialized data values represented by the attribute.

[0244] In certain embodiments, at 1218, the process includes determining whether there are additional attributes of events that need to be processed. If there are additional attributes to process, then the process loops back to 1206 to identify a next attribute (e.g., a second attribute) of an event in the batch of events and performs the process at 1208-516 for the next attribute.

[0245] In certain embodiments, if the data type of the identified attribute is not determined to be a numerical data type, at 1210, in certain embodiments, the process proceeds to 1220 to determine a second type of data compression to be performed on data values represented by the attribute. For example, continuing the example of the order processing stream described above, the process at 1206 and 1208 can identify that a second attribute of the event corresponds to the "orderStatus" attribute and that the data type of this attribute is a non-numerical data type. In this case, the process continues to 1220 to determine a second type of data compression to be performed on data values stored by the attribute. In embodiments, the second type of data compression can be different from the first type of data compression. For example, the process at 1220 can determine to apply a non-numerical compression technique (e.g., value index compression) to data values stored by the attribute. At 1214, the process includes generating a set of serialized data values represented by the attribute as a result of applying the non-numerical compression technique to data values stored by the attribute. In Figure 14 The process of generating a set of serialized data values for a non-numerical attribute of an event is discussed in the context of FIG. 12. At 1216, the process includes storing the set of serialized data values represented by the attribute.

[0246] In certain embodiments, the process can again continue to 1218 to determine whether there are any additional attributes of the event to be identified and processed. If there are more attributes, the process loops back to 1206 to identify a third attribute of the event in the batch of events. The process at 1208 can then include identifying the data type of the third attribute, and the process at 1210 can include determining a third type of data compression to be performed on data values stored by the third attribute based on the data type of the third attribute. For example, continuing the example of the order processing stream of events described above, a third type of data compression can be performed on data values stored by the "orderAmount" attribute based on the data type of the attribute. In certain examples, when all attributes of the event have been identified and processed, the process ends at 1222.

[0247] Figure 13A is an example flow diagram of a process 1300 in accordance with embodiments of the present disclosure, which describes a set of operations for generating a set of serialized data values for a numerical attribute of an event. In embodiments, the process 1300 describes a set of operations performed by the data compression module 114 of the data compression system 100 of FIG. 1. Figure 12 additional details of the operations performed by the process of 1214. In some examples, the process 1300 begins at 1302 by storing the current buffer offset to the current column number of the column (e.g., the first attribute) whose data values are being processed. At 1304, the process includes obtaining the data type of the attribute (column type). For example, in this example, the data type of the attribute can be determined to be a numeric attribute. At 1306, the process includes scanning a set of input tuples to obtain the minimum value, the maximum value, and the set of unique values represented by the attribute. At 1308, the process includes calculating the number of bits required to store the data values represented by the attribute of the range (maximum value - minimum value). At 1309, the process includes determining whether the number of bits required is greater than half the number of bits of the data type of the attribute and whether the size of the set of unique values is less than the number of input tuples / value_index_threshold, where value_index_threshold is configurable. In examples, the value_index_threshold can be configured to the value 11 as a default value. If so, the process continues to 1350 to perform the precision reduction index value compression technique. At 1310, the process includes determining whether the number of bits required is less than the number of bits of the original data type of the column. The check at 1310 is performed to ensure that the size of the serialized chunk will not increase over the original chunk. This is because if the number of bits required to cover the value range is greater than the number of bits required for the original data, then the resulting chunk created using the value index technique can be larger than the original chunk size.

[0248] If the number of bits required is less than the number of bits of the original data type of the column, then the process continues to 1312 to determine whether the size of the set of unique values is less than the number of input tuples / 2. This determination is performed to ensure that the compression ratio is large enough. If there are too many unique values, then in some examples, the values themselves will be used instead of the values and the index of the values. If it is determined that the number of bits required is less than the original data type of the column and the size of the set of unique values is less than the number of input tuples / 2, then the process described in 1314-1326 is performed.

[0249] For example, at 1314, the process includes storing the first type of data compression to be performed on the data values represented by the attribute as a precision reduction value index type of data compression. Precision reduction techniques reduce the bits representing a value from the used value by looking at the range of the value. The bits needed will depend on the range of the value. At 1316, the process includes storing the minimum data value of the attribute. At 1318, the process includes storing the number of bits of each minimum value. At 1320, the process includes performing the operations at 1322 and 1324 for each data value of the column (e.g., attribute) whose data values are currently being processed. For example, at 1322, the process includes obtaining an index from the set of unique data values. At 1324, the process includes storing the index to a buffer. After all of the data values of the column have been processed, at 1326, the process includes storing (unique value - minimum value) for each unique value in the set of tuples. These values represent the actual values indexed from the index stored at step 1324.

[0250] In certain embodiments, if the process at 1312 determines that the size of the set of unique values is not less than the number of input tuples / 2, then in some embodiments, the processes 1332-1338 described in Figure 13B In certain embodiments, if the process at 1310 determines that the number of bits needed is not less than the original data type of the column, then in one embodiment, the processes in 1342-1346 described in Figure 13C In certain embodiments, the process ends at 1328 by returning the set of serialized data values (i.e., serialized data block) for the attribute to the CQL engine to process the set of tuples received via the event stream (i.e., the batch of events).

[0251] Figure 13B is an example flowchart of a process 1350 according to embodiments of the present disclosure, which describes a set of operations for generating a set of serialized data values for a numeric attribute of an event using precision reduction compression techniques. In embodiments, the process 1350 describes a set of operations performed by the Figure 13A additional details of the operations performed by the process in 1330. In certain examples, the process 1350 begins at 1332 by storing the type of data compression to be performed on the data values represented by the attribute as precision reduction compression. At 1334, the process includes storing the minimum data value of the attribute. At 1336, the process includes storing the number of bits for each data value of the attribute. At 1338, the process includes, for each data value of the column, performing a bit copy (value - minimum value) on only the required bits. For example, a set of input values (10, 11, 12) would be stored with bit 00 for value 0, which is the result of (10 - 10 (minimum value)), 01 for value 1, which is the result of (11 - 10), and 02 for value 2, which is the result of (12 - 10). The sequence of bit values 00, 01, and 02 can be stored into a byte (8 bits) 00010200 and stored as the hexadecimal value 154.

[0252] Figure 13C is an example flowchart of a process 1360 according to embodiments of the disclosure that describes a set of operations for generating a serialized set of data values for a numeric attribute of an event using a regular compression technique. In embodiments, the process 1360 describes operations performed by the process in 1340. Figure 13A additional details of the operations performed by the process in 1330. In certain examples, the process 1350 begins at 1332 by storing the type of data compression to be performed on the data values represented by the attribute as precision reduction compression. At 1334, the process includes storing the minimum data value of the attribute. At 1336, the process includes storing the number of bits for each data value of the attribute. At 1338, the process includes, for each data value of the column, performing a bit copy (value - minimum value) on only the required bits. For example, a set of input values (10, 11, 12) would be stored with bit 00 for value 0, which is the result of (10 - 10 (minimum value)), 01 for value 1, which is the result of (11 - 10), and 02 for value 2, which is the result of (12 - 10). The sequence of bit values 00, 01, and 02 can be stored into a byte (8 bits) 00010200 and stored as the hexadecimal value 154.

[0253] Figure 13D is an example flowchart of a process 1370 according to embodiments of the disclosure that describes a set of operations for generating a serialized set of data values for a numeric attribute of an event using a precision reduction value index compression technique. In embodiments, the process 1370 describes operations performed by the process in 1330. Figure 13A additional details of the operations performed by the process 1370 of the 1350. In some examples, the process 1370 begins at 1372 by storing the type of data compression to be performed on the data values represented by the attribute as precision reduction index value compression. At 1374, a set of difference values (e.g., value - minimum value) is computed. At 1376, the process includes scanning all of the values in the set of difference values to obtain a set of enumerated values. At 1378, the process includes computing a set of indices for each data value represented by the set of difference values. At 1380, the process includes computing a minimum value and a maximum value from the set of indices. At 1382, the process includes storing the minimum data value for the attribute. At 1384, the process includes storing the number of bits for each data value of the index value. At 1386, the process includes performing bit copying for only the required bits for each data value of the column. At 1388, the process includes storing the set of enumerated difference values.

[0254] Figure 14 is an example flow diagram of a process 1400 in accordance with embodiments of the present disclosure that describes a set of operations for generating a serialized set of data values for a non-numeric attribute of an event. In embodiments, the process 1400 describes operations performed by the process 1214 of the 1200 when it is determined that a second type of data compression is to be performed on the data values represented by the attribute (e.g., non-numeric attribute). Figure 12 additional details of the operations performed by the process 1214 of the 1200. The process 1400 begins at 1402 by storing the current column offset to the current column number of the column (e.g., attribute) whose data values are being processed. At 1404, the data type of the attribute (column type) is obtained. For example, in this example, the data type of the attribute is determined to be a non-numeric attribute. At 1406, the process includes storing the type of data compression to be performed on the data values represented by the attribute as value index compression. In this case, all possible values within the input batch are enumerated and a position index is used instead of duplicating values multiple times.

[0255] At 1408, the process includes scanning all of the input tuples to obtain a set of enumerated values for the column. At 1410, the process includes computing a set of indices for each data value represented by the list. At 1412, the process includes performing the operations described below at 1414-1416 for each data value stored in the column. At 1414, the process includes obtaining an index from the set of enumerated values. At 1416, the process includes storing the index to a buffer. At 1418, the process includes storing the set of enumerated values. At 1420, the process ends by returning the serialized set of data values (i.e., serialized data block) for the attribute to the CQL engine to process a set of tuples (i.e., a batch of events) received via the event stream.

[0256] Figure 15 is an example of a way in which event stream data can be serialized based on determining a data type of attributes of events in the event stream according to embodiments of the present disclosure. In the example illustrated below, the event stream represents product-related information related to products sold by a company. Each event in the event stream can represent an order for an item and include attributes related to the item such as an order identifier, an order status, and an order quantity. A schema for such an order event stream can be represented as S(time stamp, <orderid> ， <orderstatus> ， <orderamount>). Thus, each event received via this stream can be identified by a timestamp and three attributes. As an example, a batch of events received via the order event stream can include the following events and associated timestamps:

[0257] …

[0258] (timestamp_N, 10, "open", 100)

[0259] (timestamp_N+1, 11, "open", 5000)

[0260] (timestamp_N+2, 10, "processing", 100)

[0261] (timestamp_N+3, 10, "shipped", 100)

[0262] (timestamp_N+4, 11, "processing", 5000)

[0263] (timestamp_N+5, 10, "closed", 100)

[0264] (timestamp_N+6, 11, "shipped", 5000)

[0265] (timestamp_N+7, 11, "closed", 5000)

[0266] As noted above, in certain embodiments, one or more attributes of an event can be represented as one or more columns of a set of input tuples representing a batch of events. Thus, in some examples, an attribute can refer to a column storing a data value of a tuple (event) in a set of input tuples. An example of a set of input tuples corresponding to a batch of events received via the order event stream can be as shown in Table-1 below:

[0267]

[0268]

[0269] Table-1

[0270] In certain embodiments, the events in the batch of events are serialized by identifying a data type of each attribute of the event and determining a particular type of compression technique to be applied to the data value represented by each attribute based on the data type of the attribute. For example, a first compression technique can be applied to a first attribute of the event (e.g., an order id attribute) based on a determination that the first attribute is a numeric attribute, a second compression technique can be applied to a second attribute of the event (e.g., an order status attribute) based on a determination that the second attribute is a non-numeric attribute, and a third compression technique can be applied to a third attribute of the event (e.g., an order quantity attribute) based on a determination that the third attribute is a numeric attribute. In certain examples, the first type of compression technique, the second type of compression technique, and the third type of compression technique can be different from one another.

[0271] In embodiments, columnar storage can be used to store attributes (columns) having the same data type (such that the values in the column are the same data type). In certain embodiments, a base value compression technique or a precision reduction compression technique can be used to compress the values stored in a column of numeric type. Precision reduction reduces the bits used to represent a value from the values used by looking at the range of the values. The bits needed will depend on the range of the values. Base value compression uses the minimum value as a base value and stores the difference from the base value for other values. For example, an input set of values representing "order ID" for each event in a batch of events (10, 11, 10, 10, 11, 10, 11, 11) can be compressed as (10, 1), 01001011 (binary), or 0x4B (hexadecimal), which represents the values (0, 1, 0, 0, 1, 0, 1, 1), thereby using a reduction from 32 bits to 2 bits because the minimum value is 10 and the range is 2. For another example, a precision reduction and value indexing technique can be used to compress an input set of values representing "order quantity." In this case, an input set of values representing order quantity (100, 5000, 100, 100, 5000, 100, 5000, 5000) can be compressed as (100, 2, 0x10, 0x4F) and (0, 4900) by using a precision reduction and value indexing technique. The input set can be represented as (0, 4900, 0, 0, 4900, 0, 4900, 4900), with a base value of 100. The resulting set of values is binary 00010000, 01001111, hexadecimal 10 and 4F, which represents (0, 1, 0, 0, 1, 0, 1, 1) with indices to the base value table (0, 4900) (e.g., 0 points to 0 and in turn to 100 because the base value is 100, and 1 points to 4900 and in turn to 5000 because the base value is 100).

[0272] In certain embodiments, the "value index compression" technique can be used to handle values of a column storing non-numeric values such as string values. In this case, we enumerate all possible values in a batch of inputs and use the index of the position instead of copying the value multiple times. For example, if the values of an "order status" attribute (column) are (open, open, processing, shipped, processing, closed, shipped, closed), then the corresponding enumerated unique values will be (open, processing, shipped, closed). When the values of the column are stored in a linear buffer in order, the index of each value will be (0, 5, 17, 25) as the buffer will have open / 0 processing / 0 shipped / 0 closed / 0, where / 0 indicates the end of the string token. In the case of the linear buffer of values, the final compressed result is (0, 0, 5, 17, 5, 25, 17, 25).

[0273] Figure 16 is an example flow diagram of a process 1600 according to embodiments of the present disclosure, which describes a set of operations by which data included in a batch of events can be de-serialized. In certain embodiments, the process 1600 can be performed by a batch de-serializer process (420) in a distributed event processing system. As described above, in certain embodiments, the batch de-serializer process can be invoked by a master compute node (e.g., 314 as shown in FIG. 3) when the master compute node runs a job / operation defined in an application. As described above, the master compute node identifies nodes (e.g., executor nodes 318) in a cluster of compute nodes 312 in the distributed computing system 302 to process a batch of events for a job / operation defined in an application and generate a set of output events as a result of the processing. In certain embodiments, processing the batch of events can include serialization and subsequent de-serialization of data in the batch of events. Figure 3 The process of FIG. 3 describes a technique by which data in a batch of events can be de-serialized. Figure 16 The particular series of processing steps depicted in FIG. 3 is not intended to be limiting. Alternative embodiments can have more or fewer steps than those shown in FIG. 3. Figure 16 The particular series of processing steps depicted in FIG. 3 is not intended to be limiting. Alternative embodiments can have more or fewer steps than those shown in FIG. 3. Figure 16 The particular series of processing steps depicted in FIG. 3 is not intended to be limiting. Alternative embodiments can have more or fewer steps than those shown in FIG. 3.

[0274] In certain embodiments, the process 1600 begins at 1602 by receiving a set of serialized data values corresponding to one or more attributes of events in a batch of events (a set of input tuples). At 1604, the process includes processing the set of serialized data values corresponding to one or more attributes of events in the batch to generate a set of output events. In certain examples, the process at 1604 can include generating a set of deserialized data values corresponding to the attribute based on the set of serialized data values at 1606, and processing the set of deserialized data values corresponding to the attribute for a set of one or more continuous queries to generate a first set of output events at 1608. At 1610, the process includes sending the set of output events to a user of the distributed event processing system.

[0275] With respect to the following Figures 17-18 The process 1604 of generating a set of deserialized data values corresponding to an attribute of events in a batch of events is discussed in detail. In particular, Figure 17 A process by which a set of deserialized data values corresponding to one or more attributes of events in a batch of events can be generated is described. Figure 18 A process by which a set of deserialized data values corresponding to a non-numeric attribute of an event can be generated is described. Figures 19-21 A process by which a set of deserialized data values corresponding to a numeric attribute of an event can be generated is described.

[0276] Figure 17 is an example flowchart of a process 1700 according to embodiments of the present disclosure, described as a set of operations to generate a set of deserialized data values for one or more attributes of events in a batch of events. In embodiments, the process 1700 describes additional details of the operations of the process in 1604 of Figure 16 In certain examples, the process 1700 begins at 1702 by creating an array of tuples. At 1704, the process includes identifying a first column (first attribute) of events. At 1706, the process includes obtaining a buffer offset for a current column number of the column (e.g., first attribute) for which data values are being processed. At 1708, the process includes reading a compression type for the attribute. For example, this involves reading the data compression type that was performed by the batch serializer process to serialize the data values for the attribute. At 1710, the process includes determining whether the type of data compression applied to the attribute is value index compression. If the compression type applied to the attribute is value index compression, then the process continues to 1724 to perform the process described in Figure 18

[0277] ​At 1712, the process includes determining whether the type of data compression applied to the attribute is precision reduction compression. If the type of compression applied to the attribute is precision reduction compression, then the process continues to 1726 to perform the process described in Figure 19 .

[0278] In certain embodiments, at 1714, the process includes determining whether the type of data compression applied to the attribute is precision reduction value index compression. If the type of compression applied to the attribute is precision reduction value index compression, then the process continues to 1728 to perform the process described in Figure 20 .

[0279] In certain embodiments, if the process determines that the type of compression applied to the attribute is neither value index compression nor precision reduction compression or precision reduction value index compression, then the process continues to 1716 to determine that the type of compression applied to the attribute is general compression. At 1716, the process includes performing the process described in Figure 21 .

[0280] At 1718, the process includes determining whether there are additional attributes to process. If there are additional attributes to process, then the process loops back to 1704 to identify and process the next attribute of the event. If there are no attributes to process, then in some embodiments, the process returns the array of tuples to the CQL engine for further processing at 1720. In certain embodiments, the process ends at 1722.

[0281] Figure 18 is an example flow diagram of a process 1800 in accordance with an embodiment of the present disclosure that describes a set of operations for generating a set of deserialized data values corresponding to a numeric attribute or a non-numeric attribute of events in a batch of events using value index compression. In embodiments, the process 1810 describes additional details of the process 1724 in Figure 17 , and is performed when the type of data compression applied to an attribute (e.g., at 1710 in Figure 17 ) is determined to be "value index” compression. As an example, the process 1800 can be performed for the "order status” attribute (a non-numeric attribute) in an order event stream.

[0282] An example of using value index compression techniques is described in Figure 8 The compressed values have two data sets (0, 0, 5, 17, 5, 25, 17, 25) and ('Open', 'Process', 'Ship', 'Close'). The second set is called value_arrays because this set contains the actual values. The first set is called index_values because this set includes the index values to the actual values stored in value_arrays. value_index refers to each of the individual index values in index_values.

[0283] In certain embodiments, the process 1800 begins at 1802 by reading the index values to index_values. At 1804, the process includes reading the value arrays to value_array. At 1806, the process includes performing operations in 1808, 1810, and 1812 for each data value corresponding to an attribute in a set of input tuples. For example, at 1808, the process includes obtaining an index from index_values[value_index]. At 1810, the process includes obtaining a value from value_array[index]. At 1812, the process includes setting the value to tuple column tuple[value_index].

[0284] Figure 19 is an example flow diagram of a process 1900 according to embodiments of the disclosure that describes a set of operations for generating a deserialized set of data values corresponding to a numeric attribute of events in a batch of events using a precision reduction compression technique. In embodiments, the process 1900 describes Figure 17 additional details of the process 1726 in Figure 17 The process 1900 is executed when the type of data compression applied to an attribute (e.g., at 1712 in Figure 8 is determined to be "precision reduction" compression. As an example, the process 1900 can be executed for the "Order ID" or "Order Volume" attributes (numeric attributes) in an order event stream, and the compressed result is shown in column 1 as (10, 2, 0x4B). In a set of operations described below, in one example, the term "min value" has a value of 17, which is the minimum value of the range of values, and the term "number of bits" has a value of 2, which is the number of bits representing the range of values.

[0285] In certain embodiments, the process 1900 begins at 1902 by reading the minimum value to base_value. At 1904, the process includes reading the number of bits. At 1906, the process includes performing the process in 1908 and 1910 for each data value corresponding to an attribute in a set of input tuples. For example, at 1908, the process includes reading the value bits to value_bits. At 1910, the process includes setting base_value + value_bits to tuple column tuple[value_index].

[0286] Figure 20 is an example flow diagram of a process 2000 according to embodiments of the present disclosure that describes a set of operations for generating a deserialized set of data values corresponding to a numeric attribute of events in a batch of events using precision reduced value indices. In embodiments, the process 2000 describes additional details of the process 1728 in Figure 17 and is performed when the type of data compression applied to the attribute (e.g., at 1712 in Figure 17 is determined to be "precision reduced value indices” compression. As an example, the process 2000 can be performed for the "order ID” or "order amount” attribute (numeric attribute) in an order event stream. Figure 8 An example result for the "order amount attribute” is shown in column 3, which results in (100, 2, 0x10, 0x4F) and (0, 4900).

[0287] In the set of operations described below, the term "base value” refers to the value 170 for the base value of the column value, the term "number of bits” refers to the value 2 for the number of bits of the index value, and the term "index_values” refers to the values (0x10, 0x4F) used as the index value, and the term "value_array” refers to (0, 4900) representing the set of difference values.

[0288] In certain embodiments, the process 2000 begins at 2002 by reading a minimum value to base_value. At 2004, the process includes reading a number of bits. At 2006, the process includes reading index values to index_values. At 2008, the process includes reading a value array to value_array. At 2010, the process includes setting value_array[j] to value_array[j] + base_value for each value of j <- 0 to value_array.length. At 2012, the process includes performing the processes in 2014, 2016, and 2018 for each data value corresponding to an attribute in a set of input tuples. For example, at 2014, the process includes obtaining an index from index_values[value_index]. At 2016, the process includes obtaining a value from value_array[index]. At 2018, the process includes setting the value to tuple[value_index].

[0289] Figure 21 is an example flow diagram of a process 2100 in accordance with embodiments of the present disclosure, which describes a set of operations for generating a collection of deserialized data values corresponding to a numerical attribute or a non-numerical attribute of events in a batch of events. In embodiments, the process 2100 describes additional details of the process 1716 in Figure 17 , and is performed when the type of data compression applied to the attribute (e.g., at 1716 in Figure 17 ) is determined to be a "general compression” technique. As an example, the process 2100 can be performed for an "order ID” or "order amount” attribute (numerical attribute) in an order event stream. In a set of operations described below, the term "value” refers to an uncompressed value.

[0290] In certain embodiments, the process 2100 begins at 2102 by decompressing a chunk to a value array. At 2104, the process includes setting a value in the value array to a tuple column of a tuple corresponding to an attribute in a set of input tuples for each data value.

[0291] Scheduling and managing multiple CEP engines in a micro-batch based event processing system

[0292] In recent years, data stream management systems (DSMs) have been developed that can execute queries over potentially unbounded real-time data streams in a continuous fashion. Among the new DSMs, these systems employ micro-batch based stream processing in order to provide a combination of batch and stream processing from a single framework. An example of such a system is Apache Flink® running on the Apache® Streaming applications.

[0293] Micro-batch stream processing has some drawbacks due to the nature of its generally complex system design involving stateful processing. One such drawback is the inability to perform "type matching" operations. Type matching is an important desired feature that stream processing systems should support, and it requires highly stateful processing to run state machines to detect patterns from unbounded streams of events.

[0294] To support fully stateful query processing, the disclosed techniques add a CQL query engine to micro-batch stream processing. Since there is more than one CQL engine in the cluster, issues related to scheduling, tracing, and maintaining locality must be addressed.

[0295] Figure 22 This is an example system or architecture in which the scheduling process in a CQL engine tracer can be implemented. In one embodiment, and as follows: Figure 22 As shown, a CQL engine tracker 2202 component is disclosed in the driver (main) 2206, which can communicate remotely between the CQL engine 2212 and the CQL Resilient Distributed Dataset (RDD) 2218. For startup and scheduling, the CQL engine tracker 2202 uses a two-step scheduling strategy to differentiate between different system environments. To maximize locality, in one embodiment, the CQL engine tracker 2202 uses the following affinity algorithm.

[0296] 1. All CQL engines 2212, 2214, and 2216 are started from driver 2206 by CQL engine tracker 2202. No association between CQL engines and preferred locations is set.

[0297] 2. The first CQLRDD 2218 does not have preferred location information.

[0298] 3. Scheduler 2204 will attempt to use the preferred location of the parent RDD to locate the host where the parent RDD resides.

[0299] 4. The first run of CQLRDD 2218 associates CQL engine 2212 with the same host 2208.

[0300] 5. The next CQLRDD 2220 will set the preferred location information based on the association information set in step 4.

[0301] 6. Scheduler 2204 will attempt to run CQLRDD to the preferred location where it is set.

[0302] The disclosed technology enables fully stateful CQL engines 2212, 2214, 2216 in micro-batch stream processing, maintaining locality between CQL engines and CQL RDDs, and a multi-step scheduling algorithm for starting and restarting CQL engines. In addition, the disclosed local affinity algorithm provides maximum performance compared to other event-based stream processing systems.

[0303] In certain embodiments, the disclosed CQL engine tracker 2202 is responsible for scheduling, tracking, and restarting CQL engines in a cluster. CQL engines run as long-running tasks in a cluster and can be started as regular stream jobs. The CQL engine tracker 2202 does not return except in the case of a failure scenario.

[0304] In some embodiments, the following tracking information for the CQL engine tracker 2202 can be maintained.

[0305] • state: CQLEngineState - INACTIVE, SCHEDULED, ACTIVE

[0306] - This changes throughout the life cycle of the CQL engine 2212, 2214, 2216 from INACTIVE -> SCHEDULED -> ACTIVE -> INACTIVE

[0307] • scheduleLocation: TaskLocation

[0308] - Initial scheduled location

[0309] • runningExecutor: ExecutorCacheTaskLocation

[0310] - The actual running executor 2208 location of the CQL engine

[0311] • name: String

[0312] - The name of the CQL engine 2212, 2214, 2216

[0313] • endpoint: RpcEndpointRef

[0314] - The remote procedure call (RPC) of the CQL engine 2212, 2214, 2216

[0315] endpoint to access it remotely

[0316] • errorInfo: CQLEngineErrorInfo

[0317] -Last known error information

[0318] In embodiments, the startup procedure for CQL engines 2212, 2214, 2216 can be described as follows:

[0319] • Decide on the number of CQL engines 2212, 2214, 2216 to start

[0320] • Get the list of executors 2208, 2210, 2212

[0321] • Run the round robin scheduler to schedule CQL engines 2212, 2214, 2216 to the list of executors 2208, 2210, 2212

[0322] • The task scheduler 2204 starts the actual long running task

[0323] • The newly started CQL engine calls the "register" RPC call to the CQL engine tracker (e.g., CQL engine tracker 2202)

[0324] In certain embodiments, the CQL engine locality affinity algorithm can be described by the following procedure:

[0325] 1. All CQL engines are started by the CQL engine tracker from the driver. No association of CQL engines to preferred locations is set.

[0326] 2. The first CQLRDD has no preferred location information.

[0327] 3. The scheduler will try to co-locate to the host where the parent RDD is located using the preferred location of the parent RDD.

[0328] 4. The first run of the CQLRDD associates the CQL engine to the same host.

[0329] 5. The next CQLRDD will set the preferred location information based on the association information set from step 4.

[0330] 6. The scheduler will try to run the CQLRDD to the preferred location it set.

[0331] In embodiments, the CQL engine restart scheduling procedure can be described as follows:

[0332] • Handle two cases (reject, crash)

[0333] • Reject - if the scheduled location is different from the actual location (failed to start according to the schedule)

[0334] • Obtain scheduled executors using old scheduled executors (minus inactive executors) or using new scheduled executors with schedulePolicy.rescheduleCQLEngine

[0335] - Select executors that are still alive in the list of scheduled locations

[0336] • Start CQL engine with scheduled executors

[0337] The following flow shows the data flow for the above architecture:

[0338] 1. CQL engine tracker 2202 in the driver 2206 starts a long running task for each CQL engine. The CQL engine tracker 2202 exposes its RPC endpoint to the long running task.

[0339] 2. The task scheduler 2204 executes the long running task to executors 2208, 2210, 2212 in the cluster

[0340] 3. As part of the long running task, the CQL engine runs from the executors 2208, 2210, 2212.

[0341] 4. The CQL engine registers itself with the CQL engine tracker 2202 in the driver 2206 with the CQL engine's CQL endpoint.

[0342] 5. As part of the streaming DAG, there will be CQL RDDs that are responsible for CEP processing. By consulting the CQL engine tracker, the CQL RDDs are processed by either the local CQL engine or a remote CQL engine. Remote CQL engines are invoked through RPC calls

[0343] Starting a CQL engine

[0344] CQL engine runs as a long running task in the cluster. The CQL engine is started by the CQL engine tracker as a regular job, but it never returns except for failure or crash and stays running. The CQL engine tracker starts CQL engines in the cluster according to the following algorithm:

[0345] 1. Decide the number of CQL engines to start

[0346] 2. Obtain the list of executors

[0347] 3. Run a round-robin scheduler to schedule CQL engines to the list of executors

[0348] 4. The task tracker starts the actual long running task

[0349] 5. The newly started CQL engine calls the "register" RPC call to the CQL engine tracker

[0350] 6. The long running task returns control only if the CQL engine crashes or other failure.

[0351] At step #1, the number of CQL engines in the cluster to be started is decided. The default number of CQL engines in the cluster is same as the number of executors in the cluster. Thus, one CQL engine runs from each executor. This maximum number of CQL engines can be configured.

[0352] At step #2, the list of executor information (executor host and executor id) is retrieved from the cluster.

[0353] At step #3, the scheduler is polled to assign the executors to the CQL engines.

[0354] At step #4, the long running task is started for each CQL engine. The task scheduler uses the scheduled executor information (executor host and id) to start the CQL engine in the scheduled executor.

[0355] At step #5, the newly started CQL engine calls the "register" RPC call to the CQL engine tracker. This step triggers the tracking process from the CQL engine tracker as shown below.

[0356] At step #6, the failure or crash of the CQL engine triggers the recovery process from the CQL engine tracker as shown below.

[0357] Tracking a CQL engine

[0358] In some embodiments, the following tracking information can be maintained by the CQL engine tracker for each CQL engine

[0359] state: CQLEngineState

[0360] scheduleLocation: TaskLocation

[0361] runningExecutor: ExecutorCacheTaskLocation

[0362] name: String

[0363] endpoint: RpcEndpointRef

[0364] errorInfo : CQLEngineErrorInfo

[0365] 'state' holds the state of the CQL engine. It changes from INACTIVE -> SCHEDULED -> ACTIVE -> INACTIVE throughout the life cycle of the CQL engine. INACTIVE is the initial state before the CQL engine is tracked by the CQL engine tracker. SCHEDULED is the state when the CQL engine is scheduled to be executed in an executor. ACTIVE is the state when the CQL engine is actually running from the executor runtime.

[0366] 'scheduledLocation' holds the location where the CQL engine is scheduled to be executed.

[0367] 'runningExecutor' holds the executor location where the CQL engine is actually running.

[0368] 'name' is the name of the CQL engine as an identifier.

[0369] 'endpoint' is the RPC Endpoint to communicate with.

[0370] 'errorInfo' is the last known error information of the CQL engine.

[0371] Restoration of a CQL engine

[0372] Long running tasks only return control to the CQL engine tracker when the CQL engine crashes or other failures. The CQL engine tracker uses the following CQL engine restart scheduling procedure in order to restart the CQL engine. The restart scheduling procedure is invoked from two cases: Rejected and Crashed.

[0373] Crashed is the case when the running CQL engine crashes or there is any failure when the long running task returns. Rejected is the case when the scheduled location is different from the actual location (e.g., unable to start from the scheduled executor and started through the task scheduler from a different executor). This can happen due to resource issues from the cluster.

[0374] In an embodiment, the CQL engine restart scheduling procedure can be described as follows:

[0375] 1. Obtain a list of candidate executors using the old scheduled executors (minus the inactive executors) in the cluster and the new executors

[0376] 2. Select an executor that is still alive from the list of candidate executors

[0377] 3. The long running task is started with the scheduled executor, which starts the CQL engine

[0378] Locality affinity algorithm

[0379] To support horizontal scalability, input datasets are partitioned and processed using parallelized distributed data processing. The CQL engine can use (queryld, partitionld) to affinity or association of CQL engines to process multiple partitions. To optimize the transformation with minimal network traffic for sending data between executors, it is required to create such affinity with maximized locality. To maximize locality, in one embodiment, the CQL engine tracker uses the following affinity algorithm.

[0380] 1. All CQL engines are started by the CQL engine tracker from the driver. No association of CQL engines to preferred locations is set.

[0381] 2. The first CQLRDD has no preferred location information.

[0382] 3. The scheduler will try to co-locate to the executor where the parent RDD is located using the preferred location of the parent RDD.

[0383] 4. The CQLRDD calls the "getCQLEngine" RPC to the CQL engine tracker.

[0384] 5. The first computation of the partition of the CQLRDD associates (partitionld, queryld) to the CQL engine to the same executor of the CQLRDD.

[0385] 6. The (partitionld, queryld) to CQL engine preferred location mapping is maintained in the CQL engine tracker.

[0386] 7. The CQL engine from the association returns to the CQLRDD, and the RDD is processed by the CQL engine.

[0387] 8. The next CQLRDD will set the preferred location information according to the association information set from step 5.

[0388] 9. The scheduler will try to run the CQLRDD to the preferred location it is set to.

[0389] 10. The CQLRDD calls the "getCQLEngine" RPC to the CQL engine tracker,

[0390] And (partitionld, queryld) should already be the same executor.

[0391] Figure 23 is an example flow diagram of a process 2300 in accordance with embodiments of the present disclosure, which describes a set of operations for scheduling and managing a plurality of CEP engines within a micro-batch based event processing system. In embodiments, the process 2300 describes Figure 22 additional details of the operations described in the Summary. In certain examples, the process 2300 begins at 2302 by starting a first CQL engine in a cluster of CQL engines. The CQL engine tracking engine can be used to track the first CQL engine, as well as additional CQL engines. At 2304, the CQL engine tracking engine can also schedule the first CQL engine to process a batch of continuous input event streams related to an application. At 2306, the CQL engine tracking engine can also track the first CQL engine to be scheduled for execution. At 2308, the CQL engine tracking engine can also execute the first CQL engine to process the batch of continuous input event streams to generate a set of output events related to the application.

[0392] Automatic data partitioning and parallelism using Group By and Object ID fields

[0393] In recent years, data stream management systems (DSMs) have been developed that can execute queries in a continuous manner over potentially unbounded real-time data streams. For example, a typical DSMS can receive one or more data streams, register a query against the data streams, and continuously execute the query as new data appears in the streams. Because this type of continuous query is long-running, the DSMS can provide a continuous stream of update results to a client.

[0394] A typical application in a DSMS is designed as a directed acyclic graph (DAG) shaped "topology" of operations or transformations. The topology acts as a data transformation pipeline.

[0395] Most stream processing systems, including Apache Storm, Spark Streaming, and Flink, provide application developers with an application programming interface (API) to build topologies using different programming languages such as Java, Scala, or Clojure.

[0396] The API facilitates programmers to build stream processing applications, but it is relatively complex for code generation systems such as Stream Analytics that generate stream processing applications for users due to the complexity of the code generation layer.

[0397] Figure 24 is an example architecture 2400 in which the input pipeline of a data transformation can be input into a pipeline analyzer 2402 and classified by a stage classification module 2404. In some examples, the code generation layer of Stream Analytics is responsible for automatically determining parallelism in a data transformation pipeline by analyzing the pipeline stages. A data transformation pipeline is composed of individual stages, where each stage performs a specific transformation according to the stage definition. An aggregator stage computes real-time aggregations on incoming stream data. If the stages of a pipeline can be processed on a cluster of nodes, then the data transformation pipeline processing can be optimized.

[0398] To compute stages on a cluster of nodes, it is desirable to automatically determine the parallelism characteristics of the stage operations and then create a DAG of the transformation where the computation of the transformation can be done on a collection of cluster nodes by maximizing parallelism.

[0399] In one embodiment, a data stream management system (DSMS) is constructed that analyzes a data transformation pipeline designed by a user, derives partitioning criteria for individual stages, and generates an optimized DAG of the transformation where each stage can run on a collection of cluster nodes.

[0400] In certain embodiments, the following stages can be included in a pipeline designed by the Stream Analytics platform.

[0401] 1) Query

[0402] 2) Business Rule

[0403] 3) Spatial

[0404] 4) Pattern

[0405] A pipeline can be composed of one or more stages of the above types.

[0406] An example of a sample pipeline is shown below:

[0407] Input -> Query -> Query -> Spatial -> Output

[0408] In some examples, a user creates a pipeline to implement a desired business logic. When designing a pipeline, a user can select each stage of the pipeline and configure the stage properties of the pipeline. In some examples, the configuration properties of a stage become stage metadata.

[0409] If the stage type is Query, then the disclosed techniques determine the automatic data partitioning of the stage. To transform a user-created pipeline into a DAG of native runtime transformations, the Stream Analytics platform can perform the following steps:

[0410] 1) Traverse the pipeline from source to sink.

[0411] 2) For each stage

[0412] i) Determine the stage type

[0413] ii) If the stage type is "query", then mark whether the stage can compute the transform for this stage in a distributed manner

[0414] a) Determine the CQL query associated with the query stage.

[0415] b) Parse the CQL query into tokens.

[0416] c) Perform semantic analysis of the parsed query for CQL query.

[0417] d) Determine the query classification using various rules

[0418] These rules classify continuous queries into the following categories:

[0419] Stateless, Semi-stateful, Fully stateful

[0420] e) If the query is stateless, then mark the stage as to be partitioned without any partition attribute (standard) 2406. In this way, the stage will depend on the partitioning standard of the parent stage.

[0421] f) If the query is stateful, then mark the stage as unpartitioned 2408. In this way, the stage will only execute on a single cluster of nodes.

[0422] g) If the query is semi-stateful, then mark the stage as to be partitioned with a partition attribute 2410. The partition attribute will be obtained from the result of step 2.ii.d. In this way, the stage computation can be partitioned on the automatically determined partition attribute.

[0423] 3) For each stage, generate the transform in the DAG of the data transform pipeline.

[0424] i) If the stage is marked as to be partitioned without any partition attribute 2406, then generate the transform for this stage in the DAG without any repartitioning transform. The number of partitions for the stage will be determined by the number of partitions from the previous stage.

[0425] ii) If the stage is marked as to be partitioned with a partition attribute 2410, then generate the transform for this stage in the DAG with a repartitioning transform. The inputs to the repartitioning transform will be the partition attribute and the number of partitions. The repartitioning transform will repartition the incoming event stream with the new partitioning standard.

[0426] iii) If the stage is marked as unpartitioned 2408, then generate a transform for the stage in the DAG with a repartitioning transform followed by a stage transform. The input to the repartitioning transform will be the partition attribute and the number of partitions which will be 1. The repartitioning transform will re-partition the partitioned / unpartitioned stream into a single partition.

[0427] In some pipelines, if the system does not have enough metadata or if the partitioning cannot be determined from query analysis, then the system marks the object id as the partition attribute 2412.

[0428] If the stage type is spatial and the DSMS is processing a stream of geo-location events of mobile objects where each object has a unique identity, then the Stream Analytics platform marks the object id as the partition attribute for the stage and generates a transform for this stage in the DAG with a partitioning transform followed by a stage transform.

[0429] Embodiments of the disclosed technology provide the following features:

[0430] - Perform metadata scan of the pipeline stages and classify the stages based on CQL query classification.

[0431] - Automatically determine the partition attribute by performing semantic analysis of the query based on the Continuous Query Language.

[0432] - Generate a DAG of transforms by using partitioning.

[0433] Prior art involves explicitly defining the parallelism characteristics of the pipeline stages by the user using the pipeline data transformation system. If not specified, then the system can process the pipeline stages without fully utilizing the computing resources.

[0434] The disclosed technology automatically determines the data partitioning criteria by analyzing the stages of the pipeline stream processing system. This significantly reduces the complexity of designing data processing pipelines for the Stream Analytics platform.

[0435] Figure 25 is an example flow diagram of a process 2500 that describes a set of operations for data partitioning and parallelism in a distributed event processing system. In embodiments, the process 2500 describes Figure 24 Additional details of the operations described in the middle. In certain examples, the process 2500 begins at 2502 by determining a stage of a continuous query language (CQL) query processed by an event processing system. At 2504, the system can be configured to determine a stage type associated with the stage. In some examples, the process 2500 can continue at 2506 by determining a transformation to be computed for the stage based at least in part on the stage type. The process 2500 can also determine a classification of the CQL query at 2508 based at least in part on a plurality of rules. In some examples, the process 2500 can include labeling the stage as a partitioned stage or an unpartitioned stage at 2510 by applying partitioning criteria to the stage. Further, in some examples, the process 2500 can generate a transformation in a directed acyclic graph (DAG) of a data transformation pipeline for the stage at 2512 based at least in part on the partitioning criteria for the stage. At 2514, the process 2500 can determine a partition for the stage based at least in part on the transformation. The process 2500 can also process the CQL query based at least in part on the determined partition.

[0436] Illustrative system

[0437] Figure 28 FIG. 13 illustrates an example computer system 2800 that can be used to implement embodiments of the disclosure. In some embodiments, the computer system 2800 can be used to implement any of the various servers and computer systems described above. As shown, the computer system 2800 includes various subsystems including a processing unit 2804 in communication with a number of peripheral subsystems via a bus subsystem 2802. These peripheral subsystems can include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. The storage subsystem 2818 can include tangible computer-readable storage media 2822 and a system memory 2810. Figure 28

[0438] In various embodiments, the server 2612 can be adapted to execute one or more services or software applications, such as services and applications that provide event processing services. In certain embodiments, the server 2612 can also provide other services, or the software applications can include non-virtual and virtual environments. In some embodiments, these services can be provided as web-based or cloud services or under a software as a service (SaaS) model to users of the client computing devices 2602, 2604, 2606, and / or 2608. Users operating the client computing devices 2602, 2604, 2606, and / or 2608 can utilize one or more client applications to interact with the server 2612 to utilize the services provided by these components.

[0439] In Figure 26 ​In the depicted configuration, software components 2618, 2620, and 2622 of system 2600 are shown as implemented on server 2612. In other embodiments, one or more components of system 2600 and / or the services provided by these components may also be implemented by one or more of client computing devices 2602, 2604, 2606, and / or 2608. A user operating the client computing device can then utilize one or more client applications to use the services provided by these components. These components can be implemented in hardware, firmware, software, or a combination thereof. It should be understood that various different system configurations are possible and may differ from the distributed system 2600. Therefore, Figure 26 The embodiments shown are examples of distributed systems for implementing the system of the embodiments, and are not limiting.

[0440] Client computing devices 2602, 2604, 2606, and / or 2608 may include various types of computing systems. For example, client devices may include portable handheld devices (e.g., Cellular phone Computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google) Head-mounted displays (or similar devices) that run on Microsoft Windows Software such as iOS and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 26, and Palm OS. Devices can support a wide range of applications (such as various internet-related applications, email, and short message service (SMS) applications) and can use various other communication protocols. Client computing devices can also include general-purpose personal computers, for example, running various versions of Microsoft... Apple Personal computers and / or laptops running Linux operating systems. Client computing devices can be any of various commercial operating systems. Workstation computers running UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS). Client computing devices may also include electronic devices capable of providing one or more network 2610 communications (such as thin client computers, internet-enabled gaming systems, e.g., with or without network 2610 communication). Microsoft gesture input devices (game consoles) and / or personal messaging devices).

[0441] Although Figure 26 The distributed system 2600 in the example of FIGURE 26 is shown with four client computing devices, but can support any number of client computing devices. Other devices, such as devices with sensors, etc., can interact with the server 2612.

[0442] The network(s) 2610 in the distributed system 2600 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP (transmission control protocol / Intemet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk, and the like. Merely by way of example, the network(s) 2610 can be a LAN, an Ethernet network, a Token-Ring network, and the like; a wide-area network; the Internet; virtual networks, virtual private networks (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infra-red network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics (IEEE) 2602.11 suite of protocols, Bluetooth®, and / or any other wireless protocol); and / or any combination and / or the like. Of course, it should be understood that the network connectedness implied by the above-described network(s) 2610 can in fact comprise multiple networks and / or different networks at various locations and / or owned / controlled by various

[0443] The server 2612 can be composed of one or more general purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX® servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other arrangement and / or combination that can be devised and implemented in fact. The server 2612 can include one or more virtual machines running virtual operating systems, and other computing architectures employing virtualization technologies. One or more flexible pools of logical storage devices can be virtualized to maintain virtual storage devices for the server. Virtual networks can be controlled by the server 2612 using software defined networking. In various embodiments, the server 2612 can be adapted to execute one or more services or software applications described in the foregoing disclosure. For example, the server 2612 can correspond to a server for performing processes as described above in accordance with embodiments of the present disclosure.

[0444] The server 2612 can be adapted to execute an operating system including any of the operating systems discussed above as well as any commercially available server operating system. Server 2612 can also be adapted to execute any one of a variety of servers, database servers, and the like. Exemplary database servers include, without limitation, database servers commercially available from Oracle, Microsoft, Sybase, IBM (International Business Machines), and the like.

[0445] In some implementations, the server 2612 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client computing devices 2602, 2604, 2606, and 2608. As an example, data feeds and / or event updates can include, but are not limited to, Twitter® feeds, Facebook® updates or real-time updates from one or more third party information sources and continuous data streams. feeds, updates or real-time updates, which can include real-time events related to sensor data applications, financial quotes, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 2612 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 2602, 2604, 2606, and 2608.

[0446] The distributed system 2600 can also include one or more databases 2614 and 2616. These databases can provide mechanisms for storing information such as event information and other information used by embodiments of the present disclosure. The databases 2614 and 2616 can reside in a variety of locations. For example, one or more of the databases 2614 and 2616 can reside on a non-transitory storage medium local to (and / or resident in) the server 2612. Alternatively, the databases 2614 and 2616 can be remote from the server 2612 and in communication with the server 2612 via a network-based or dedicated connection. In one set of embodiments, the databases 2614 and 2616 can reside in a storage area network (SAN). Similarly, any necessary files for performing the functions attributed to the server 2612 can be stored locally on the server 2612 and / or remotely as appropriate. In one set of embodiments, the databases 2614 and 2616 can include relational databases adapted to store, update, and retrieve data in response to SQL-formatted commands, such as the Oracle database available from Oracle.

[0447] The systems depicted in some of the figures can be provided in various configurations. In some embodiments, the systems can be configured as a distributed system, where one or more components of the system are distributed across one or more networked infrastructure systems.

[0448] A cloud infrastructure system is a set of one or more server computing devices, network devices, and / or storage devices working together. These resources can be divided into groups, and assigned to customers in some fashion. For example, a cloud service provider, such as Oracle Corporation of Redwood Shores, California, can provide various types of cloud services, including but not limited to one or more services provided under the Software as a Service (SaaS) category, services provided under the Platform as a Service (PaaS) category, services provided under the Infrastructure as a Service (IaaS) category, or other categories of services including hybrid services. Examples of SaaS services include, without limitation, functionality to build and deliver a suite of on-demand applications, such as the Oracle Fusion Applications. SaaS services enable customers to utilize applications executed on the cloud infrastructure system without the need for customers to purchase software licenses. Examples of PaaS services include, without limitation, services to enable organizations, such as Oracle, to consolidate existing applications on shared, common architectures, as well as the ability to build new applications by utilizing shared services provided by the platform, such as the Oracle Java Cloud Service (JCS), the Oracle Database Cloud Service (DBCS), and others. IaaS services generally provide the ability to manage and control the underlying computing resources, such as storage, networks and other fundamental computing resources for utilizing the services provided by the SaaS and PaaS platforms.

[0449] Figure 27 is a simplified block diagram of one or more components of a system environment 2700 in accordance with an embodiment of the present disclosure, through which services provided by one or more components of an embodiment system can be offered as cloud services. In the illustrated embodiment, the system environment 2700 includes one or more client computing devices 2704, 2706, and 2708 that can be used by users to interact with a cloud infrastructure system 2702 that provides cloud services. The client computing devices can be configured to operate a client application such as a web browser, a proprietary client application (e.g., Oracle Forms), or some other application that can be used by a user of the client computing device to interact with the cloud infrastructure system 2702 to use services provided by the cloud infrastructure system 2702.

[0450] It should be appreciated that the cloud infrastructure system 2702 depicted in the figure can have other components than those depicted. Further, the embodiment shown in the figure is only one example of a cloud infrastructure system that can incorporate an embodiment of the present application. In some other embodiments, the cloud infrastructure system 2702 can have more or fewer components than shown in the figure, can combine two or more components, or can have a different configuration or arrangement of components.

[0451] Client computing devices 2704, 2706, and 2708 can be similar to the devices described above for 502, 504, 506, and 508.

[0452] While the example system environment 2700 is shown with three client computing devices, any number of client computing devices can be supported. Other devices, such as devices with sensors, etc., can interact with the cloud infrastructure system 2702.

[0453] Network(s) 2710 can facilitate communication and exchange of data between clients 2704, 2706, and 2708 and cloud infrastructure system 2702. Each network can be any type of network familiar to those skilled in the art that can support data communication using any of a variety of commercially-available protocols, including those described above for network(s) 2710.

[0454] Cloud infrastructure system 2702 can comprise one or more computers and / or servers that can include those described above for server 2712.

[0455] In certain embodiments, the services provided by the cloud infrastructure system can include a number of services made available to users of the cloud infrastructure system on demand, such as online data storage and backup solutions, Web-based e-mail services, managed office suites and document collaboration services, database processing, managed technical support services, and the like. The services provided by the cloud infrastructure system can dynamically scale to meet the needs of its users. A specific instantiation of the services provided by the cloud infrastructure system is referred to herein as a "service instance.” In general, any

[0456] In some examples, a service in a computer network cloud infrastructure can include protected computer network access to storage devices, hosted databases, hosted web servers, software applications, or other services provided by the cloud vendor to users, or as otherwise known in the art. For example, a service can include password-protected access over the Internet to remote storage devices on the cloud. As another example, a service can include a web services-based hosted relational database and a scripting language middleware engine for private use by a networked developer. As another example, a service can include access to an email software application hosted on a cloud vendor's website.

[0457] In certain embodiments, cloud infrastructure system 2702 can include a suite of applications, middleware, and database service offerings that are delivered to a customer in a self-service, subscription-based

[0458] In various embodiments, cloud infrastructure system 2702 can be adapted to automatically provision, manage and track a customer's subscriptions for services offered by cloud infrastructure system 2702. Cloud infrastructure system 2702 can provide cloud services via different deployment models. For example, services can be provided under a public cloud model where cloud infrastructure system 2702 is owned by an organization selling cloud services (e.g., owned by Oracle) and the services are made available to the general public or different industries or to a private cloud model where cloud infrastructure system 2702 is operated solely for a single organization and can provide services for this organization alone. Cloud services can also be provided under a community cloud model where cloud infrastructure system 2702 and the services provided by cloud infrastructure system 2702 are shared by several organizations in a related community.

[0459] In some embodiments, services provided by cloud infrastructure system 2702 can include one or more services provided under Software as a Service (SaaS) category, Platform as a Service (PaaS) category, Infrastructure as a Service (IaaS) category, or other categories of services including hybrid services. Customers can subscribe to one or more services offered by cloud infrastructure system 2702 via subscription orders. Cloud infrastructure system 2702 then performs processes to provide the services in the customers' subscription orders.

[0460] In some embodiments, the services provided by cloud infrastructure system 2702 can include, without limitation, application services, platform services, and infrastructure services. In some examples, application services can be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform can be configured to provide cloud services that fall into the SaaS category. For example, the SaaS platform can provide capabilities for building and delivering on-demand applications, on an integrated development and deployment platform. The SaaS platform can manage and control the underlying software and infrastructure for providing the SaaS services. By utilizing the services provided by the SaaS platform, customers can utilize applications executing on the cloud infrastructure system. Customers can acquire the application services without the need for customers to purchase, install, and

[0461] In some embodiments, the platform services can be provided by the cloud infrastructure system via a PaaS platform. The PaaS platform can be configured to provide cloud services that fall into the PaaS category. Examples of platform services can include, without limitation, services that enable organizations such as Oracle to

[0462] By utilizing the services provided by the PaaS platform, customers can use programming languages and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system can include database cloud services, middleware cloud services (for example, Oracle Fusion Middleware services), and Java cloud services. In one embodiment, the database cloud services can support a shared services deployment model that enables organizations to pool database resources and supply database-as-a-Service offerings to customers in the form of a database cloud. Middleware cloud services can provide a platform for customers to develop and deploy various business applications in the cloud infrastructure system, and Java cloud services can provide a platform for customers to deploy Java applications.

[0463] Various different infrastructure services can be provided by the IaaS platform in the cloud infrastructure system. The infrastructure services facilitate the management and control of the underlying computing resources, such as storage, networks, and / or other fundamental computing resources for customers utilizing services provided by the SaaS platform and the PaaS platform.

[0464] In certain embodiments, cloud infrastructure system 2702 can also include infrastructure resources 2730 for providing various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources 2730 can include pre-integrated and optimized combinations of hardware, such as servers, storage, and networking resources, to execute the services provided by the PaaS and SaaS platforms.

[0465] In some embodiments, resources in cloud infrastructure system 2702 can be shared by multiple users and dynamically re-allocated per demand. Additionally, resources can be allocated to users in different time zones. For example, cloud infrastructure system 2730 can enable a first group of users in a first time zone to utilize resources of the cloud infrastructure system for a specified number of hours and then enable the same resources to be re-allocated to another group of users located in a different time zone, thereby maximizing the utilization of the resources.

[0466] In certain embodiments, a number of internal shared services 2732 can be provided that are shared by different components or modules of cloud infrastructure system 2702 and by the services provided by cloud infrastructure system 2702. These internal shared services can include, but are not limited to: security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, cloud support enablement services, email services, notification services, file transfer services, etc.

[0467] In certain embodiments, cloud infrastructure system 2702 can provide consolidated management of the cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one embodiment, cloud management functionality can include the ability to provision, manage and track customer subscriptions received by the cloud infrastructure system 2702, etc.

[0468] In one embodiment, as depicted in the figure, cloud management functionality can be provided by one or more modules, such as order management module 2720, order orchestration module 2722, order provisioning module 2724, order management and monitoring module 2726, and identity management module 2728. These modules can include or be provided using one or more computers and / or servers, which can be general purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0469] In example operation 2734, customers using client devices, such as client devices 2704, 2706, or 2708, can interact with cloud infrastructure system 2702 by requesting one or more services offered by cloud infrastructure system 2702 and placing orders to subscribe for one or more services offered by cloud infrastructure system 2702. In certain embodiments, customers can access cloud user interfaces (UIs) (cloud UI 2712, cloud UI 2714, and / or cloud UI 2716) and place orders to subscribe via these UIs. Order information that cloud infrastructure system 2702 receives in response to customers placing orders can include information that identifies the customer and the one or more services that the customer wants to subscribe to that are offered by cloud infrastructure system 2702.

[0470] After a customer places an order, order information is received via cloud UIs 2712, 2714, and / or 2716.

[0471] At operation 2736, the order is stored in order database 2718. Order database 2718 can be one of several databases operated by cloud infrastructure system 2718 and in operation with other system elements.

[0472] At operation 2738, order information is forwarded to order management module 2720. In some cases, order management module 2720 can be configured to perform billing and accounting functions related to orders, such as validating orders, and reserving orders after validation.

[0473] At operation 2740, information about the order is communicated to order orchestration module 2722. Order orchestration module 2722 can utilize order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some cases, order orchestration module 2722 can use the services of order provisioning module 2724 to orchestrate the provisioning of resources to support the subscribed services.

[0474] In certain embodiments, order orchestration module 2722 enables management of business processes associated with each order and application of business logic to determine whether an order should proceed to provisioning. At operation 2742, upon receiving a newly subscribed order, order orchestration module 2722 sends a request to order provisioning module 2724 to allocate resources and configure those resources needed to fulfill the subscribed order. Order provisioning module 2724 enables the allocation of resources for the services ordered by the customer. Order provisioning module 2724 provides an abstraction layer between the cloud services offered by cloud infrastructure system 2700 and the physical implementation layer of resources used to provision the requested services. Thus, order orchestration module 2722 can be insulated from implementation details, such as whether services and resources are provisioned instantaneously or in advance and merely assigned / assigned upon request.

[0475] At operation 2744, once services and resources are supplied, a notification of the supplied services can be sent to customers on client devices 2704, 2706, and / or 2708 via the order provisioning module 2724 of the cloud infrastructure system 2702. At operation 2746, the order management and monitoring module 2726 can manage and track customer subscription orders. In some cases, the order management and monitoring module 2726 can be configured to collect service usage statistics from subscription orders, such as storage usage, data transfer volume, number of users, system uptime, and system downtime.

[0476] In some embodiments, the cloud infrastructure system 2700 may include an identity management module 2728. The identity management module 2728 may be configured to provide identity services, such as access management and authorization services within the cloud infrastructure system 2700. In some embodiments, the identity management module 2728 may control information about customers who wish to utilize services provided by the cloud infrastructure system 2702. Such information may include information authenticating the identities of these customers and information describing what actions these customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, storage segments, etc.). The identity management module 2728 may also include management of descriptive information about each customer and how and by whom this descriptive information is accessed and modified.

[0477] Figure 28 An exemplary computer system 2800 that can be used to implement embodiments of the present disclosure is illustrated. In some embodiments, the computer system 2800 can be used to implement any of the various servers and computer systems described above. Figure 28 As shown, the computer system 2800 includes various subsystems, including a processing unit 2804 that communicates with multiple peripheral subsystems via a bus subsystem 2802. These peripheral subsystems may include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. The storage subsystem 2818 may include a tangible computer-readable storage medium 2822 and system memory 2810.

[0478] Bus subsystem 2802 provides a mechanism for letting the various components and subsystems of computer system 2800 communicate with each other as desired. Although bus subsystem 2802 is illustrated as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. Bus subsystem 2802 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard, etc.

[0479] Processing subsystem 2804 controls the operation of computer system 2800 and can comprise one or more processing units 2832, 2834, etc. Processing units can include one or more processors, including single core or multicore processors, one or more cores of a processor, or combinations thereof. In some embodiments, processing subsystem 2804 can include one or more special-purpose co-processors such as graphics processors, digital signal processors (DSPs), etc. In some embodiments, some or all of the processing units of processing subsystem 2804 can be implemented with customized circuitry such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0480] In some embodiments, processing units in processing subsystem 2804 can execute instructions stored in system memory 2810 or on computer-readable storage media 2822. In various embodiments, processing units can execute a variety of programs or code instructions and can maintain multiple concurrently executing programs or processes. Some or all of the program code to be executed can reside on system memory 2810 and / or computer-readable storage media 2822, potentially including on one or more storage devices, at any given time. Through suitable programming, processing subsystem 2804 can provide various functionality for dynamically modifying documents (e.g., web pages) in response to usage patterns.

[0481] In certain embodiments, a processing acceleration unit 2806 can be provided for performing customized processing or for offloading some of the processing required by processing subsystem 2804 in order to accelerate the overall processing performed by computer system 2800.

[0482] I / O subsystem 2808 can include devices and mechanisms for inputting information to and / or for outputting information from computer system 2800. In general, use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to computer system 2800. User interface input devices can include, for example, a keyboard, pointing devices such as a mouse or trackball, touchpads, touch screens such as a touch panel or touch display, wheels, a trackwheel, dials, buttons, switches, key pads, voice recognition devices that provide an interface for receiving input utilizing gestures and spoken commands. User interface input devices can also include eye gesture recognition devices, such as Google Glass® from Google, Inc., that detect eye activity such as "blinking" to enter input into the Google Glass® device. devices that provide an interface for receiving input utilizing gestures and spoken commands. User interface input devices can also include eye gesture recognition devices, such as Google Glass® from Google, Inc., that detect eye activity such as "blinking" to enter input into the Google Glass® device. Moreover, user interface input devices can include voice recognition sensing devices, such as Apple Siri® from Apple Inc., that enable a user to interact with a voice recognition system (e.g., Apple Siri®) with voice commands.

[0483] Other examples of user interface input devices include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. Additionally, user interface input devices can include, for example, medical imaging input devices such as computerized tomography scanners, magnetic resonance imaging systems, position emission tomography scanners, and medical ultrasonic imaging equipment. User interface input devices can also include, for example, audio input devices such as MIDI keyboards, digital musical instruments and the like.

[0484] ​​User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those utilizing liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from computer system 2800 to a user or other computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0485] Storage subsystem 2818 provides a repository or data storage for storing information used by computer system 2800. Storage subsystem 2818 provides a tangible, non-transient, computer-readable storage medium for storing basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the above-described functionality when executed by processing subsystem 2804 can be stored in storage subsystem 2818. The software can be executed by one or more processing units of processing subsystem 2804. Storage subsystem 2818 may also provide a repository for storing data used according to this disclosure.

[0486] The storage subsystem 2818 may include one or more non-transient memory devices, including volatile and non-volatile memory devices. For example... Figure 28 As shown, the storage subsystem 2818 includes system memory 2810 and computer-readable storage medium 2822. System memory 2810 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some implementations, a basic input / output system (BIOS) containing basic routines that facilitate the transfer of information between elements within computer system 2800, such as during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 2804. In some implementations, system memory 2810 may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0487] As an example rather than a limitation, such as in Figure 28 As illustrated, system memory 2810 can store application programs 2812, which may include client applications, web browsers, middleware applications, relational database management systems (RDBMS), program data 2814, and operating system 2816. As an example, operating system 2816 may include various versions of Microsoft... Apple and / or Linux operating system, various commercial Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Phone OS 26OS and A mobile operating system based on the OS operating system.

[0488] Computer-readable storage medium 2822 may store programming and data structures that provide functionality for some embodiments. Software (programs, code modules, instructions) that enables the processor to provide the aforementioned functionality when executed by processing subsystem 2804 may be stored in storage subsystem 2818. As an example, computer-readable storage medium 2822 may include non-volatile memory, such as a hard disk drive, a disk drive, or a device such as a CD-ROM or DVD. Optical disc drives for (Blu-ray) discs or other optical media. Computer-readable storage media 2822 may include, but is not limited to, Drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVDs, digital video tapes, etc. Computer-readable storage media 2822 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs), and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Computer-readable media 2822 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 2800.

[0489] In some embodiments, the storage subsystem 2800 may also include a computer-readable storage medium reader 2820, which may be further connected to a computer-readable storage medium 2822. Optionally, together with and in combination with the system memory 2810, the computer-readable storage medium 2822 may comprehensively represent remote, local, fixed and / or removable storage devices plus storage media for storing computer-readable information.

[0490] In certain embodiments, computer system 2800 can provide support for executing one or more virtual machines. Computer system 2800 can execute a program, such as an operating system, in order to facilitate the execution of virtual machines. Each virtual machine can be allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs its own operating system, which can be the same as or different from operating systems executed by other virtual machines executed by computer system 2800. Accordingly, multiple operating systems can potentially be concurrently executed by computer system 2800. Each virtual machine generally operates independently of the other virtual machines.

[0491] Communication subsystem 2824 provides an interface to other computer systems and networks. Communication subsystem 2824 is used in the reception and transmission of data with other systems maintaining database 2802. Communication subsystem 2824 can also be used to transmit notifications from the privileged account manager to requesting users of a successful login or a notification to re-enter a password.

[0492] Communication subsystem 2824 can support both wired and / or wireless communication protocols. For example, in certain embodiments, communication subsystem 1724 can include radio-frequency (RF) transceiver components for accessing wireless voice and / or data networks, Global Positioning System (GPS) receiver components, and / or other components. In some embodiments, communication subsystem 2824 can provide wired networking capabilities (e.g., Ethernet) in addition to or instead of wireless networking capabilities.

[0493] Communication subsystem 2824 can receive and transmit data in various forms. For example, in some embodiments, communication subsystem 2824 can receive input communications in the form of structured and / or unstructured data feeds 2826, event updates 2828, event updates 2830, and the like from various networks (e.g., social media networks). For example, communication subsystem 1724 can be configured to receive (or send) data feeds 2826 in the form of structured and / or unstructured data feeds 2826 (e.g., RSS feeds), event updates, web feeds (e.g., Facebook updates, Twitter tweets, and the like), and the like. feeds, updates, such as rich site summary (RSS) feeds, web feeds, and other communications services that provide updates in real time (or near real time), and the like from one or more third party information sources.

[0494] In some embodiments, the communication subsystem 2824 may be configured to receive data that may be inherently continuous or unbounded without a definite end, in the form of a continuous data stream, wherein the continuous data stream may include an event stream 2828 of real-time events and / or event updates 2830. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0495] The communication subsystem 2824 can also be configured to output structured and / or unstructured data feeds 2826, event streams 2828, event updates 2830, etc. to one or more databases, wherein the one or more databases can communicate with one or more streaming data source computers coupled to the computer system 2800.

[0496] The computer system 2800 can be of a variety of types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), and wearable devices (e.g., Google). Head-mounted displays, personal computers, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0497] Due to the constantly changing nature of computers and networks, Figure 28 The description of the computer system 2800 shown in the illustration is intended only as a concrete example. It has more than Figure 28 Many other configurations of the system with more or fewer components are possible. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0498] While specific embodiments of this disclosure have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments of this disclosure are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although embodiments of this disclosure have been described using a specific series of transactions and steps, it will be clear to those skilled in the art that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0499] In addition, while embodiments of the disclosure have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the disclosure. Embodiments of the disclosure can be implemented only in hardware, or only in software, or using combinations thereof. The various processes described herein can be implemented on the same processor or different processors in any combination. Accordingly, where components or modules of the disclosure have been described as being configured to perform certain operations, such configuration can be accomplished, e.g., through design of electronic circuits, through programming of programmable electronic circuits (such as microprocessors), or in any combination thereof. Processes can communicate using a variety of technologies, including but not limited to conventional technologies for interprocess communication, and different processes can use different technologies for communication, or the same pair of processes can use different technologies at different times.

[0500] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it will be apparent to those skilled in the art that modifications, additions and subtractions can be made to the disclosed embodiments without departing from the broader spirit and scope of the disclosure as set forth in the following claims. Thus, while specific embodiments have been described in detail, these particular embodiments are not intended to limit the scope of the disclosure. Various modifications and equivalents can be used in the practice of the disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of the disclosure. Modifications and variations are included in the scope of the following claims.< / orderamount> < / orderstatus> < / orderid> < / orderamount> < / orderstatus> < / orderid> ​< / stockprice> < / attributes>

Claims

1. A method for processing a continuous data stream of events using a distributed event processing system, the method comprising: A batch of events from an event stream is received at a computing device among multiple computing devices in the distributed event processing system; The computing device identifies a first attribute of a first data type of an event in the batch of events; The computing device determines the number of bits used to store the data value represented by the first attribute and a unique set of values ​​represented by the first attribute; The computing device determines, at least in part, the first type of data compression to be performed on the data value represented by the first attribute of the event, based on the first data type of the first attribute, the number of bits used to store the data value represented by the first attribute, and the unique set of values ​​represented by the first attribute. Based at least in part on the determination, the computing device generates a first set of serialized data values ​​for the first attribute; The computing device processes the first set of serialized data values ​​corresponding to the first attribute for a set of one or more continuous queries to generate a first output event; and The computing device sends the first output event to the user.

2. The method according to claim 1, wherein, The processing of the first set of serialized data values ​​by the computing device further includes: The computing device generates a first set of deserialized data values ​​corresponding to the first attribute, based at least in part on the first set of serialized data values; and The computing device processes the first set of deserialized data values ​​corresponding to the first attribute for the set of one or more continuous queries to generate the first output event.

3. The method according to claim 2, wherein, The generation of the first set of deserialized data values ​​corresponding to the first attribute by the computing device further includes: The computing device identifies the first type of data compression performed on the data value represented by the first attribute; and The computing device deserializes the first set of serialized data values ​​represented by the first attribute according to the first type of data compression.

4. The method according to claim 1, further comprising: The computing device identifies a second attribute of a second data type for an event in the batch of events, wherein the second data type is different from the first data type. The computing device determines a second type of data compression to be performed on the data value represented by the second attribute, the second type of data compression being different from the first type of data compression; Based at least in part on the determination, the computing device generates a second set of serialized data values ​​represented by the second attribute; and The computing device stores the second set of serialized data values ​​represented by the second attribute.

5. The method according to claim 4, wherein, The second type of data compression is determined at least in part based on the second data type of the second attribute.

6. The method of claim 4, further comprising processing the second set of serialized data values ​​by the computing device, the processing comprising: The computing device generates a second set of deserialized data values ​​corresponding to the second attribute based on the second set of serialized data values; and The computing device processes the second set of deserialized data values ​​corresponding to the second attribute for the set of one or more continuous queries to generate the first output event.

7. The method according to claim 4, wherein, The first type of data compression or the second type of data compression includes at least one of the following: base compression technology, value index compression technology, or precision reduced value index compression technology.

8. The method according to claim 1, wherein, The processing of the first set of serialized data values ​​corresponding to the first attribute by the computing device further includes: The computing device identifies a set of one or more operations to be performed on the events in the batch of events based on the set of one or more continuous queries; The computing device represents the set of one or more operations as transformations in the Continuous Query Language (CQL) Resilient Distributed Dataset (RDD) Directed Acyclic Graph (DAG); and The computing device processes the first set of serialized data values ​​corresponding to the first attribute using a CQL transformation to generate the first output event.

9. The method according to claim 4, wherein, The first data type of the first attribute is a numeric data type, and the second data type of the second attribute is a non-numeric data type.

10. A computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, configure one or more computer systems to perform at least the following: Instructions to enable the one or more processors to receive a batch of events from an event stream at a computing device among multiple computing devices in a distributed event processing system; Instructions that enable the one or more processors to identify a first attribute of a first data type of an event in the batch of events; The one or more processors are instructed to determine the number of bits for storing the data value represented by the first attribute and the set of unique values ​​represented by the first attribute. Instructions that cause the one or more processors to determine, at least in part, a first type of data compression to be performed on the data value represented by the first attribute of the event, based on the first data type of the first attribute, the number of bits for storing the data value represented by the first attribute, and the unique set of values ​​represented by the first attribute. Instructions that cause the one or more processors to generate, at least in part, a first set of serialized data values ​​for the first attribute based on the determination; Instructions to cause the one or more processors to process the first set of serialized data values ​​corresponding to the first attribute in response to one or more continuous queries to generate a first output event; and Instruct the one or more processors to send the first output event to the user.

11. The computer-readable medium of claim 10, wherein, The instructions for causing the one or more processors to process the first set of serialized data values ​​further include: Instructions that cause the one or more processors to generate, at least in part, a first set of deserialized data values ​​corresponding to the first attribute, based on the first set of serialized data values; and Instructions to cause the one or more processors to process the first set of deserialized data values ​​corresponding to the first attribute in response to the one or more continuous queries to generate the first output event.

12. The computer-readable medium of claim 11, wherein, The instructions for causing the one or more processors to generate the first set of deserialized data values ​​corresponding to the first attribute further include: The one or more processors are made to recognize instructions for data compression of the first type performed on the data value represented by the first attribute; and Instructions that cause the one or more processors to deserialize the first set of serialized data values ​​represented by the first attribute according to the first type of data compression.

13. The computer-readable medium of claim 10, further comprising: Instructions that enable the one or more processors to recognize a second attribute of a second data type of the event, wherein the second data type is different from the first data type; The one or more processors determine instructions to perform a second type of data compression on a data value represented by the second attribute of the event in the batch of events, the second type of data compression being different from the first type of data compression; The instructions, at least in part, cause the one or more processors to generate a second set of serialized data values ​​represented by the second attribute, based on instructions that cause the one or more processors to determine the second type of data compression to be performed on the data values ​​represented by the second attribute. and Instructions to cause the one or more processors to store the second set of serialized data values ​​represented by the second attribute.

14. The computer-readable medium according to claim 13, wherein, The second type of data compression is determined at least in part based on the second data type of the second attribute.

15. The computer-readable medium according to claim 13, wherein, The first type of data compression or the second type of data compression includes at least one of the following: base compression technology, value index compression technology, or precision reduced value index compression technology.

16. A distributed event processing system, comprising: Memory, which stores multiple instructions; and A processor, configured to access the memory, and further configured to execute the plurality of instructions to at least: A batch of events from an event stream is received at a computing device among multiple computing devices in the distributed event processing system; Identify the first attribute of the first data type of the events in the batch of events; Determine the number of bits used to store the data value represented by the first attribute and a unique set of values ​​represented by the first attribute; The first type of data compression to be performed on the data value represented by the first attribute of the event is determined based at least in part on the first data type of the first attribute, the number of bits used to store the data value represented by the first attribute, and the unique set of values ​​represented by the first attribute. At least in part based on the instructions used to make the determination, a first set of serialized data values ​​for the first attribute is generated; Process the first set of serialized data values ​​corresponding to the first attribute for a set of one or more continuous queries to generate a first output event; and Send the first output event to the user.

17. The system according to claim 16, wherein, The processor is further configured to execute the plurality of instructions to process the first set of serialized data values ​​by: At least in part, a first set of deserialized data values ​​corresponding to the first attribute are generated based on the first set of serialized data values; and The first set of deserialized data values ​​corresponding to the first attribute are processed for one or more continuous queries to generate the first output event.

18. The system according to claim 16, wherein, The processor is further configured to execute the plurality of instructions to: Identify a second attribute of a second data type for an event in the batch of events, wherein the second data type is different from the first data type; Determine a second type of data compression to be performed on the data value represented by the second attribute, the second type of data compression being different from the first type of data compression; A second set of serialized data values, represented by the second attribute, is generated, based at least in part on the instructions used to make the determination; and Store the second set of serialized data values ​​represented by the second attribute.

19. The system according to claim 18, wherein, The second type of data compression is determined at least in part based on the second data type of the second attribute.

20. The system according to claim 16, wherein, The processor is further configured to execute the plurality of instructions to process the first set of serialized data values ​​through the following steps: Based on the set of one or more continuous queries, identify one or more sets of operations to be performed on the events in the batch of events; The set of one or more operations is represented as a transformation of the Continuous Query Language (CQL) Resilient Distributed Dataset (RDD) Directed Acyclic Graph (DAG); and The first set of serialized data values ​​corresponding to the first attribute are processed using CQL transformation to generate the first output event.

21. A method comprising: This confirms that the event handling system is in the stage of processing a Continuous Query Language (CQL) query; Determine the phase type associated with the phase; The transformations to be computed for the stage are determined at least in part based on the stage type; The classification of the CQL query is determined at least in part based on multiple rules; Based on the determination that the classification of the CQL query is stateless, the stage is marked as a partition stage; Based on the determination that the classification of the CQL query is stateful, the stage is marked as an unpartitioned stage; In response to marking the stage as a partitioned stage: The partitioning phase generates a first transformation in the first directed acyclic graph (DAG) of the data transformation pipeline without any repartitioning transformation; and In response to marking the stage as an unpartitioned stage: In the second directed acyclic graph, a second transformation is generated by using a repartitioning transformation followed by a stage transformation to the unpartitioned stage; and Input the partition attributes and the number of partitions into the repartition transformation.

22. The method according to claim 21, wherein, Determining the transformation to be computed for the stage includes parsing the CQL query into one or more passwords.

23. The method according to claim 21, wherein, The multiple rules include rules that classify the CQL query as at least one of stateless query, semi-stateful query, or fully stateful query.

24. The method of claim 21, further comprising: In response to marking the stage as a partitioned stage or an unpartitioned stage, a partitioning criterion is applied to the stage, wherein the partitioning criterion includes one or more partitioning attributes.

25. The method of claim 24, further comprising performing semantic analysis on the CQL query.

26. The method of claim 25, wherein, The one or more partition attributes are determined at least in part based on the semantic analysis of the CQL query.

27. The method of claim 21, further comprising traversing the data transformation pipeline from its source to its sink.

28. The method according to claim 21, wherein, In response to marking the stage as a partitioned stage, the number of partitions in the stage is determined at least in part based on the number of partitions from the previous stage.

29. A system comprising: Memory, configured to store computer-executable instructions; and The processor is configured to access the memory and execute the computer-executable instructions to: This confirms that the event handling system is in the stage of processing a Continuous Query Language (CQL) query; Determine the phase type associated with the phase; The transformations to be computed for the stage are determined at least in part based on the stage type; The classification of the CQL query is determined at least in part based on multiple rules; Based on the determination that the classification of the CQL query is stateless, the stage is marked as a partition stage; Based on the determination that the classification of the CQL query is stateful, the stage is marked as an unpartitioned stage; In response to marking the stage as a partitioned stage: The partitioning phase determines the first transformation in the first directed acyclic graph (DAG) of the data transformation pipeline without any repartitioning transformation; and In response to marking the stage as an unpartitioned stage: In the second directed acyclic graph, a second transformation is determined using a repartitioning transformation followed by a stage transformation to the unpartitioned stage; and Input the partition attributes and the number of partitions into the repartition transformation.

30. The system according to claim 29, wherein, Determining the transformation to be computed for the stage includes parsing the CQL query into one or more passwords.

31. The system according to claim 29, wherein, The multiple rules include rules that classify the CQL query as at least one of stateless query, semi-stateful query, or fully stateful query.

32. The system according to claim 29, further comprising: In response to marking the stage as a partitioned stage or an unpartitioned stage, a partitioning criterion is applied to the stage, wherein the partitioning criterion includes one or more partitioning attributes.

33. The system according to claim 32, wherein, The computer-executable instructions are further executed to perform semantic analysis on the CQL query.

34. The system according to claim 33, wherein, The one or more partition attributes are determined at least in part based on the semantic analysis of the CQL query.

35. The system according to claim 29, wherein, The computer-executable instructions are further executed to traverse the data transformation pipeline from its source to its sink.

36. A non-transitory computer-readable medium storing computer-executable code, said computer-executable code causing the processor to perform operations when executed by a processor, said operations including: This confirms that the event handling system is in the stage of processing a Continuous Query Language (CQL) query; Determine the phase type associated with the phase; The transformations to be computed for the stage are determined at least in part based on the stage type; The classification of the CQL query is determined at least in part based on multiple rules; Based on the determination that the classification of the CQL query is stateless, the stage is marked as a partition stage; Based on the determination that the classification of the CQL query is stateful, the stage is marked as an unpartitioned stage; In response to marking the stage as a partitioned stage: The partitioning phase generates a first transformation in the directed acyclic graph (DAG) of the data transformation pipeline without any repartitioning transformation; and In response to marking the stage as an unpartitioned stage: In the second directed acyclic graph, a second transformation is generated by using a repartitioning transformation followed by a stage transformation to the unpartitioned stage; and Input the partition attributes and the number of partitions into the repartition transformation.

37. The non-transitory computer-readable medium according to claim 36, wherein, Determining the transformation to be computed for the stage includes parsing the CQL query into one or more passwords.

38. The non-transitory computer-readable medium according to claim 36, wherein, The multiple rules include rules that classify the CQL query as at least one of stateless query, semi-stateful query, or fully stateful query.

39. The non-transitory computer-readable medium of claim 36, further comprising: In response to marking the stage as a partitioned stage or an unpartitioned stage, a partitioning criterion is applied to the stage, wherein the partitioning criterion includes one or more partitioning attributes.

40. The non-transitory computer-readable medium according to claim 39, wherein, The operation also includes performing semantic analysis on the CQL query, wherein the one or more partition attributes are determined at least in part based on the semantic analysis of the CQL query.

Images