Dynamic replacement of degenerate processing elements in streaming applications

Abstract

A computer-implemented method is provided. The method includes monitoring, at runtime, an occurrence of a distributed computing application condition, wherein the condition includes an occurrence of a degradation condition on a processing element (PE). The method also includes launching, at runtime, a temporary PE on a second node while maintaining runtime operations of the PE. The method compares operations of the PE with operations of the temporary PE based on performance metrics associated with computing resources. The method connects the temporary PE to replace a tuple stream from the PE to a downstream PE with a tuple stream from the temporary PE to the downstream PE.

Description

Background Technology

[0001] This invention generally relates to methods, systems, and computer program products for stream computing. More specifically, this invention relates to methods, systems, and computer program products for dynamically replacing processing elements in stream applications.

[0002] Database systems are typically configured to separate the process of storing data from accessing, manipulating, or using the data stored in the database. In traditional static database systems, data is first stored and indexed in memory before subsequent queries and analysis. Often, such static database systems are not always suitable for performing real-time processing and analysis of streaming data. For example, static database systems sometimes cannot efficiently or in real-time store, index, and analyze large amounts of streaming data.

[0003] In recent years, stream-based computing and stream-based database computing have emerged as development techniques for database systems. In streaming applications, nodes interconnected via a network allow data to flow from one node to the next. Such data streams encapsulate blocks of data in "tuples." A tuple is a block of data of one or more different data types, such as integers, floating-point numbers, boolean values, or string data. Groups of tuples are transmitted sequentially, and this is called a "stream" or "data stream." Specifically, data may arrive substantially consecutively, as a stream of data points corresponding to ongoing or continuous events.

[0004] For example, data representing a particular stock price may fluctuate throughout the day, and a data stream management system can continuously receive updated stock prices, for example, at equal time intervals or as prices change. Other examples of such data streams include temperature or other environmental data collected by sensors, computer network analysis, patient health data collected in hospitals, or data describing manufacturing processes or other business processes. Summary of the Invention

[0005] Illustrative embodiments provide dynamic replacement of degrading processing elements in streaming applications. One embodiment includes monitoring the occurrence of conditions in a distributed computing application at runtime, wherein the distributed computing application includes a first node, the first node including processing elements (PEs), the PE receiving a first tuple stream from an upstream PE, executing a defined process on the first tuple stream node and generating a second tuple stream, and transmitting the second tuple stream to a downstream PE, wherein the conditions include a degradation condition occurring on the PE. This embodiment also includes launching a first temporary PE at runtime on a second node while maintaining runtime operation of the PE, wherein the first temporary PE receives the first tuple stream from the upstream PE and executes a defined process on the first tuple stream. This embodiment also includes comparing the operation of the PE with the operation of the first temporary PE based on performance metrics associated with computing resources. This embodiment also includes connecting the first temporary PE to replace the second tuple stream from the PE to the downstream PE with a third tuple stream from the first temporary PE to the downstream PE. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of this embodiment.

[0006] One embodiment includes a computer-usable program product. The computer-usable program product includes a computer-readable storage medium and program instructions stored on the storage medium.

[0007] One embodiment includes a computer system. The computer system includes a processor, a computer-readable storage device, and a computer-readable storage medium, as well as program instructions stored on the storage medium for execution by the processor via the memory. Attached Figure Description

[0008] Novel features considered characteristic of the invention are set forth in the appended claims. However, the invention itself, as well as preferred modes of use, further objects and advantages, will be best understood by referring to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 A block diagram of a data processing system network in which illustrative embodiments can be implemented is shown;

[0010] Figure 2 A block diagram of a data processing system in which illustrative embodiments can be implemented is shown;

[0011] Figure 3 A block diagram of an example streaming application environment according to an illustrative embodiment is shown;

[0012] Figure 4 A block diagram of an example compute node according to an illustrative embodiment is shown;

[0013] Figure 5 A block diagram of an example management system according to an illustrative embodiment is shown;

[0014] Figure 6 A block diagram of an example temporary PE analysis module according to an illustrative embodiment is shown;

[0015] Figure 7 A block diagram of an operator graph for a stream computing application according to an illustrative embodiment is shown;

[0016] Figure 8 A block diagram of the operator graph of a stream computing application with a temporary PE according to an illustrative embodiment is shown;

[0017] Figure 9 A flowchart depicts an example dynamic PE replacement process according to an illustrative embodiment;

[0018] Figure 10 A flowchart illustrating an example temporary PE fault troubleshooting process according to an illustrative embodiment is shown;

[0019] Figure 11 A flowchart illustrating an example temporary PE fault troubleshooting process according to an illustrative embodiment is shown; and

[0020] Figure 12 A flowchart illustrating an example temporary PE fault troubleshooting process according to an illustrative embodiment is shown. Detailed Implementation

[0021] Stream-based computing and stream-based database computing are emerging technologies in database systems. Available products allow users to create applications that process and query streaming data before it arrives at the database file. With this emerging technology, users can specify processing logic to be applied to inbound data records while they are "in flight," obtaining results in a very short time (typically a fraction of a second). Building applications using this type of processing opens up a new programming paradigm that will allow for the development of a wide range of innovative applications, systems, and processes, presenting new challenges for application programmers and database developers alike.

[0022] The main components of a stream processing application include tuples, data streams, operators, PEs, and jobs. A "tuple" is a single piece of data in a data stream. As used herein, a "data stream" refers to a sequence of tuples in operation. As used herein, an "operator" is a logical function that manipulates tuple data from an incoming data stream and produces results as an output data stream. As used herein, an "operator graph" is a visual representation of operators connected by data streams flowing through them and defines the analysis application. A "processing element" or "PE" as used herein is an executable file containing one or more operators that run in the same process on a network node or computing resource (e.g., a server, client, host, container) or other computing device (e.g., a data processing system). A "node" as used herein is a network node, including any electronic device addressable via a computer network capable of creating, receiving, and / or transmitting information over the network, and may refer to an element, module, component, board, device, or system.

[0023] A stream processing job consists of a directed graph of PEs that send data tuples between PEs. PEs operate on the incoming tuples and generate output tuples. Each PE has an independent processing unit and runs on a host. A streaming platform can consist of a group of hosts that are qualified to host PEs.

[0024] In stream computing applications, PEs are thus connected to each other, allowing data to flow from one PE to the next via a network connection (e.g., via a TCP / IP socket). Data flows from one stream operator to another in the form of tuples, which are sequences of one or more attributes associated with an entity.

[0025] Stream computing applications handle massive amounts of data that require efficient and real-time processing. Stream computing achieves high-performance streaming and scalability by distributing the application across multiple nodes using executables (i.e., multiple physical instances), and by replicating these instances and load balancing them. Therefore, stream computing applications can continuously ingest and analyze hundreds of thousands of messages per second, and up to petabytes of data per day. Consequently, each stream operator in a stream computing application may need to process received tuples within fractions of a second.

[0026] One advantage of products like IBM Streams is their tremendous scalability and performance (IBM is a registered trademark of International Business Machines Corporation). Performance advantages include improvements in ingesting, filtering, analyzing, and correlating potentially large volumes of continuous data streams within a given timeframe. However, if a streaming application "lags behind" in its processing, the analysis it is performing loses its value—that is, the analysis results are based on past, not recent, data.

[0027] Streaming applications may fall behind simply because a part of the application is performing poorly. Streaming applications may also fall behind due to failures within the application itself. Streaming service providers or operating systems may support high availability and application failover, but fault recovery mechanisms include recovering from the period when data is not being processed. For example, the time required for a processing element (PE) to fail and restart can range from 20 to 30 seconds to several minutes, depending on a variety of factors (e.g., rapid failure versus gradual failure of the PE running slowly and slowing down before eventually failing, the amount of time to create / schedule new resources to host the restarted PE, the amount of time to deploy a new PE, etc.).

[0028] The longer the fault recovery process takes, the longer the backlog of unprocessed real-time data becomes, causing the application to lag behind real-time until it "catches up" with the latest data. If multiple faults occur, the long recovery period may cause the application to be completely overwhelmed by real-time data, leading to its complete failure.

[0029] The illustrative embodiments of this disclosure address streaming-specific problems and performance goals by monitoring the degradation of a streaming processing element (PE) and replacing the degraded PE with a better-performing copy of itself (before it fails). Some of these embodiments offer one or more advantages over the prior art, such as improving the performance of streaming applications by replacing poorly performing PEs, providing a method to avoid PE failures, and providing a method to avoid PE failures and associated lengthy recovery by proactively replacing degraded PEs with healthier copies of themselves before failure, and enabling a faster switch to an alternative PE than through a full failure recovery scenario, thereby reducing or preventing the accumulation of unprocessed real-time data during failure recovery.

[0030] In illustrative embodiments, the flow manager monitors the streaming application and maintains statistics representing the performance of the various PEs that constitute the streaming application. In some embodiments, the flow manager monitors PE degradation and, when degradation of one or more PEs is detected, performs a troubleshooting process or sends notification signals or data to a troubleshooting module or a separate application. For example, in some embodiments, the PE monitors its own degradation metrics and sends a signal to the flow manager when it detects degradation.

[0031] In some embodiments, the flow manager detects degradation of a PE by monitoring whether a condition used as a degradation indicator occurs, which may vary from PE to PE. In some embodiments, the threshold for the degradation indicator condition may also vary from PE to PE. Therefore, in some such embodiments, the degradation indicator condition is configurable for each PE.

[0032] In some embodiments, degradation indication conditions include execution conditions associated with a PE that deviates from a predefined or expected pattern. For example, in some embodiments, degradation indications include an abnormal number of queued tuples (i.e., tuples waiting to be processed at the input ports of operators in the PE), abnormal memory usage, the number and type of exception handling, and abnormal CPU consumption.

[0033] In some embodiments, the flow manager monitors multiple PEs. In an alternative embodiment, the flow manager monitors only a single PE. In some embodiments, the flow manager maintains statistics representing the normal or expected performance of the PE before it degrades. In some embodiments, the flow manager records the PE's utilization of such computing resources, thereby allowing the establishment of normal utilization patterns and thus also allowing the detection of anomalous utilization patterns.

[0034] In some embodiments, an abnormal number of queued tuples includes a higher number of queued tuples than normal. In some embodiments, abnormal memory usage includes an abnormally high rate of change in memory usage. In some embodiments, abnormal CPU consumption includes an abnormally high rate of change in CPU consumption.

[0035] In some embodiments, when the flow manager detects a degradation of a performance degradation PE, the flow manager performs an initial analysis related to the degraded PE. In some such embodiments, the flow manager analyzes the degraded PE to determine whether the indication of performance degradation is actually an indication of a problem rather than a degradation of the performance degradation PE, such as backpressure or one or more cluster computing resources reaching capacity.

[0036] In some such embodiments, when the flow manager detects PE degradation and no alternative issues are detected, the flow manager tests one or more temporary PEs as possible replacements for the degraded PE. In some such embodiments, the flow manager communicates with the scheduler to identify the node hosting the temporary PE. In some embodiments, the scheduler searches for nodes other than the node hosting the PE. In some embodiments, the scheduler also searches for nodes based on predetermined rules or criteria, such as the availability of computing resources and proximity to upstream and downstream PEs. In some embodiments, once the scheduler identifies the host node, the scheduler informs the flow manager of the location to be selected for the temporary PE.

[0037] In some embodiments, the flow manager initiates a temporary PE on the identified node at runtime, while maintaining the runtime operation of the performance-degraded PE. In some such embodiments, the flow manager connects the temporary PE to the upstream PE of the same upstream PE as the degraded PE, such that both the temporary and degraded PEs simultaneously receive the same tuple stream from the same upstream PE. In some such embodiments, both the temporary and degraded PEs perform the same definition procedure on the tuple stream, which allows the flow manager to compare the performance of the degraded PE with that of the temporary PE. In some such embodiments, the degraded PE performs the definition procedure on the tuple stream, producing an output tuple stream that the degraded PE transmits to downstream PEs, while the output of the temporary PE is not connected to any downstream element.

[0038] In some embodiments, the flow manager compares the operation of a degraded PE with the operation of a temporary PE based on performance metrics associated with one or more computing resources. For example, in some embodiments, the flow manager compares the operation of a degraded PE with the operation of a temporary PE by comparing one or more of CPU utilization, memory utilization, and tuple throughput. In some such embodiments, if the flow manager determines that the temporary PE performs better than the degraded PE, the flow manager replaces the degraded PE with the temporary PE, for example by connecting the output of the temporary PE to the downstream location of the degraded PE, and the flow manager shuts down the degraded PE. In some embodiments, the flow manager sends a notification to the user via a user interface regarding the replacement of the degraded PE with a temporary PE.

[0039] For clarity of description and without implying any limitation thereof, the illustrative embodiments are described using some example configurations. Based on this disclosure, those skilled in the art will be able to conceive of many changes, adaptations, and modifications to the described configurations to achieve the described purposes, and the same is contemplated within the scope of the illustrative embodiments.

[0040] Furthermore, simplified diagrams of the data processing environment are used in the accompanying drawings and illustrative embodiments. In a real computing environment, additional structures or components not shown or described herein, or structures or components different from those shown but used for similar functions as described herein, may exist without departing from the scope of the illustrative embodiments.

[0041] Furthermore, the exemplary embodiments are described merely as examples for specific actual or hypothetical components. For instance, the steps described in the various illustrative embodiments can be applied to explain the decisions made by machine learning classifier models.

[0042] Any specific representation of these and other similar artifacts is not intended to limit the invention. Any suitable representation of these and other similar artifacts may be chosen within the scope of the illustrative embodiments.

[0043] The examples in this disclosure are for clarity of description only and are not intended to limit the illustrative embodiments. Any advantages listed herein are merely examples and are not intended to limit the illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, specific illustrative embodiments may have some, all, or none of the advantages listed above.

[0044] Furthermore, exemplary embodiments can be implemented for any type of data, data source, or data source accessed via a data network. Within the scope of this invention, any type of data storage device can provide data to embodiments of the invention either locally on the data processing system or via a data network. Within the scope of the illustrative embodiments, where embodiments are described using mobile devices, any type of data storage device suitable for use with mobile devices can provide data to such embodiments either locally on the mobile device or via a data network.

[0045] Exemplary embodiments are described using specific code, comparative illustrations, computer-readable storage media, advanced features, historical data, designs, architectures, protocols, layouts, diagrams, and tools as examples only, and are not limited to exemplary embodiments. Furthermore, in some cases, specific software, tools, and data processing environments are used to describe illustrative embodiments only as examples for clarity of description. Illustrative embodiments may be used in conjunction with other comparable or similar structures, systems, applications, or architectures. For example, other similar mobile devices, structures, systems, applications, or architectures may be used in conjunction with such embodiments of the present invention within the scope of the invention. Illustrative embodiments may be implemented in hardware, software, or a combination thereof.

[0046] The examples in this disclosure are for illustrative purposes only and are not intended to limit the scope of the illustrative embodiments. Additional data, operations, actions, tasks, activities, and manipulations will arise from this disclosure and are contemplated similarly within the scope of the illustrative embodiments.

[0047] Any advantages listed herein are merely illustrative and are not intended to limit the illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, specific illustrative embodiments may have some, all, or none of the advantages listed above.

[0048] Refer to the attached diagram and for details. Figure 1 and Figure 2 These figures are example diagrams of a data processing environment in which illustrative embodiments can be implemented. Figure 1 and Figure 2 This is merely an example and is not intended to assert or imply any limitation regarding the environments in which different embodiments may be implemented. Based on the following description, specific implementations may make many modifications to the described environment.

[0049] Figure 1 A block diagram of a data processing system network in which illustrative embodiments can be implemented is shown. Data processing environment 100 is a computer network in which illustrative embodiments can be implemented. Data processing environment 100 includes network 102. Network 102 is a medium for providing communication links between various devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0050] The client or server is merely an example role for certain data processing systems connected to network 102 and is not intended to exclude other configurations or roles of these data processing systems. Data processing system 104 is coupled to network 102. Software applications can execute on any data processing system within data processing environment 100. Figure 1 The description states that any software application executing in processing system 104 can be configured to execute in another data processing system in a similar manner. Figure 1 Any data or information stored or generated in data processing system 104 can be configured to be stored or generated in another data processing system in a similar manner. Data processing systems such as data processing system 104 may contain data and may have software applications or software tools that perform computational processes thereon. In one embodiment, data processing system 104 includes memory 124 including an application program 105A that can be configured to implement one or more data processor functions described herein according to one or more embodiments.

[0051] Server 106, together with storage unit 108, is coupled to network 102. Storage unit 108 includes database 109, which is configured to store data as described herein with respect to various embodiments, such as image data and attribute data. Server 106 is a conventional data processing system. In one embodiment, server 106 includes a PE for a stream processing application 105B, which may be configured to implement one or more processor functions described herein according to one or more embodiments.

[0052] Clients 110, 112, and 114 are also coupled to network 102. Conventional data processing systems, such as server 106 or clients 110, 112, or 114, can contain data and may have software applications or software tools that perform conventional calculations on them.

[0053] This is merely an example and does not imply any limitations on this architecture. Figure 1 Some components that may be used in an example implementation of the embodiments are shown. For example, server 106 and clients 110, 112, 114 are described as servers and clients only, and are not intended to limit the client-server architecture. As another example, one embodiment may be distributed across several data processing systems and data networks, as shown, while another embodiment may be implemented on a single data processing system within the scope of the illustrative embodiments. Conventional data processing systems 106, 110, 112, and 114 also represent example nodes, partitions, and other configurations suitable for implementing the embodiments in a cluster.

[0054] Device 132 is an example of a conventional computing device described herein. For example, device 132 may take the form of a smartphone, tablet, laptop, client 110 in fixed or portable form, wearable computing device, or any other suitable device. In one embodiment, device 132 sends a request to server 106 to perform one or more data processing tasks via stream processing application 105B, such as initiating the process described herein. Figure 1 Any software application running in another conventional data processing system can be configured to run in device 132 in a similar manner. Figure 1 Any data or information stored or generated in another conventional data processing system can be configured to be stored or generated in device 132 in a similar manner.

[0055] Server 106, storage unit 108, data processing system 104, and clients 110, 112, and 114, as well as device 132, can be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connections. Clients 110, 112, and 114 can be, for example, personal computers or network computers.

[0056] In the example shown, server 106 can provide clients 110, 112, and 114 with data such as boot files, operating system images, and applications. In this example, clients 110, 112, and 114 can be clients of server 106. Clients 110, 112, 114, or some combination thereof, can include their own data, boot files, operating system images, and applications. Data processing environment 100 can include additional servers, clients, and other devices not shown.

[0057] In the illustrated example, memory 124 can provide data, such as boot files, operating system images, and applications, to processor 122. Processor 122 may include its own data, boot files, operating system images, and applications. Data processing environment 100 may include additional memory, processors, and other devices not shown.

[0058] In the example shown, data processing environment 100 can be the Internet. Network 102 can represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. The core of the Internet is the backbone of data communication links between major nodes or hosts, including thousands of commercial, government, educational, and other computer systems routing data and messages. Of course, data processing environment 100 can also be implemented as various different types of networks, such as intranets, local area networks (LANs), or wide area networks (WANs). Figure 1 This is intended as an example, not as an architectural limitation on different illustrative embodiments.

[0059] In other applications, the data processing environment 100 can be used to implement a client-server environment in which illustrative embodiments can be implemented. The client-server environment enables software applications and data to be distributed across a network, allowing applications to run by interacting with a conventional client data processing system and a conventional server data processing system. The data processing environment 100 can also employ a service-oriented architecture, where interoperable software components distributed across the network can be packaged together as a coherent business application. The data processing environment 100 can also be cloud-based, employing a service-delivered cloud computing model to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and deployed with minimal management effort or interaction with service providers.

[0060] refer to Figure 2The figure illustrates a block diagram of a data processing system in which illustrative embodiments can be implemented. Data processing system 200 is an example of a conventional computer, such as... Figure 1 The data processing system 104, server 106, or client 110, 112, and 114, or another type of device, wherein the computer can implement the process using program code or instructions, can be located for the illustrative embodiment.

[0061] Data processing system 200 also represents a conventional data processing system or a configuration thereof, such as Figure 1 The conventional data processing system 132 in which computer-usable program code or instructions for implementing the processes of the illustrative embodiments can be located. The data processing system 200 is described as a computer only as an example, but is not limited thereto. Implementations in other device forms, such as... Figure 1 The device 132 in the data processing system 200 can modify the data processing system 200, for example by adding a touch interface, or even by removing certain described components from the data processing system 200 without departing from the general description of the operation and functions of the data processing system 200 described herein.

[0062] In the example shown, the data processing system 200 employs a hub architecture, including a Northbridge and Memory Controller Hub (NB / MCH) 202 and a Southbridge and Input / Output (I / O) Controller Hub (SB / ICH) 204. A processing unit 206, main memory 208, and graphics processor 210 are coupled to the Northbridge and Memory Controller Hub (NB / MCH) 202. The processing unit 206 may contain one or more processors and may be implemented using one or more heterogeneous processor systems. The processing unit 206 may be a multi-core processor. In some implementations, the graphics processor 210 may be coupled to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0063] In the example shown, a local area network (LAN) adapter 212 is coupled to the Southbridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and a PCI / PCIe device 234 are coupled to the Southbridge and I / O controller hub 204 via bus 238. A hard disk drive (HDD) or solid-state drive (SSD) 226 and a CD-ROM 230 are coupled to the Southbridge and I / O controller hub 204 via bus 240. The PCI / PCIe device 234 may include, for example, an Ethernet adapter, an add-in card, and a PC card for a laptop. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). Hard disk drive 226 and CD-ROM 230 can use, for example, integrated drive electronics (IDE), Serial Advanced Technology Attachment (SATA) interface, or variants such as external SATA (eSATA) and micro SATA (mSATA). Super I / O (SIO) device 236 can be coupled to the southbridge and I / O controller hub (SB / ICH) 204 via bus 238.

[0064] Memory such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer-available storage devices. Hard disk drives or solid-state drives 226, CD-ROMs 230, and other similar available devices are some examples of computer-available storage devices that include computer-available storage media.

[0065] The operating system runs on the processing unit 206. The operating system coordinates and provides services to... Figure 2 The data processing system 200 controls various components within it. The operating system can be a commercial operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. Object-oriented or other types of programming systems can operate alongside the operating system and provide calls to the operating system from programs or applications executing on the data processing system 200.

[0066] Used in operating systems, object-oriented programming systems, and applications or programs (e.g.) Figure 1The instructions of the application 105 in the process reside on a storage device, such as in the form of code 226A on hard disk drive 226, and may be loaded into at least one of one or more memories, such as main memory 208, for execution by processing unit 206. The processes of the illustrative embodiment may be executed by processing unit 206 using computer-implemented instructions, which may reside in memory, such as main memory 208, read-only memory 224, or in one or more peripheral devices.

[0067] Furthermore, in one scenario, code 226A can be downloaded from a remote system 201B via network 201A, where a similar code 201C is stored on storage device 201D. In another scenario, code 226A can be downloaded to a remote system 201B via network 201A, where the downloaded code 201C is stored on storage device 201D.

[0068] Figure 1-2 The hardware used may vary depending on the implementation. Other internal hardware or peripherals, such as flash memory, equivalent non-volatile memory, or optical disc drives, may be used to supplement or replace them. Figure 1 and Figure 2 The hardware depicted in the illustration. Furthermore, the processes of the illustrative embodiments can be applied to multiprocessor data processing systems.

[0069] In some illustrative examples, the data processing system 200 may be a personal digital assistant (PDA), which is typically configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data files. The bus system may include one or more buses, such as a system bus, I / O bus, and PCI bus. Of course, any type of communication structure or architecture can be used to implement the bus system, providing data transfer between different components or devices connected to the structure or architecture.

[0070] The communication unit may include one or more devices for sending and receiving data, such as a modem or network adapter. The memory may be, for example, main memory 208 or a cache, such as the cache in the northbridge and memory controller hub 202. The processing unit may include one or more processors or CPUs.

[0071] Figure 1 and Figure 2 The examples described herein and the examples above are not intended to imply architectural limitations. For example, the data processing system 200 could take the form of a tablet computer, laptop computer, or telephone device, in addition to being a mobile or wearable device.

[0072] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in a manner similar to that of the data processing system 200, using virtualized representations of some or all of the components depicted in the data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 may be represented as a virtualized instance of all or a certain number of hardware processing units 206 available in the host's processing system data; main memory 208 may be represented as a virtualized instance of all or part of the main memory 208 available in the host's data processing system; and disk 226 may be represented as a virtualized instance of all or part of the disk 226 that may be available in the host's data processing system. In this case, the host data processing system is represented by data processing system 200.

[0073] refer to Figure 3 The figure illustrates a block diagram of an example streaming application environment 300 according to an illustrative embodiment. In one embodiment, the streaming application environment 300 is configured to execute a streaming application, which is Figure 1 Examples of applications 105A / 105B.

[0074] In the illustrated embodiment, the streaming application environment 300 includes a management system 305 and multiple compute nodes 310A-310D (generally referred to herein as node 310) – i.e., hosts – which are communicatively coupled to each other using one or more communication networks 320. In one embodiment, the communication network 320 includes one or more servers, networks, or databases and uses a communication protocol to transfer data between the compute nodes 310. In one embodiment, a database system 315 containing a database 340 is communicatively coupled to the network 320 to communicate with the compute nodes 310. In a particular embodiment, the database system 315 is a system with… Figure 1 The example of database 109 on server 106, and nodes 310A-310D are... Figure 1 Examples of data processing system 104, clients 110, 112 and 114, and device 132.

[0075] In the illustrated embodiment, one or more compute nodes 310A-310D are configured as shown for compute node 310D, such that each compute node 310A-310D includes one or more PEs 312, and each PE includes one or more operators 314. Operators 314 are fundamental building blocks of streaming applications, allowing the streaming application to become a distributed computing application. In some embodiments, compute nodes 310A-310D each run a Linux operating system, and each PE 312 represents a Linux process, with operators 314 running one or more aspects of the streaming application within each Linux process, such that the streaming application is distributed among operators 314, PEs 312, and compute nodes 310A-310D.

[0076] In some embodiments, the management system 305 includes an operator graph 332 and a stream manager 334. In some embodiments, developers using the stream computing service write stream applications by defining the operator graph 332, which are then compiled and submitted to the management system 305.

[0077] The flow manager 334 places the operators in the operator diagram 332 on one or more PEs on one or more compute nodes, which may include one or more compute nodes 310A-310D. In some embodiments, the operator diagram 332 includes a description of the flow network topology, such as the location, arrangement, connection, and function of various operators on nodes 310 of the flow application environment 300.

[0078] A PE, such as PE 312, hosts one or more operators, such as operator 314, that operate on input tuples and produce output tuples. PEs have independent processing units and run on nodes, such as any of compute nodes 310A-310D. In streaming applications, PEs 312 are interconnected, allowing data to flow from one PE to the next. Data flows from one PE to another in the form of tuples, each tuple having a sequence of one or more attributes associated with an entity.

[0079] In some embodiments, the stream manager 334 monitors the degradation of the PE 312 and replaces the degraded PE with a better-performing copy of itself before the degraded PE fails. Streaming applications process potentially massive amounts of data in real time. If a streaming application "falls behind" in its processing, the analysis it is performing may lose its value, i.e., the analysis results are based on past data rather than recent data. An application may fall behind simply because a part of the application is performing poorly, such as a single PE. Therefore, as described herein, detecting and replacing a degraded PE before it fails provides a significant improvement in the performance and efficiency of streaming applications.

[0080] refer to Figure 4 The figure depicts a block diagram of an example computing node 400 according to an illustrative embodiment. In one embodiment, computing node 400 is Figure 3 Examples of compute nodes 310A-310D.

[0081] In the illustrated embodiment, computing node 400 includes one or more processors (CPUs) 402, a network interface 408, an interconnect 410, a memory 412, and a storage device 414. Computing node 400 may also include an I / O device interface 404 for connecting I / O devices 406 (e.g., keyboard, display, and mouse devices) to computing node 400. In some embodiments, the functionality described herein is distributed across multiple systems, which may include combinations of software and / or hardware-based systems, such as application-specific integrated circuits (ASICs), computer programs, or smartphone applications.

[0082] In some embodiments, each CPU 402 retrieves and executes programming instructions stored in memory 412 or storage device 414. Similarly, in some embodiments, CPU 402 stores and retrieves application data residing in memory 412. In some embodiments, interconnect 410 transmits programming instructions and application data between each CPU 402, I / O device interface 404, storage device 414, network interface 408, and memory 412.

[0083] In some embodiments, one or more PEs 416 are stored in memory 412. In some such embodiments, PE 416 includes one or more stream operators 418. In some embodiments, PE 416 is allocated to be executed by only one CPU 402, although in other embodiments, the stream operators 418 of PE 416 include one or more threads executing on two or more CPUs 402. Memory 412 is typically included to represent random access memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or flash memory. Storage device 414 includes buffer 422 and is typically included to represent non-volatile memory, such as hard disk drives, solid-state devices (SSDs) or removable memory cards, optical storage, flash memory devices, network-attached storage (NAS), or devices connected to storage area network (SAN) devices, or other devices that may store non-volatile data.

[0084] In some embodiments, a streaming application includes one or more streaming operators 418, which are compiled into PEs 416. In some embodiments, memory 412 includes two or more PEs 416, each PE having one or more streaming operators 418. In some embodiments, each streaming operator 418 includes a portion of code that processes tuples flowing into a PE and outputs tuples to other streaming operators 418 in the streaming computing application, either the same PE, other PEs, or both. In some embodiments, PEs 416 pass tuples to other PEs on the same compute node 400 or on other compute nodes accessible via a communication network. For example, in one embodiment, a first PE 416 on a first compute node 400 outputs tuples to a second PE 416 on a second compute node 400.

[0085] In some embodiments, PE 416 includes a performance monitor 420. In some embodiments, the performance monitor 420 monitors for degradation of PE 416 and issues a notification signal or data when degradation of PE 416 is detected. For example, in some embodiments, when the performance monitor 420 detects a performance degradation of PE 416, the performance monitor 420 sends a notification signal or data to a flow manager (e.g., Figure 3 The stream manager (334) sends a signal.

[0086] In some embodiments, the performance monitor 420 detects degradation by monitoring whether conditions used as degradation indicators occur on PE 416, which may vary from PE 416 to PE 416. In some embodiments, the threshold for the degradation indicator condition may vary from PE 416 to PE 416. Therefore, in some such embodiments, the degradation indicator condition is configurable for each PE 416. In some embodiments, the degradation indicator condition includes execution conditions associated with PE 416 that deviate from a predefined or expected pattern. For example, in some embodiments, the degradation indicator includes an abnormal number of queued tuples (i.e., tuples waiting to be processed at the input ports of operators in the PE), abnormal memory usage, the number and type of abnormal processing, and abnormal CPU consumption. In some embodiments, the performance monitor 420 monitors PE 416 and maintains statistics representing the normal or expected performance of PE 416. In some embodiments, the performance monitor 420 records the utilization of such computing resources by PE 416, allowing the establishment of normal usage patterns and also allowing the detection of abnormal usage patterns. In some embodiments, an abnormal number of queued tuples includes a higher number of queued tuples than normal. In some embodiments, abnormal memory usage includes abnormally high variations in memory usage. In some embodiments, abnormal CPU consumption includes abnormally high CPU consumption change rates.

[0087] refer to Figure 5The figure illustrates a block diagram of an example management system 500 according to an illustrative embodiment. In one embodiment, the management system 500 is... Figure 3 An example of a management system 305.

[0088] In the illustrated embodiment, the management system 500 includes one or more processors (CPUs) 502, a network interface 508, an interconnect 510, a memory 512, and a storage device 514. The management system 500 may also include an I / O device interface 504 to connect I / O devices 506 (e.g., keyboard, display, and mouse devices) to the management system 500. In some embodiments, the functionality described herein is distributed across multiple systems, which may include software and / or hardware-based systems such as application-specific integrated circuits (ASICs), computer programs, or smartphone applications.

[0089] In some embodiments, each CPU 502 retrieves and executes programming instructions stored in memory 512 or storage device 514. Similarly, in some embodiments, each CPU 502 stores and retrieves application data residing in memory 512 or storage device 514. Interconnect 510 is used to move data, such as programming instructions and application data, between CPU 502, I / O device interface 504, memory unit 514, network interface 508, and memory 512. Memory 514 is typically included to represent a connection between non-volatile memory (e.g., hard disk drive, solid-state drive (SSD), removable memory card, optical storage, flash memory device, network attached storage (NAS)) and storage area network (SAN) devices or the cloud. In some embodiments, network interface 508 is configured to transmit data via a communication network.

[0090] In some embodiments, memory 512 stores flow manager 516. Furthermore, in some such embodiments, memory 514 stores operator graph 522, which defines how tuples are routed to PEs for processing. In the illustrated embodiment, flow manager 516 also includes a temporary PE analysis module 518 and a scheduler 524. In the illustrated embodiment, temporary PE analysis module 518 includes performance monitor 520.

[0091] In some embodiments, the analysis module 518 monitors the degradation of the PE and replaces the degraded PE with a copy of itself that has better performance before the degraded PE fails. Streaming applications process potentially massive amounts of data in real time. If a streaming application "falls behind" in its processing, the analysis it is performing may lose its value, i.e., the analysis results are based on past data rather than recent data. An application may fall behind simply because a part of the application is performing poorly, such as a single PE. Therefore, as described herein, detecting and replacing degraded PEs before they fail provides a significant improvement in the performance and efficiency of streaming applications.

[0092] In some embodiments, performance monitor 520 monitors the streaming application and maintains statistics representing the performance of the various PEs that constitute the streaming application. In some embodiments, performance monitor 520 is part of the stream manager 516 as shown. In other embodiments, performance monitor 520 is invoked at runtime, but may be a separate entity from the runtime stream manager 516.

[0093] In some embodiments, performance monitor 520 monitors for PE degradation and issues notification signals or data when degradation of one or more PEs is detected. For example, in some embodiments, when performance monitor 520 detects performance degradation of a PE, performance monitor 520 sends a signal to stream manager 516.

[0094] In some embodiments, the performance monitor 520 detects degradation by monitoring whether a PE 416 exhibits conditions that serve as a degradation metric, which may vary from PE 416 to PE 416. In some embodiments, the threshold for the degradation metric condition may vary from PE 416 to PE 416. Therefore, in some such embodiments, the degradation metric condition is configurable for each PE 416. In some embodiments, the degradation metric condition includes execution conditions associated with a PE that deviates from a predefined or expected pattern. For example, in some embodiments, the degradation metric includes an abnormal number of queued tuples (i.e., tuples waiting to be processed on the input ports of operators in the PE), abnormal memory usage, the number and type of abnormal processing, and abnormal CPU consumption. In some embodiments, the performance monitor 520 monitors the PE and maintains statistics representing the normal or expected performance of the PE. In some embodiments, the performance monitor 520 records the PE's utilization of such computing resources, allowing the establishment of normal usage patterns and also allowing the detection of abnormal usage patterns. In some embodiments, an abnormal number of queued tuples includes a higher number of queued tuples than normal. In some embodiments, abnormal memory usage includes an abnormally high rate of change in memory usage. In some embodiments, abnormal CPU consumption includes an abnormally high rate of change in CPU consumption.

[0095] In some embodiments, analysis module 518 analyzes PEs identified by performance monitor 520 as exhibiting one or more degradation indicators. In some embodiments, analysis module 518 analyzes the PEs to determine whether the indicators identified by performance monitor 520 are actually indicators of problems other than PE degradation, such as backpressure or one or more cluster computing resources reaching capacity. In some such embodiments, if analysis module 518 does not detect problems other than PE degradation, analysis module 518 then evaluates the PEs using temporary PEs. In some such embodiments, analysis module 518 uses scheduler 524 to identify nodes hosting temporary PEs. In some embodiments, scheduler 524 searches for nodes other than those hosting PEs. In some embodiments, scheduler 524 also searches for nodes based on predetermined rules or criteria, such as the availability of computing resources and proximity to upstream and downstream PEs.

[0096] In some embodiments, once the scheduler 524 identifies the host node, the analysis module 518 initiates a temporary PE on the identified node at runtime, while maintaining the runtime operation of the PE. In some such embodiments, the temporary PE is connected to the same upstream PE as the PE in analysis, such that both the temporary PE and the PE in analysis simultaneously receive the same tuple stream from the same upstream PE. In some such embodiments, both the temporary PE and the PE in analysis perform the same definition procedure on the tuple stream. In some such embodiments, the PE in analysis performs the definition procedure on the tuple stream, producing an output tuple stream that the PE transmits to downstream PEs, while the output of the temporary PE is not connected to any downstream element.

[0097] In some embodiments, the analysis module 518 compares the operation of the analyzed PE with the operation of the temporary PE based on performance metrics associated with one or more computing resources. In some such embodiments, if the analysis module 518 determines that the temporary PE performs better than the analyzed PE, the analysis module 518 replaces the analyzed PE with the temporary PE, for example by connecting the output of the temporary PE to a downstream PE instead of the analyzed PE, and then shuts it down.

[0098] refer to Figure 6 The figure illustrates a block diagram of an exemplary temporary PE analysis module 600 according to an illustrative embodiment. In one embodiment, the temporary PE analysis module 600 is... Figure 5 Example of temporary PE analysis module 518.

[0099] In the illustrated embodiment, the temporary PE analysis module 600 includes a performance monitor 602, an initial analysis module 604, and a temporary troubleshooter 606. In the illustrated embodiment, the temporary troubleshooter 606 includes a temporary PE generator 610, a performance comparator 612, and a system updater 614. In some embodiments, the functionality described herein is distributed across multiple systems, which may include a combination of software- and / or hardware-based systems, such as application-specific integrated circuits (ASICs), computer programs, or smartphone applications.

[0100] In the illustrated embodiment, performance monitor 602 monitors the streaming application and maintains statistics representing the performance of the various physical components (PEs) constituting the streaming application. In some embodiments, performance monitor 602 monitors for PE degradation and issues notification signals or data when degradation of one or more PEs is detected. For example, in some embodiments, when performance monitor 602 detects performance degradation of a PE, performance monitor 602 sends a signal to initial analysis module 604, such as... Figure 6 The PE 608 shown here is experiencing performance degradation.

[0101] In some embodiments, the performance monitor 602 detects degradation of the performance-degraded PE 608 by monitoring whether a condition used as a degradation metric occurs, which may vary from PE to PE. In some embodiments, the threshold for the degradation metric condition may also vary from PE to PE. Therefore, in some such embodiments, the degradation metric condition is configurable for each PE. In some embodiments, the degradation metric condition includes execution conditions associated with a PE that deviates from a predefined or expected pattern. For example, in some embodiments, the degradation metric includes an abnormal number of queued tuples (i.e., tuples waiting to be processed on the input ports of operators in the PE), abnormal memory usage, the number and type of exception handling, and abnormal CPU consumption. In some embodiments, the performance monitor 602 monitors multiple PEs, including the performance-degraded PE 608. In alternative embodiments, the performance monitor 602 monitors only the performance-degraded PE 608. In some embodiments, the performance monitor 602 maintains statistics indicating that the performance of the performance-degraded PE 608 was normal or expected prior to its degradation. In some embodiments, performance monitor 602 records the usage of such computing resources by performance-degraded PEs 608, allowing the establishment of normal usage patterns and thus also allowing for the detection of anomalous usage patterns. In some embodiments, an anomalous number of queued tuples includes a higher number of queued tuples than normal. In some embodiments, anomalous memory usage includes an abnormally high rate of change in memory usage. In some embodiments, anomalous CPU consumption includes an abnormally high rate of change in CPU consumption.

[0102] In some embodiments, the initial analysis module 604 analyzes the degraded performance PE 608 when it is identified by the performance monitor 602 as exhibiting one or more degradation indicators. In some embodiments, the initial analysis module 604 analyzes the performance-degraded PE 608 to determine whether the indication identified by the performance monitor 602 is actually an indication of a problem rather than a degradation of the performance-degraded PE 608, such as backpressure or one or more cluster computing resources reaching capacity.

[0103] In some such embodiments, if the initial analysis module 604 does not detect any problems other than the degradation of the PE 608 with degraded performance, the initial analysis module 604 notifies the temporary troubleshooter 606, which then uses a temporary PE 616 to evaluate the PE 608 with degraded performance. In some such embodiments, the temporary troubleshooter 606 works with a scheduler (e.g., Figure 5 The scheduler 524 communicates with the host PE 616 to identify the node hosting the temporary PE 616. In some embodiments, the scheduler searches for nodes other than the host PE. In some embodiments, the scheduler also searches for nodes based on predetermined rules or criteria, such as the availability of computing resources and proximity to upstream and downstream PEs. In some embodiments, once the scheduler identifies the host node, it notifies the temporary PE generator 610. The temporary PE generator 610 starts the temporary PE 616 on the identified node at runtime, while maintaining the runtime operation of the performance-degraded PE 608. In some such embodiments, the temporary PE generator 610 connects the upstream side of the temporary PE 616 to the same upstream PE as the performance-degraded PE 608, such that both the temporary PE 616 and the performance-degraded PE 608 simultaneously receive the same tuple stream from the same upstream PE. In some such embodiments, both the temporary PE 616 and the performance-degraded PE 608 perform the same definition process on the tuple stream, which allows the performance of the performance-degraded PE 608 and the temporary PE 616 to be compared by the performance comparator 612. In some such embodiments, the performance-degraded PE 608 performs a defined process on the tuple stream to produce a tuple output stream, which the performance-degraded PE 608 transmits to the downstream PE, while the output of the temporary PE 616 is not connected to any downstream component.

[0104] In some embodiments, performance comparator 612 compares the operation of a degraded PE 608 with the operation of a temporary PE 616 based on performance metrics associated with one or more computing resources. For example, in some embodiments, performance comparator 612 compares the operation of the degraded PE 608 with the operation of the temporary PE 616 by comparing one or more of CPU utilization, memory utilization, and tuple throughput rate. In some such embodiments, if performance comparator 612 determines that the temporary PE 616 performs better than the degraded PE 608, performance comparator 612 notifies system updater 614. In response to the notification from performance comparator 612 that the temporary PE 616 performs better than the degraded PE 608, system updater 614 replaces the degraded PE 608 with the temporary PE 616, for example by connecting the output of the temporary PE 616 to a downstream PE instead of the degraded PE 608, and system updater 614 shuts down the degraded PE 608. In some embodiments, the system updater 614 sends a notification to the user via the user interface 618 regarding the replacement of the performance-degraded PE 608 with a temporary PE 616.

[0105] refer to Figure 7 This figure is a block diagram of an operator diagram 700 for a stream computing application according to an illustrative embodiment. For clarity, Figure 7 The operator diagram shown is presented as a non-limiting example of a streaming application that may benefit from the disclosed embodiments.

[0106] In the illustrated embodiment, operator graph 700 is a graph of a stream computing application starting from one or more sources 710 to one or more sinks 712, 714. The stream from source to sink may also be collectively referred to herein as an execution path. Although Figure 7 Abstracted to show multiple connected PEs PE1-PE10, but operator diagram 700 may include flow operators within the same or different PEs (e.g., Figure 3 Data flow between operators (314). Typically, the PE receives tuples from the stream and outputs tuples back to the stream (except where the sink—the end of the stream, or the source—the beginning of the stream)).

[0107] In the illustrated embodiment, operator graph 700 includes ten process instances (PEs) (labeled PE1-PE10) running on compute nodes 702, 704, 706, and 708. In some embodiments, a PE includes one or more merged stream operators to form an independently running process with its own process ID (PID) and memory space. In embodiments where two (or more) PEs run independently, inter-process communication can occur using "transfers" (e.g., network sockets, TCP / IP sockets, or shared memory). However, when stream operators are merged, the merged stream operators can use faster communication techniques to pass tuples between the stream operators in each PE.

[0108] Operator graph 700 begins at source 710 and ends at sinks 712 and 714. Compute node 702 includes multiple PEs (PE1, PE2, and PE3). Source 710 flows into PE PE1, which in turn outputs tuples received by PE2 and PE3. For example, PE1 can split a received data attribute into a tuple, pass part of the data attribute as a new tuple to PE2, and pass the remaining data attributes as another new tuple to PE3. As a second example, PE1 can pass some received tuples to PE2 while passing the rest to PE3. Tuples flowing to PE2 are processed by the stream operators contained in PE2, and the resulting tuples are then output to PE4 on compute node 704. Similarly, tuples output by PE4 flow to the operator sink PE6 712. Likewise, tuples flowing from PE3 to PE5 also reach the operators in sink PE6 712. Therefore, in addition to serving as the sink in this example operator graph, PE6 can be configured to perform join operations, combining tuples received from PE4 and PE5. This example operator graph also shows tuples flowing from PE3 to PE7 on compute node 706, which itself shows tuples flowing to PE8 and looping back to PE7. Tuples output from PE8 flow to PE9 on compute node 708, which in turn outputs tuples for operator processing in sink PEs (e.g., PE10 714).

[0109] In some embodiments, tuples received by a particular PE are generally not considered the same tuples as downstream outputs because the output tuples have been changed in some way. For example, in some embodiments, attributes or metadata are added, deleted, or altered. However, some embodiments include PEs that do not change the output tuples from the input tuples in any way. Typically, a particular tuple output by a PE may not be considered the same tuple as the corresponding input tuple, even if the input tuples have not been changed by the PE. However, for the sake of simplicity in this description and claims, an output tuple having the same data attributes as the corresponding input tuple may be referred to herein as the same tuple. In the illustrated embodiments, a performance monitor as described herein is configured to monitor the degradation performance of one or more PEs PE1-PE10.

[0110] refer to Figure 8 This figure shows a block diagram of operator diagram 800 for a stream computing application with a temporary PE according to an illustrative embodiment. For clarity, Figure 8 The operator diagram shown is as follows Figure 7 A non-limiting example of a modified version of the operator diagram 700, which begins at source 810 and ends at the embodiments disclosed in sinks 812, 814 that may benefit from it.

[0111] In the example shown, the performance monitor (e.g., Figure 5 Performance Monitor 520 or Figure 6 The performance monitor 602 detects degradation of PE2 by monitoring PE2 and detecting deviations in execution conditions associated with PE2 from a predefined or expected pattern. For example, in some embodiments, the performance monitor detects degradation metrics including an unusual number of queued tuples (i.e., tuples waiting to be processed at the input ports of operators in the PE), unusual memory usage, the number and type of exceptions processed, and / or unusual CPU consumption. In the example shown, no issues other than degradation were detected, therefore the temporary PE ( Figure 8 The PPE specified in the code is started on different nodes: Since PE2 is on node 802, PPE is on node 804, but may or may not have been started on node 806 or node 808.

[0112] The temporary PE starts at runtime while maintaining the runtime operation of PE2. Initially, to compare the performance of the temporary PE and PE2, the temporary PE is connected to PE1 only on its upstream side via edge 816, so that the temporary PE and PE2 simultaneously receive the same tuple stream from the same upstream PE1, and both perform the same definition process on the tuple stream. This allows the performance of the temporary PE to be compared with that of PE2.

[0113] In the illustrated example, a performance comparator or flow manager compares the operation of the temporary PE with that of PE2 based on performance metrics associated with one or more computing resources. For example, in some embodiments, the temporary PE and PE2 are compared based on one or more of CPU utilization, memory utilization, and / or tuple throughput rate. In some such embodiments, if the temporary PE performs better than PE2, the temporary PE replaces PE2, for example by replacing PE2 with its output / downstream side connected to PE4, downstream of PE2, for example by connecting edge 818 from the temporary PE to PE4 and shutting down PE2.

[0114] refer to Figure 9 The figure illustrates a flowchart of an example dynamic PE replacement process 900 according to an illustrative embodiment. In one specific embodiment, Figure 6 The temporary PE analysis module 600 in the process 900 is executed.

[0115] In one embodiment, at box 902, process 900 initiates the streaming application. Next, at box 904, process 900 monitors for performance degradation of processing elements. Next, at box 906, process 900 checks if performance degradation is detected at any processing element. If not, process 900 returns to box 904 to continue monitoring. Otherwise, process 900 proceeds to box 908, where process 900 begins evaluating the degraded PE. At box 908, process 900 checks if there is an indication that overpressure is the cause of performance degradation in the degraded PE, followed by box 912, where process 900 checks the run level of the node cluster's compute resources, where the degraded PE has been deployed. If process 900 detects that overpressure or a lack of available or sufficient cluster compute resources may be the cause of the degraded PE's degradation, the process continues to box 910, where the process applies other remedies known in the art to address the overpressure and / or lack of cluster compute resources. Otherwise, the process continues to box 914, where process 900 performs a temporary PE troubleshooting procedure, such as according to... Figure 10 Process 1000 Figure 11 Process 1100 Figure 12 Process 1200, or a process combining two or more parts of processes 1000, 1100 and / or 1200.

[0116] Following the temporary PE troubleshooting process in box 914, process 900 returns to box 904 to continue monitoring the degradation performance of the processed element. In the illustrated embodiment, monitoring continues until another degraded PE is detected or until the streaming application stops, as shown in box 916, after which process 900 terminates.

[0117] refer to Figure 10The figure illustrates a flowchart of an example dynamic PE replacement process 1000 according to an illustrative embodiment. In one specific embodiment, Figure 6 The temporary PE analysis module 600 in the middle executes process 1000. In one embodiment, process 1000 is Figure 9 Example of a temporary PE troubleshooting process in box 914.

[0118] In the illustrated embodiment, process 1000 tests the use of temporary processing elements as possible alternatives to the degraded PE. At block 1002, process 1000 determines the location of the temporary PE. In some embodiments, process 1000 uses a scheduler to identify nodes on which the temporary PE is deployed. In some such embodiments, the scheduler determines the optimal node for running the temporary PE. In some such embodiments, when the optimal node for running the temporary PE is determined, the scheduler excludes nodes running the degraded PE.

[0119] Next, in block 1004, process 1000 initiates a temporary PE on the node selected in block 1002. In some embodiments, the temporary PE is initiated at runtime while the degenerate PE remains operational at runtime. In some embodiments, the temporary PE is connected only on its upstream side and is connected to the next upstream PE immediately following the degenerate PE. This connection causes the temporary PE and the degenerate PE to simultaneously receive the same tuple stream from the same upstream PE.

[0120] Next, in box 1006, process 1000 compares the performance of the degraded PE and the temporary PE. In some such embodiments, the temporary PE performs the same defined processing on the tuple stream as the degraded PE, which allows for a performance-based comparison of the temporary and degraded PEs. In some embodiments, a performance comparator or flow manager compares the operation of the temporary PE with that of the degraded PE based on performance metrics associated with one or more computing resources. For example, in some embodiments, the temporary PE and the degraded PE are compared based on one or more of CPU utilization, memory utilization, and / or tuple throughput rate.

[0121] Next, in block 1008, process 1000 determines whether the performance of the temporary PE is superior to that of the degraded PE. In some such embodiments, if the temporary PE performs better than the degraded PE, then in block 1010, process 1000 performs a switch to the temporary PE as a replacement for the degraded PE, and the degraded PE is shut down. In some embodiments, the switch includes connecting the output / downstream side of the temporary PE to the next direct downstream PE of the degraded PE.

[0122] refer to Figure 11 The figure illustrates a flowchart of an example dynamic PE replacement process 1100 according to an illustrative embodiment. In one specific embodiment, Figure 6The temporary PE analysis module 600 in the middle executes process 1100. In one embodiment, process 1100 is Figure 9 Example of a temporary PE troubleshooting process in box 914.

[0123] In the illustrated embodiment, process 1100 tests the use of multiple temporary processing elements as possible alternatives to the degraded PE. In some embodiments, the number of temporary PEs is a fixed, predetermined value. In some embodiments, the number of temporary PEs is a user-configurable value, initially set to a default value and adjustable by user input. In some embodiments, process 1100 determines the number of temporary PEs to use based on any number of implementation-specific factors, such as the type of degraded PE being replicated (i.e., the operation performed on the degraded PE by one or more operators), the size of the streaming application, the number of available nodes already used by the streaming application, and / or any other desired factors.

[0124] In block 1102, process 1100 determines the location of each of a plurality of temporary PEs. In some embodiments, process 1100 uses a scheduler to identify nodes on which temporary PEs should be deployed. In some such embodiments, the scheduler determines a list of optimal nodes to run temporary PEs. In some embodiments, the scheduler also determines the number of temporary PEs to be deployed, for example based on the availability of nodes on which temporary PEs should be deployed. In some such embodiments, the scheduler excludes nodes running degraded PEs when determining the optimal nodes to run temporary PEs.

[0125] Next, in block 1104, process 1100 initiates a temporary PE on the appropriate node selected in block 1102. In some embodiments, the temporary PE is initiated at runtime while the degenerate PE remains operational at runtime. In some embodiments, the temporary PEs are connected only on their upstream side and are all connected to the next upstream PE immediately following the degenerate PE. This connection results in the set of degenerate PEs and temporary PEs simultaneously receiving the same tuple stream from the same upstream PE.

[0126] Next, in block 1106, process 1100 compares the performance of the degraded PE and the temporary PE. In some such embodiments, each temporary PE performs the same defined processing on the tuple stream as the degraded PE, which allows for a performance-based comparison between the degraded PE and the temporary PE. In some embodiments, a performance comparator or flow manager compares the operation of the degraded PE with that of the temporary PE based on performance metrics associated with one or more computing resources. For example, in some embodiments, the temporary PE and the degraded PE are compared based on one or more of CPU utilization, memory utilization, and / or tuple throughput rate.

[0127] Next, at block 1108, process 1100 determines whether the performance of any temporary PE is superior to that of the degraded PE. In some such embodiments, if the performance of any temporary PE is superior to that of the degraded PE, then at block 1110, process 1100 performs a switch to the best-performing temporary PE as a replacement for the degraded PE, and the degraded PE and the remaining temporary PEs are shut down. In some embodiments, the switch includes connecting the output / downstream side of the temporary PE to the next direct downstream PE of the degraded PE.

[0128] refer to Figure 12 The figure illustrates a flowchart of an example dynamic PE replacement process 1200 according to an illustrative embodiment. In one specific embodiment, Figure 6 The temporary PE analysis module 600 in the middle executes process 1200. In one embodiment, process 1200 is Figure 9 Example of a temporary PE troubleshooting process in box 914.

[0129] In the illustrated embodiment, process 1200 tests the use of temporary processing elements as possible alternatives to degraded PEs. At block 1202, process 1200 determines the location of the temporary PE. In some embodiments, process 1200 uses a scheduler to identify nodes on which to deploy the temporary PE. In some such embodiments, the scheduler determines a list of optimal nodes to run the temporary PE. In some embodiments, the scheduler also determines the number of temporary PEs to deploy, for example, based on the availability of nodes on which temporary PEs are deployed. In some such embodiments, the scheduler excludes nodes running degraded PEs when determining the optimal nodes to run temporary PEs.

[0130] Next, in block 1204, process 1200 initiates a temporary PE on the node selected in block 1202. In some embodiments, the temporary PE is initiated at runtime while the degenerate PE remains operational at runtime. In some embodiments, the temporary PE is connected only on its upstream side and is connected to the next upstream PE immediately following the degenerate PE. This connection causes the temporary PE and the degenerate PE to simultaneously receive the same tuple stream from the same upstream PE.

[0131] Next, in box 1206, process 1200 compares the performance of the degraded PE and the temporary PE. In some such embodiments, the temporary PE performs the same defined processing on the tuple stream as the degraded PE, which allows for a performance-based comparison of the temporary and degraded PEs. In some embodiments, a performance comparator or flow manager compares the operation of the temporary PE with that of the degraded PE based on performance metrics associated with one or more computing resources. For example, in some embodiments, the temporary PE and the degraded PE are compared based on one or more of CPU utilization, memory utilization, and / or tuple throughput rate.

[0132] Next, in block 1208, process 1200 determines whether the performance of the temporary PE is better than that of the degraded PE. In some such embodiments, if the temporary PE does not perform better than the degraded PE, then in block 1212, process 1200 determines whether the maximum number of temporary PEs has been tried. If yes, the process ends. If not, the process returns to block 1202 to perform the next iteration of blocks 1202-1208 at a different location using another temporary PE (e.g., deployed on a node different from the node where the degraded PE was deployed, and also different from the node used for the previous iterations of blocks 1202-1208).

[0133] In some embodiments, the maximum number of temporary PEs at block 1212 is a fixed, predetermined value. In some embodiments, the maximum number of temporary PEs at block 1212 is a user-configurable value that is initially set to a default value and can be adjusted by user input. In some embodiments, process 1100 determines the maximum number of temporary PEs at block 1212 based on any number of implementation-specific factors, such as the type of degenerate PE being replicated (i.e., the operation performed on the degenerate PE by one or more operators), the size of the streaming application, the number of available nodes already used by the streaming application, and / or any other desired factors.

[0134] In some such embodiments, if any iteration of blocks 1202-1208 results in the temporary PE performing better than the degraded PE, then in block 1210, process 1200 performs a switch to the temporary PE as an alternative to the degraded PE and shuts down the degraded PE. In some embodiments, the switch includes connecting the output / downstream side of the temporary PE to the next direct downstream PE of the degraded PE.

[0135] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains,” or “containing,” or any other variation thereof, are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus comprising a list of elements is not necessarily limited to those elements, but may include other elements, articles, or apparatus not expressly listed or inherent to such compositions, mixtures, processes, or methods.

[0136] Furthermore, the term "illustrative" is used herein to mean "used as an example, instance, or illustration." Any embodiment or design described herein as "illustrative" is not necessarily to be construed as superior to or better than other embodiments or designs. The terms "at least one" and "one or more" should be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" should be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0137] References to "one embodiment," "an embodiment," "an example embodiment," etc., in the specification indicate that the described embodiment may include a particular feature, structure, or characteristic; however, each embodiment may or may not include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, it is assumed that its influence on such feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.

[0138] The terms “about,” “substantially,” “approximately,” and their variations are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of application submission. For example, “about” could include a range of ±8%, 5%, or 2% of a given value.

[0139] Various embodiments of the present invention have been described for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to techniques found in the market, or to enable others skilled in the art to understand the embodiments described herein.

[0140] Therefore, computer-implemented methods, systems, or apparatuses, as well as computer program products, are provided in the illustrative embodiments for managing participation in online communities and other related features, functions, or operations. Where embodiments or portions thereof are described with respect to a type of device, the computer-implemented methods, systems, or apparatuses, computer program products, or portions thereof are adapted or configured for use with a suitable and comparable manifestation of that device type.

[0141] Where the embodiments are described as being implemented within an application, the delivery of the application in a Software as a Service (SaaS) model is contemplated within the scope of the illustrative embodiments. In the SaaS model, the capabilities of the application implementing the embodiments are provided to the user by executing the application within a cloud infrastructure. The user can access the application using various client devices through a thin client interface such as a web browser (e.g., web-based email) or other lightweight client applications. The user does not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage of the cloud infrastructure. In some cases, the user may not even be able to manage or control the functionality of the SaaS application. In some other cases, the SaaS implementation of the application may allow for limited user-specific application configuration settings, with exceptions possible.

[0142] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0143] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0145] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0146] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0149] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in a flowchart or block diagram may represent a portion of a module, segment, or instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the boxes may occur in a non-consecutive order. For example, two boxes shown consecutively may actually execute substantially simultaneously, or these boxes may sometimes execute in reverse order, depending on the functions involved. It should also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs a specific function or action, or by a combination of dedicated hardware and computer instructions.

[0150] Embodiments of the present invention may also be delivered as part of a service agreement with client companies, non-profit organizations, government entities, internal organizational structures, etc. Aspects of these embodiments may include configuring computer systems to execute and deploy software, hardware, and network services to implement some or all of the methods described herein. Aspects of these embodiments may also include analyzing client operations, creating recommendations in response to the analysis, building systems to implement parts of the recommendations, integrating the systems into existing processes and infrastructure, metering system usage, allocating fees to users of the system, and billing for system usage. Although the above embodiments of the invention have been described separately by stating their respective advantages, the invention is not limited to their specific combinations. Rather, these embodiments may be combined in any manner and number according to the intended deployment of the invention without losing their beneficial effects.

Claims

1. A computer-implemented method, comprising: During runtime, monitor the occurrence of degradation conditions on processing elements PE in a distributed computing application, where the PE operating on the first node processes the first tuple stream from the upstream PE and transmits the second tuple stream to the downstream PE. The first temporary PE on the second node processes the first tuple stream from the upstream PE, while maintaining the runtime operation of the PE operating on the first node; as well as The first temporary PE is connected to replace the second tuple flow from the PE operating on the first node to the downstream PE with a third tuple flow from the first temporary PE to the downstream PE.

2. The computer-implemented method according to claim 1 further includes selecting the second node from a plurality of nodes to execute the first temporary PE.

3. The computer-implemented method of claim 2, wherein the selection of the second node from the plurality of nodes is based at least in part on the availability of computing resources on the second node.

4. The computer-implemented method according to claim 1, wherein the first temporary PE receives the first tuple stream from the upstream PE and performs a definition process on the first tuple stream, while the PE operating on the first node receives the first tuple stream from the upstream PE and performs the definition process on the first tuple stream.

5. The computer-implemented method according to claim 4 further includes starting a second temporary PE on a third node during runtime, while maintaining the runtime operation of the PE operating on the first node. The process in which the second temporary PE receives the first tuple stream from the upstream PE and performs the defined process on the first tuple stream, while the first temporary PE and the PE operating on the first node each receive the first tuple stream from the upstream PE and each perform the defined process on the first tuple stream.

6. The computer-implemented method of claim 5, wherein the connection is in response to comparing the operation of the PE operating on the first node with the operation of the first temporary PE, and also in response to comparing the operation of the second temporary PE with the operation of the PE operating on the first node and the operation of the first temporary PE based on a performance metric associated with computing resources.

7. The computer-implemented method according to claim 1, further comprising: Start the first temporary PE; as well as Before starting the first temporary PE, a second temporary PE is started on the third node during runtime, while maintaining the runtime operations of the PE operating on the first node. The second temporary PE receives the first tuple stream from the upstream PE and performs a definition process on the first tuple stream.

8. The computer-implemented method according to claim 7, further comprising, before starting the first temporary PE: The operations of the PE operating on the first node are compared with the operations of the second temporary PE based on performance metrics associated with computing resources.

9. The computer-implemented method of claim 8, wherein comparing the operation of the PE operating on the first node with the operation of the second temporary PE results in identifying, based on the performance metric, that the operation of the PE operating on the first node exceeds the operation of the second temporary PE.

10. A computer program product comprising program instructions executable by a processor to cause the processor to perform operations including: During runtime, monitor the occurrence of degradation conditions on processing elements PE in a distributed computing application, where the PE operating on the first node processes the first tuple stream from the upstream PE and transmits the second tuple stream to the downstream PE. The first temporary PE on the second node processes the first tuple stream from the upstream PE, while maintaining the runtime operation of the PE operating on the first node; as well as The first temporary PE is connected to replace the second tuple flow from the PE operating on the first node to the downstream PE with a third tuple flow from the first temporary PE to the downstream PE.

11. The computer program product of claim 10, wherein the program instructions are stored in a computer-readable storage device in a data processing system, and wherein the stored program instructions are transmitted from a remote data processing system via a network.

12. The computer program product of claim 10, wherein the program instructions are stored in a computer-readable storage device in a server data processing system, and wherein the stored program instructions are downloaded in response to a request for a remote data processing system via a network for use in a computer-readable storage device associated with the remote data processing system, further comprising: Used to measure the use of program instructions related to the request; as well as The program instructions for generating invoices are used according to the measurement.

13. The computer program product of claim 10, further comprising selecting the second node from a plurality of nodes to execute the first temporary PE.

14. The computer program product of claim 10, wherein the first temporary PE receives the first tuple stream from the upstream PE and performs a definition process on the first tuple stream, while the PE operating on the first node receives the first tuple stream from the upstream PE and performs the definition process on the first tuple stream.

15. The computer program product of claim 14, further comprising starting a second temporary PE on a third node during runtime, while maintaining the runtime operation of the PE operating on the first node. The process in which the second temporary PE receives the first tuple stream from the upstream PE and performs the defined process on the first tuple stream, while the first temporary PE and the PE operating on the first node each receive the first tuple stream from the upstream PE and each perform the defined process on the first tuple stream.

16. The computer program product of claim 15, wherein comparing the operation of the PE operating on the first node with the operation of the first temporary PE further comprises comparing the operation of the second temporary PE with the operation of the PE operating on the first node and with the operation of the first temporary PE based on a performance metric associated with computing resources.

17. A computer system comprising a processor and one or more computer-readable storage media, and program instructions commonly stored on the one or more computer-readable storage media, the program instructions being executable by the processor to cause the processor to perform operations, the operations including: During runtime, monitor the occurrence of degradation conditions on processing elements PE in a distributed computing application, where the PE operating on the first node processes the first tuple stream from the upstream PE and transmits the second tuple stream to the downstream PE. The first temporary PE on the second node processes the first tuple stream from the upstream PE, while maintaining the runtime operation of the PE operating on the first node; as well as The first temporary PE is connected to replace the second tuple flow from the PE operating on the first node to the downstream PE with a third tuple flow from the first temporary PE to the downstream PE.

18. The computer system of claim 17, further comprising selecting the second node from a plurality of nodes to execute the first temporary PE.

19. The computer system of claim 17, wherein the first temporary PE receives the first tuple stream from the upstream PE and performs a definition process on the first tuple stream, while the PE operating on the first node receives the first tuple stream from the upstream PE and performs the definition process on the first tuple stream.

20. The computer system of claim 19, further comprising starting a second temporary PE on a third node during runtime, while maintaining the runtime operation of the PE operating on the first node. The process in which the second temporary PE receives the first tuple stream from the upstream PE and performs the defined process on the first tuple stream, while the first temporary PE and the PE operating on the first node each receive the first tuple stream from the upstream PE and each perform the defined process on the first tuple stream.

Images