Rule-based autonomous database cloud service framework
By introducing a rules engine-based asynchronous operation framework and machine learning automatic tuning technology into the database system, the problem that traditional data warehouse clouds are difficult to cope with explosive growth in data demands is solved, and efficient and automated database management and performance optimization are achieved.
Patent Information
- Application Number
- CN201880068978.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-19
- Filing Date
- 2018-09-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-04-15
AI Technical Summary
Traditional data warehouse cloud solutions are difficult to meet the explosive growth of data demands, resulting in increased management complexity, risks and costs, and lack of automated management capabilities, affecting the reliability and performance of database systems.
Adopt the asynchronous job framework based on the rules engine, optimize the database system by automatically submitting asynchronous jobs that reconfigure the database, realizing dynamic topology and load distribution, and using machine learning to automatically tune rules to improve the automation and performance of database management.
It realizes automated management and performance optimization of database systems, reduces the need for manual intervention, improves the reliability, availability and maintainability of data grids, reduces the possibility of errors, and improves management efficiency and performance.
Smart Images

Figure CN111263938B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the management of multi-tenant data grids. A data cloud management technique is presented herein that achieves autonomy by using a rule engine that reacts to database system events by autonomously submitting asynchronous jobs that reconfigure the database. Background Art
[0002] Today’s cutting-edge organizations are differentiating themselves through analytics by extracting value from all data sources, further increasing their competitive advantage. Other companies are looking to become data-driven by modernizing their data management deployments. These strategies include challenges such as the management of growing volumes of data. Today’s digital world is already creating data at an explosive rate, and the next wave is coming with the advent of Internet of Things (IoT) data sources. Physical data warehouses of the past were great for collecting data across the enterprise for analysis, but the storage and computing resources required to support them could not keep up with the explosive growth. In addition, manual and tedious patching, updating, and upgrading tasks can also put data at risk due to human error. To reduce risk, cost, complexity, and time to value, many organizations are taking their data warehouses to the cloud. Whether hosted locally in a private cloud, outsourced to a public cloud service, or a hybrid of the two, the cloud offers simpler management, higher scalability and availability without sacrificing performance, and new ways to reduce data storage and processing costs.
[0003] The velocity and volume of incoming data places demanding demands on traditional data marts, enterprise data warehouses, and analytical systems. Typical data warehouse cloud solutions may not be able to meet such demands. Many customers have proven the value of data warehouses in the cloud through "sandbox" environments, line-of-business data marts, and database backups. More advanced monetization use cases include high-performance data management projects, data warehouses coupled with cloud computing analytics, and big data cloud implementations.
[0004] Production operations staff typically include database administrators (DBAs) and other system administrators who perform database lifecycle management and administration tasks. The hosting relationship between databases and computers is more or less strictly arranged in the network topology, making software upgrades, horizontal scaling, and planned or emergency provisioning / configuration potentially cumbersome and error-prone, especially when the cloud scale is orders of magnitude larger than a simple computer cluster.
[0005] Strict topologies may hinder or prevent horizontal scaling (elasticity) and capacity planning. The management of databases and data centers is often self-organized (ad hoc) and is often based on necessary commands that are expected to be executed almost immediately. Therefore, management chores tend to be synchronous in nature, making opportunities for parallelization limited. Therefore, management job submission and job execution tend to be tightly coupled, requiring a lot of manual care to achieve sequencing of related management activities. Therefore, conventional opportunities may be limited for cloud-scale management automation. Therefore, the evolution and migration of installed pluggable databases may also be limited, such as when it is easier to tolerate a medium topology than to optimize the rearrangement of pluggable databases. For example, relocating a pluggable database from one container database and / or computer to another container database and / or computer may require repackaging and redeploying many clients and / or services, which may not be conducive to such relocation. Human database tuning and management are slow and error-prone. The quality of management also varies greatly depending on the knowledge level of the human database administrator (DBA). One way the industry solves this problem is to hire professional DBAs to reduce errors to some extent and improve performance to some extent, which can more or less alleviate but not completely solve various problems related to scale. What is needed is a solution that enables business insight with unparalleled reliability, a solution that virtually ensures that error-prone manual management processes are avoided. As a result, the reliability, availability, and serviceability (RAS) of the data grid, as well as efficiency and performance, should be improved to previously unattainable levels. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] In the attached figure:
[0007] Figure 1 is a block diagram depicting an example cloud in an embodiment that uses asynchronous jobs to ultimately execute management configuration requests for a pluggable database;
[0008] Figure 2 is a flow chart describing an example asynchronous configuration process in an embodiment that ultimately performs a management request for a pluggable database;
[0009] Figure 3 is a block diagram depicting an example cloud with automatic reflection to heal topology in an embodiment;
[0010] Figure 4 is a block diagram depicting an example cloud with a rules engine that reacts to database system events by autonomously submitting asynchronous jobs that reconfigure the database in an embodiment;
[0011] Figure 5is a flow chart depicting an example autonomous process using a rules engine in an embodiment that reacts to database system events by submitting asynchronous jobs that reconfigure the database;
[0012] Figure 6 is a block diagram depicting an example cloud in an embodiment that uses machine learning to automatically tune rules to optimally manage a container database and its pluggable databases;
[0013] Figure 7 is a block diagram depicting an example cloud in an embodiment that is trained with historical events to improve rules;
[0014] Figure 8 is a block diagram depicting an example cloud for avoiding interference between rules in an embodiment;
[0015] Fig. 9 is a flow chart depicting an example database tuning process for faster query execution in an embodiment;
[0016] Fig.10 is a block diagram illustrating a computer system upon which embodiments of the present invention may be implemented;
[0017] Fig.11 is a block diagram illustrating a basic software system that may be used to control the operation of a computing system. DETAILED DESCRIPTION
[0018] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.
[0019] Various embodiments are described herein according to the following outline:
[0020] 1.0 General Overview
[0021] 2.0 Sample Data Grid
[0022] 2.1 Container Database and Pluggable Database
[0023] 2.2 Management Requests
[0024] 2.3 Configuration Details
[0025] 2.4 Configuration Descriptor
[0026] 2.5 Asynchronous
[0027] 2.6 Hosting Metadata
[0028] 2.7 Post-processing
[0029] 3.0 Asynchronous Configuration Processing
[0030] 4.0 Autonomous Reconfiguration
[0031] 4.1 Monitor
[0032] 4.2 Upgrade / Downgrade
[0033] 5.0 Example Autonomous Database Cloud
[0034] 5.1 Database System Events
[0035] 5.2 Event categories
[0036] 5.3 Rules Engine
[0037] 5.4 Rules
[0038] 6.0 Example Autonomous Management Process
[0039] 6.1 Stimulus
[0040] 6.2 Reaction
[0041] 6.3 Chain Reaction
[0042] 7.0 Machine Learning
[0043] 7.1 Optimization Algorithm
[0044] 7.2 Rules API
[0045] 8.0 Training
[0046] 8.1 Event Delivery
[0047] 8.2 History Replay
[0048] 8.3 Weighted Attributes
[0049] 8.4 Training Confidence
[0050] 8.5 Retraining
[0051] 8.6 Preserving Priorities
[0052] 8.7 Example Technology Stack
[0053] 9.0 Rule Interference
[0054] 9.1 Condition Identification
[0055] 9.2 Templating
[0056] 9.3 Overall Efficiency
[0057] 9.4 Example Administration and Maintenance Tasks
[0058] 10.0 Example Autotune Process
[0059] 10.1 Statistics
[0060] 10.2 Predicate Selectivity
[0061] 10.3 Cost-Benefit Analysis
[0062] 10.4 Example Tuning Scenario
[0063] 11.0 Database System Overview
[0064] 12.0 Overview of Pluggable Databases and Container Database Systems
[0065] 13.0 Hardware Overview
[0066] 14.0 Software Overview
[0067] 15.0 Cloud Computing
[0068] 1.0 General Overview
[0069] Presented herein are autonomous techniques for managing database clouds. The two main components of the example system in this article are an asynchronous guaranteed execution job framework and an event-based automatic rule engine. These main components facilitate automated management and performance optimization of an ecosystem of (one or more) database management systems (DBMSs) that may include multi-tenant databases and / or hierarchies or alliances of databases. The DBMS is described in more detail in the "Database System Overview" section below. In some cases, there may be structural aggregation (e.g., nesting) of databases, such as when a pluggable database (PDB) resides within a container database (CDB). The nesting of PDBs within a CDB is described in more detail in the "Overview of Pluggable Databases and Container Database Systems" section below.
[0070] The elastic nature of online clouds such as data clouds requires dynamic topologies and, at least in the case of public clouds, multi-tenancy. Using the techniques of this paper, a data cloud can be configured as a dynamic ecosystem of resources (including data repositories and analytics) that may experience disturbances with varying degrees of unpredictability such as demand spikes, network weather, maintenance, and capacity growth. This paper is a rule engine for DBMS integration in an asynchronous framework based on various mechanisms and techniques for configuring intelligent reactions that optimize cloud performance under changing environmental conditions.
[0071] The DBMS herein may provide or cooperate with a database portability mechanism so that a database may be automatically copied or relocated. Thus, the logical topology and load distribution of the cloud may be dynamic and optimizable. For example, a point of delivery (PoD) may be a logical co-location unit of a service. In an embodiment, a computer may host one or more PoDs, each PoD may host one or more container databases, and each container database may host one or more pluggable databases. The creation, copying, moving, and removal of (one or more) database instances may occur according to a well-defined (e.g., unified) lifecycle of a database.
[0072] The rule engine can submit asynchronous jobs to perform some or all pluggable database (PDB) lifecycle tasks across a set of delivery points (PoDs), container databases, or other databases hosting middleware. In an embodiment, the asynchronous job framework provides a representative state transfer (REST) API for clients to submit requests, which are then converted into asynchronous jobs. Asynchronous jobs are units of work that can be postponed, such as scheduling or queuing. Asynchronous jobs can be submitted in one order and then executed in another time order. All aspects of job management, including persistence, planning, allocation, execution, recovery, and error handling, are handled within the system. The job is routed to the correct PoD, where it is executed using the PoD API. The main components including request brokers, event brokers, and rule engines are independently and horizontally scalable.
[0073] The rule engine accepts events and converts them into job specifications for execution. Events from the PoD database are ingested by the event broker, which then filters and categorizes the events and feeds them into the rule engine and machine learning engine. Rules are both automatically and manually generated to enable the system to self-tune and manage.
[0074] In general, a rule engine can be a logic interpreter that interprets rules by dividing its code base into editable / replaceable rules and an enhanced engine to achieve customizability and maintainability. The rules can conform to a formal grammar that the engine can parse. To some extent, even if the rules encode logic, the rules can also be processed and / or stored as data. Therefore, the rules can be automatically modified, such as according to analysis and / or machine learning. In an embodiment, the rules can be self-modifying. The rules can contain predicates to be evaluated, and if the predicates are met, the results will occur. The predicates can evaluate a combination of conditions such as variables and / or events. The rules can change the conditions by updating variables and / or triggering additional events. Therefore, related rules can interact to achieve inferences that derive information and / or sequencing of composite behaviors. Rules, rule activation, and rule evolution will be discussed later in this article for autonomous management and performance optimization of data clouds. Rule-based generation and orchestration of asynchronous database management jobs will be discussed later in this article. Rule tuning and evolution based on machine learning will be discussed later in this article.
[0075] The solution proposed herein has an asynchronous job that configures or reconfigures a pluggable database, such as an asynchronous database management job. For example, an asynchronous database management job can create or discard an index of a relational table, or create, copy, move or remove a database instance. Generally speaking, database configuration or reconfiguration is a system change that defines, removes or adjusts (e.g., redefines) database objects such as database instances, patterns, namespaces, tables, relations, indexes, files or logic. Configuration can be a part of supplying or deploying a database or database object at a given computer and / or in a given database or DBMS during relocation from another computer or DBMS. Reconfiguration can be to adjust or remove a database or database object. Configuration or reconfiguration may require execution of a data definition language (DDL), adjustment of the content of the data dictionary of the DBMS and / or addition, change or removal of resources such as files. Asynchronous can promote planning, workload balancing and prioritization.
[0076] Each solution in this article has a rule engine that choreographs asynchronous jobs for context-appropriate behavior. The rule engine has a template engine that processes rules encoded in templates. The rules are conditioned by events. Each of these solutions has a database management system (DBMS) that dynamically issues events indicating operating conditions. Events can cause rules to generate management requests to configure or reconfigure pluggable databases. For example, a pluggable database may run out of disk space. In response, the DBMS can send events to the rule engine. The execution of the rules by the rule engine can generate management requests to move the pluggable database from a container database or computer to another container database or computer. The rule engine sends the management request as a REST request to a request agent, which indicates the configuration details of identifying the pluggable database and specifying the ejection (relocation) of the pluggable database. The request agent dispatches asynchronous jobs based on the REST request. Based on the configuration details, the asynchronous job moves the pluggable database to another container database on another computer.
[0077] In an embodiment, a rule engine receives an event from a DBMS. Based on the event, the rule engine executes a rule to generate a request indicating database configuration details. The rule engine sends the request to a request broker. The request broker schedules an asynchronous job based on the request. The asynchronous job configures the database based on the configuration details. Thus, a database in a cloud, data grid, or data center can be autonomously managed (without human intervention) based on foreseeable and unforeseeable dynamic conditions.
[0078] The solution can have components that cause asynchrony, such as queues, thread pools, swimlanes, and executors. As shown below, Figure 1-3 Depicts a cloud infrastructure tailored for asynchronous execution of database management jobs. Figure 4-9 Depicts the refinement of that asynchronous database management cloud infrastructure that achieves autonomy based on a rule engine that reacts intelligently to spontaneous events from (one or more) DBMSs by initiating asynchronous jobs. The asynchronous database management cloud infrastructure can be combined Figure 10-11 Depicted computing system. In practice, embodiments may combine some or all of the database clouds and computing systems of the Figures discussed herein, in whole or in part, as well as those set forth in related US Application 16 / 055,468.
[0079] 2.0 Sample Data Grid
[0080] Figure 11 is a block diagram depicting an example cloud 100 in an embodiment. Cloud 100 uses asynchronous jobs to ultimately execute management requests for pluggable databases. In order to operate as a data grid, cloud 100 consists of (one or more) computer clusters such as 111-112 hosting databases such as 121-122 and 130. For example, each of databases 121-122 and 130 can be a relational database managed by the same database management system (DBMS). Each of computers 111-112 can be a personal computer, a rack server such as a blade, a mainframe, a virtual machine, or any networked computing device capable of hosting a database. Computers 111-112 can communicate with each other and other computers (not shown) via a network such as a local area network (LAN) or a network such as the global Internet. Computers 111-112 can be co-located, such as on the same rack, or can be separated, such as in different data centers.
[0081] 2.1 Container Database and Pluggable Database
[0082] Cloud 100 hosts databases such as pluggable database 130 in a manner that can hide topology details from clients (not shown). For example, a client may use pluggable database 130 without knowing which data center, computer, and container database actually hosts pluggable database 130. Container databases such as 121-122 may contain pluggable databases such as 130. For example, container database 121 may be shared by multiple tenants, which may or may not share the various pluggable databases hosted by container database 121.
[0083] 2.2 Management Requests
[0084] In operation, a database administrator (DBA) person or management software may submit a management request 140 to create, configure, or reconfigure a pluggable database 130 without knowing where the pluggable database 130 is or will be hosted. For example, the cloud 100 may implicitly or spontaneously relocate the pluggable database 130 to a different container database and / or computer repeatedly at nearly any time. Thus, the configuration details 150 within the management request 140 need not identify the container database or computer.
[0085] For example, a DBA may submit management requests 140 to the cloud 100 for execution from virtually any computer that may or may not reside within the cloud 100. The management requests 140 may be delivered to the cloud 100 as a Hypertext Transfer Protocol (HTTP) request (such as a Common Gateway Interface (CGI) post), a Representational State Transfer (REST) request, a Remote Procedure Call (RPC), a Simple Object Access Protocol (SOAP) call, an Asynchronous JavaScript XML (Extensible Markup Language) (AJAX) request, or other remote data delivery mechanism.
[0086] 2.3 Configuration Details
[0087] Management requests 140 are goal / result oriented and need not specify a strategy or policy to be executed.For example, configuration details may specify that a pluggable database 130 should be created or tuned to suit certain high-level requirements.
[0088] For example, if the pluggable database 130 is running out of disk space, the configuration details 150 may specify a larger amount of disk space. The cloud 100 has flexibility in fulfilling the management request 140. For example, the larger disk space may be achieved by moving the tablespace files of the pluggable database 130 from one file system to another file system as was previously (cross) installed on the same computer. Alternatively, the more disk space may be achieved by adjusting the virtual machine allocations (and possibly restarting the virtual machine) in which the pluggable database 130 is hosted. Alternatively, the cloud 100 may actually have to move the pluggable database 130 from one container database and / or computer to another container database and / or computer.
[0089] Likewise, if the pluggable database 130 is not performing well, the configuration details 150 may specify more processing power, such as more processors, processor cores, threads, or parallelism. The cloud 100 may fulfill the management request 140 by granting more processor cores to the virtual machine hosting the pluggable database 130. Alternatively, the cloud 100 may actually have to move the pluggable database 130 from one container database and / or computer to another container database and / or computer to obtain more processing power.
[0090] The cloud 100 does not immediately (synchronously) execute the management request 140. For example, fulfilling the management request 140 may require temporarily taking the pluggable database 130 out of service. Thus, the configuration details 150 may specify a time, a time window, an amount of idle or light workload, or other dynamic conditions that should occur before executing the management request 140. Alternatively, the cloud 100, the pluggable database 130, or any involved container databases or computers may be temporarily too busy (e.g., a demand spike) to immediately execute the management request 140. Thus, the cloud 100 may queue, buffer, or otherwise defer execution of the management request 140.
[0091] 2.4 Configuration Descriptor
[0092] The cloud 100 uses the configuration descriptor 160 and the asynchronous job 170 to postpone the execution of the management request 140. After receiving the management request 140, the cloud 100 generates the configuration descriptor 160 based on the management request 140 and the configuration details 150. The configuration descriptor 160 encodes the normalized specification of the management request 140 in a format suitable for storage and ultimately dispatching to the asynchronous job 170 for execution. The configuration descriptor 160 can be an encoded document, such as XML or JavaScript Object Notation (JSON). Normalization may require conversion and / or scaling of numerical units (as discussed later in this document) or range restrictions. For example, the configuration details 150 may specify nightly execution, which the cloud 100 may convert into a configuration descriptor 160 that is a time window for a specific late night period. For example, requests for less than a minimum amount of resources may be rounded. Also as discussed later in this document, a specified number of resources may be dependent on each other in an enforceable manner, such as an allocated volatile memory should not exceed an allocated persistent storage device.
[0093] The cloud 100 may buffer or store the configuration descriptor 160 in a memory, database, or file. The cloud 100 may (e.g., synchronously) confirm to any client that submitted the management request 140 that the request is accepted and pending. For example, if the management request 140 arrives as an HTTP request, the cloud 100 may synchronously send an HTTP response confirming that the management request 140 has at least been queued. The confirmation may include a tracking identifier that the client may subsequently use to cancel the management request 140, suspend the management request 140, or poll the status of the management request 140. The confirmation may include an object-oriented receiver (such as a computational future, a remote future, or a smart stub) that may poll and / or wait for the management request 140 to actually be fulfilled. The receiver may be an instance of an object-oriented class. The receiver may be active, such as a smart stub that polls for status. A stub is a remote agent. A smart stub is a stub that contains additional logic for flexibly locating (one or more) services and / or managing repeated operations such as polling or retries. The receiver may be passive, such as suspended while waiting for a status or completion message or signal from the cloud 100 .
[0094] 2.5 Asynchronous
[0095] After generating the configuration descriptor 160, the cloud 100 may immediately or eventually create (or reuse) an asynchronous job 170 to execute the management request 140 based on the configuration descriptor 160. For example, the asynchronous job 170 may be reused from a lightweight thread pool, a heavyweight operating system process, or other execution program. After the configuration descriptor 160 is assigned to the asynchronous job 170, the asynchronous job 170 may immediately or eventually execute (fulfill) the management request 140 according to the configuration descriptor 160. For example, the asynchronous job 170 may be idle until the configuration descriptor 160 is assigned to it. For example, the configuration descriptor 160 may specify that the execution of the management request 140 should not start before midnight.
[0096] Alternatively, the asynchronous job 170 can execute one other request at a time from a queue of pending configuration descriptors. When the configuration descriptor 160 finally reaches the head of the queue, the asynchronous job 170 can exit the queue and execute the configuration descriptor 160. For example, the configuration descriptor 160 can be a JSON document in a Java Message Service (JMS) queue. In an embodiment, the queue can be a priority queue based on any of the following: a) a priority indicated in a configuration descriptor such as 160, b) a deadline (completion time) or a time to live (TTL) indicated in the configuration descriptor, or c) a request type, such as creating a database, moving a database, improving database service quality, copying a database, etc. For example, backup and copy creation are naturally low priorities. However, improving service quality is naturally a high priority because the database is becoming insufficient, such as increased waiting time or running out of disk space. Restoring a database from a backup may be urgent.
[0097] 2.6 Managed Metadata
[0098] The asynchronous job 170 can load, parse, or otherwise process the configuration descriptor 160 to discover which configuration change or changes to perform for the pluggable database. If the configuration descriptor 160 does not identify the container database or computer, the cloud 100 can retrieve the required identifier from the managed metadata 180. The managed metadata 180 provides directory information that maps the identifier of the pluggable database to the identifier of the container database and / or computer that hosts the pluggable database. The managed metadata 180 can be implemented with a lookup table, a database table, or other reference data structure (such as a mapping). The managed metadata can be deployed as a file, a network service, or other means. The managed metadata 180 can have additional information, such as measurements of resource capacity and utilization, resource requirements, and database interdependencies. For example, the hosting metadata 180 may indicate facts such as: a) a particular container database has a particular pluggable database, b) a particular pluggable database uses a certain amount of disk space, c) a count of processing cores that a particular computer has, d) a particular physical computer hosts a particular virtual computer, e) a particular computer hosts a particular container database, f) a particular container database has a replica of a particular pluggable database, g) a particular container database hosts a data partition of a particular multi-instance pluggable database, and h) a particular computer and database have a particular flavor, release level, and patch level.
[0099] For example, asynchronous job 170 may detect that configuration descriptor 160 specifies three processor cores for pluggable database 130. Asynchronous job 170 may consult managed metadata 180 to discover that pluggable database 130 resides in current container database 121 on computer 111. Managed metadata 180 may indicate that: a) current container database 121 is limited to two processing cores, even though b) computer 111 has four processing cores. Therefore, asynchronous job 170 may decide that pluggable database 130 needs to be moved to a different container database. For example, managed metadata 180 may indicate that: a) computer 112 also has four processing cores, and b) target container database 122 has no restrictions on the use of processing cores. Therefore, asynchronous job 170 may select target container database 122 as to where to move pluggable database 130. Therefore, asynchronous job 170 may fulfill management request 140 by moving pluggable database 130 to target container database 122.
[0100] 2.7 Post-processing
[0101] After fulfilling the management request 140, the asynchronous job 170 can additionally perform post-processing in various ways. The asynchronous job 170 can notify the cloud 100 or client that the management request 140 is complete. The asynchronous job 170 can update the managed metadata 180 to reflect the changes made by the asynchronous job 170. The asynchronous job 170 may be idle, returned to the worker pool, dequeue another management request, or await destruction, such as being garbage collected or otherwise deallocated.
[0102] 3.0 Asynchronous configuration processing
[0103] Figure 2 is a flow chart depicting an example asynchronous configuration process in an embodiment that ultimately uses an asynchronous job to perform management requests for a pluggable database. Figure 1 discuss Figure 2 .
[0104] In step 202, an administration request indicating configuration details of a pluggable database is received. For example, cloud 100 receives an administration request 140 from an administrator (either from an external client or from an internal facility) who wants to reconfigure pluggable database 130. For example, the administrator wants to reduce the latency of pluggable database 130 by giving pluggable database 130 more memory (e.g., a larger database cache). The administrator can use a web browser to create configuration details 150 that specify a larger amount of memory in administration request 140.
[0105] In step 204, the cloud generates a configuration descriptor specifying the asynchronous job based on the configuration details of the management request. For example, the cloud 100 refers to the configuration details 150 and the management request 140 when generating the configuration descriptor 160. For example, the configuration descriptor 160 can be a work ticket that defines a pending unit of work. The cloud 100 can: a) prepare the configuration descriptor 160 for immediate execution, b) queue the descriptor for later execution, or c) schedule the configuration descriptor 160 for execution at a relative or absolute time in the future.
[0106] In step 206, the cloud accesses the managed metadata to detect the location(s) of the container database(s) and / or computer(s) that have or will host the pluggable database. Although step 206 is shown as occurring between steps 204 and 208, some or all of the activities of step 206 may actually occur eagerly during step 204, or lazily during step 208. For example, the selection of a target container database or computer to which the pluggable database 130 is moved may occur at step 204. And the identification of the current container database of the pluggable database 130 may occur at step 208, or vice versa. Step 204 may occur almost immediately upon receiving the management request 140. And step 208 may be postponed to a convenient time to actually fulfill the management request 140. Any of those times may be suitable for performing part or all of step 206. For example, accessing the managed metadata 180 may be performed by the asynchronous job 170 or by the cloud 100 before the asynchronous job 170 is created just in time to actually perform the management request 140.
[0107] In step 208, the cloud performs an asynchronous job to configure or reconfigure the pluggable database. For example, the cloud 100 may detect that the pluggable database 130 and / or the current container database 121 may have increased its memory allocation, and therefore the cloud 100 does not need to move the pluggable database 130 to another container database. Therefore, in step 208, the asynchronous job 170 may simply adjust the pluggable database 130 and / or the current container database 121 to reserve more memory.
[0108] At any time during the life cycle of the asynchronous job 170 (including any of the above steps), the asynchronous job 170 may generate additional management requests. For example, while fulfilling the management request 140, the asynchronous job 170 may clone the pluggable database 130 by submitting a second management request to create a backup of the pluggable database 130 from the current container database 121 and a third management request to restore the backup of the pluggable database 130 to the target container database 122.
[0109] 4.0 Autonomous Reconfiguration
[0110] Figure 3 is a block diagram depicting an example cloud 300 in an embodiment. Cloud 300 has an automatic reflection function that dynamically optimizes the performance of the topology. In this example, cloud 300 automatically relocates and upgrades pluggable database E in response to disk fullness. Cloud 300 may be an implementation of cloud 100.
[0111] 4.1 Monitor
[0112] Cloud 300 includes components 310, 321-322, 330, 360, and 370, each of which may be a computer or a co-located software service that shares a computer. Monitor 310 receives telemetry, such as measurements and alarms, from components of cloud 300. Monitor 310 may be configured with reflections, such as rule-based reactions, that: a) detect environmental conditions, such as alarms, threshold crossings, and other conditions in a configurable manner; b) make configurable decisions on how to react; and c) issue management requests to execute its decisions.
[0113] Disk Full A can be an alert indicating that a disk is full, or depending on a threshold that is crossed, that the disk is close to being full, such as when the container database (CDB) 321 has a busy period generating a large number of audit logs. In response to Disk Full A, the monitor 310 can decide to: a) switch to using a storage area network (SAN) for some files (such as audit files), b) move the container database (and its pluggable databases) to a point of delivery (PoD) with a larger disk, or c) evict some pluggable databases from the container database. As shown, the monitor 310 decides to evict the pluggable database E from the CDB 321 by generating request B.
[0114] 4.2 Upgrade / Downgrade
[0115] The execution of request B requires a query to the managed repository 360, which responds by posting an indication that the pluggable database E should be moved to the previous and next managed C in CDB 322. The previous and next managed C also indicate a software version mismatch, for which the worker 370 should apply a software upgrade (or downgrade) to the pluggable database E during the move. Each of the CDBs 321-322 has a large number of system and application software layers, and each software layer has its own release version and patch life cycle. Therefore, the version mismatch may involve one, some, or all of the container database DBMS, PoD framework, bytecode interpreter, virtualization middleware, operating system, or other installed software infrastructure.
[0116] Worker 370 can use the previous and next hosting C to detect version mismatches and execute migration scripts to migrate pluggable database E to the target release version. Migration scripts can include artifacts such as shell scripts, python scripts, native executable programs, stored procedures, and other structured query languages (SQL) such as data definition languages (DDL) or data manipulation languages (DML). Migration scripts can adjust artifacts such as schemas, table contents, and files. Some migration scripts may be run to prepare pluggable database E for relocation. Other migration scripts may be run to prepare the target CDB 322 to host the pluggable database E. After the pluggable database E is moved into the CDB 322, there are other migration scripts that can also adjust the pluggable database E.
[0117] The architecture and techniques for implementing and operating an autonomous multi-tenant database cloud service framework are presented in the related U.S. application Ser. No. 16 / 055,468.
[0118] 5.0 Example Autonomous Database Cloud
[0119] Figure 4 4 is a block diagram depicting an example cloud 400 in an embodiment. Cloud 400 has a rule engine that reacts to database system events by autonomously submitting asynchronous jobs for reconfiguring the database. Cloud 400 may be an implementation of cloud 100. Cloud 400 includes a DBMS such as 410, a rule engine 440, and a request broker 470 (which may be a related U.S. application Ser. No. 16 / 055,468). Figure 6 The DBMS 410 includes a database such as 420, which may be a relational database, a NoSQL database, or other managed database.
[0120] 5.1 Database System Events
[0121] The operating conditions of DBMS 410 can fluctuate dynamically, such as disk space, processing load, or query mix. DBMS 410 can be configured with thresholds and other triggers that detect significant changes in operating conditions. For example, unused disk space can be reduced to below a threshold percentage or absolute amount. In response to significant changes in operating conditions, DBMS 410 can issue events such as 430 to announce such changes for a given pluggable database, container database, or DBMS 410 itself. For example, DBMS 410 can send a JMS message or operating system signal to deliver event 430 to other (one or more) software programs. Therefore, based on a series of emitted events, other software can easily see the operating conditions of DBMS 410.
[0122] 5.2 Event categories
[0123] Event 430 can represent a collection of attributes that can be checked to determine if a significant condition is occurring. In an embodiment, an event broker handles event delivery to subscribers. Each event subscription consists of an event type and an event subscriber. Events can be broadly divided into categories such as operations, monitors, and SQL-related.
[0124] Operational events indicate state transitions within the life cycle of a database. For a particular database, an operational event may indicate at least one of: provisioning, deprovisioning, relocating, cloning and / or horizontal scaling, refreshing, backing up, restoring, upgrading, patching, registering, or deregistering.
[0125] Monitoring events indicate the performance status of the database. Monitoring events can indicate performance metrics of SQL commands in general or important statistics (such as measured during a health check).
[0126] SQL-related events indicate statistics for a particular clause, predicate, or entire or partial statement. A statement may be a DDL or DML, such as a query. For example, a SQL event may indicate that a particular predicate is shared by different queries, which may mean that the predicate is a candidate for materialization as a virtual column.
[0127] 5.3 Rules Engine
[0128] The rule engine 440 may receive events such as 430 to monitor the state of a DBMS such as 410. For example, the rule engine 440 may subscribe to events published by the DBMS 410 according to a publish / subscribe framework. Between the rule engine 440 and the DBMS 410 may be an event bus, queue, or proxy (not shown) that may provide functions such as buffering, persistence, reliability, and decoupling so that the DBMS 410 and the rule engine 440 do not need to interface directly with each other.
[0129] The rule engine 440 can be an inference engine that loads and applies rules such as 450 to implement customizable behavior. For example, the rule engine 440 can load the rule 450 as data that can be compiled or interpreted as the behavior of the extended rule engine 440. For example, the rule 450 can be a text that conforms to a grammar that is used to encode which events can match the rule and what actions to take when a match occurs. Multiple rules can be loaded to configure the rule engine 440 to associate specific behavioral reactions with specific operating conditions that an event can indicate. For example, the rule 450 can specify that when the event 430 indicates that the free disk space is less than 1GB, the database 420 should be moved to another computer.
[0130] 5.4 Rules
[0131] However, rule engine 440 does not directly execute the action specified by the rule that matches event 430. Instead, rule 450 generates and sends request 460 to cause submission of asynchronous job 480, as shown below. Rule 450 can issue configuration details 465 within request 460 to directly or indirectly specify configuration adjustments for a database such as 420. Configuration details 465 can identify database 420 as a reconfiguration target.
[0132] Rules 450 may send request 460 to request agent 470, such as by sending JSON, XML, or other CGI data over HTTP using a REST request. In an embodiment, request 460 includes an executable script such as JavaScript, SQL, python, or a shell script for execution by asynchronous job 480. Based on request 460 and configuration details 465, request agent 470 may create asynchronous job 480, and execution of asynchronous job 480 may be performed, for example, in accordance with the related U.S. application Ser. No. 16 / 055,468. Figure 6 Asynchronous job 480 is eventually executed, which includes reconfiguring database 420 according to configuration details 465. Thus, rule engine 440 can autonomously manage databases such as 420 according to changes in operating conditions of DBMSs such as 410. Thus, rule engine 440 provides a central repository of autonomous behavior required to maintain the databases of cloud 400.
[0133] 6.0 Example of autonomous management processing
[0134] Figure 5 is a flow chart depicting an example autonomous process using a rules engine in an embodiment that reacts to database system events by submitting asynchronous jobs for reconfiguring the database. Figure 4 discuss Figure 5 .
[0135] 6.1 Stimulation
[0136] In step 501, the rule engine receives events from the DBMS. In an embodiment, the DBMS 410 directly sends events 430 to the rule engine 440. In an embodiment, the DBMS 410 and the rule engine 440 are hosted by separate computers or operating system processes, and the events 430 are delivered directly according to remote protocols such as HTTP, REST, Simple Object Access Protocol (SOAP), and / or Remote Procedure Call (RPC). In an embodiment, the DBMS 410 sends events 430 indirectly. For example, an enterprise service bus (ESB), a message queue (MQ), or other agent conveys, relays, and / or buffers events 430 for ultimate delivery to the rule engine 440. In an embodiment, the rule engine 440 is embedded within the operating system process and address space of the DBMS 410 so that the events 430 can be delivered during a subroutine call.
[0137] Some events may indicate at least one of the following: execution statistics of different queries, such as average response time, execution statistics of data manipulation language (DML) commands, execution statistics of data definition language (DDL) commands, execution statistics of specific queries, results of integrity checks of the database, usage of resources of the database exceeding a threshold, or availability of resources of the database falling below a threshold.
[0138] 6.2 Reaction
[0139] In step 502 and based on the event, the rule engine executes the rule to generate a request indicating configuration details of the database. For example, event 430 matches rule 450, which generates configuration details 465 and request 460 for database 420. For example, configuration details 465 may specify that database 420 should be horizontally scaled by cloning / copying to additional computers. In an embodiment, configuration details 465 may specify adjusting at least one of the following DBMS tuning parameters: the maximum count of open database cursors, the maximum count of connected sessions, the maximum size of memory storage for session-related variables, the maximum size of memory storage for session-independent variables, or the amount of parallelism (e.g., CPU cores) used to process queries. Configuration details 465 may also have other tunable attributes for DBMS tuning, such as: transaction log size, number, and host, checkpoint interval for recovery, size of execution plan cache (also called procedure cache), size, key, and host of data partitions, column indexes or encodings, size of query cache, size of row cache, size of data block cache, transaction lock granularity, log buffer size, log refresh frequency or granularity, view materialization, index rebuild, index data structure, schema denormalization, amount of replication, and cache victim strategy. Configuration details 465 may also have tunable attributes for tablespace tuning, such as size and number, amount of memory, undo tablespace size, extension size, and prefetch size. As discussed later herein, a template mechanism may generate request 460 and / or configuration details 465.
[0140] In step 503, the rule engine sends the request to the request agent. For example, the rule engine 440 can send the request 460 to the request agent 470. Figure 4-6 An example of sending a request to a request broker is discussed. Ideally, request broker 470 initially buffers or minimally processes request 460 so that demand spikes for inbound requests can be absorbed quickly and without causing network back pressure on rule engine 440 or interactive users such as DBAs. For example, some processing associated with the ingestion of request 460 may or may not be deferred to the next step 504.
[0141] In step 504, the request agent creates and assigns an asynchronous job based on the request. For example, the request agent 470 can generate an asynchronous job 480 based on the configuration details 465. In some embodiments, the asynchronous job 480 can be a logic script such as python or JavaScript or a Java job, such as at least one of the following: Runnable, Callable, or Future defined by the java.util.concurrent package. The execution of the asynchronous job 480 can be postponed to a scheduled time, such as a time maintenance window, until execution resources become available (such as pooled threads), or until the transaction load of the database 420 just drops below a threshold.
[0142] In step 505, the asynchronous job configures the database based on the configuration details. For example, asynchronous job 480 eventually executes, which reconfigures database 420 according to configuration details 465. For example, asynchronous job 480 may call a stored procedure of database 420 to cause the reconfiguration.
[0143] 6.3 Chain Reaction
[0144] Asynchronous job 480 may perform various post-processing after execution, such as evaluating the system state and condition of DBMS 410 and / or database 420, which may result in (one or more) additional events being sent to rule engine 440. For example, asynchronous job 480 may create database 420, which causes an event indicating how much free disk space is left on the local drive, which may or may not trigger other rules to activate (i.e., be applied). Asynchronous job 480 may send an event indicating successful or unsuccessful completion of asynchronous job 480, which may or may not trigger other rules to activate.
[0145] For example, after asynchronous job 480 configures database 420, asynchronous job 480 may initiate a database condition check that generates additional events indicating at least one of the following: execution statistics of different queries, execution statistics of DML writes, execution statistics of DDL commands, execution statistics of specific queries, results of integrity checks of the database, usage of resources of the database exceeding a threshold, or availability of resources of the database falling below a threshold. Real-time telemetry techniques for automatically monitoring the health of remote systems are presented in related U.S. Patent No. 9,692,819.
[0146] 7.0 Machine Learning
[0147] Figure 66 is a block diagram depicting an example cloud 600 in an embodiment. Cloud 600 uses machine learning to automatically tune rules for optimal management of a container database and its pluggable databases. Cloud 600 may be an implementation of cloud 100. Cloud 600 includes a machine learning engine 630, a rules engine 640, and a container database (such as 610).
[0148] 7.1 Optimization Algorithm
[0149] The machine learning engine 630 includes an optimization algorithm 635, such as 671-672, that uses adaptive tuning to generate improved rules. The algorithm 635 can be at least one of the following: nearest neighbor, artificial neural network, linear regression, or logistic regression. The algorithm 635 can generate rules that are custom tuned for a specific cloud customer (such as database tenant 650) and / or a specific service level agreement (SLA) (such as 660). For example, for a luxury tenant, the machine learning engine 630 can generate the following rules: the rule treats the pluggable database 620 as elastic and replicates the pluggable database 620 to other container databases when in high demand. For an economic tenant, the machine learning engine 630 can generate a rule that only reserves more disk space during high demand to accommodate a larger redo log.
[0150] Although not shown, the machine learning engine 630 can periodically or continuously receive facts, measurements, and statistics from the cloud 600 and / or databases 610 and 620 to use as training inputs for the algorithm 635, as discussed later herein. Thus, the machine learning engine 630 can periodically or continuously regenerate rules, such as 671-672, which can be sent to the rule engine 640 to supplement or replace other rules previously loaded by the rule engine 640. Thus, the rule engine 640 can have an evolving rule base that improves based on the deep learning and training of the algorithm 635. Thus, as more facts and measurements are provided to the machine learning engine 630, the configuration of the database of the cloud 600 will be better based on the evolving rules generated by the training algorithm 635.
[0151] 7.2 Rules API
[0152] The rule engine 640 may have many rules, including some manually made rules and other automatically generated rules. Therefore, the management of the rule base of the rule engine 640 may be more or less complex. The rule engine 640 may have an application programming interface (API) for automated rule management. For example, the rule API may support operations such as create, read, update and delete (CRUD) of generated and / or manually made rules. The rule API may have operations for injecting events into the rule engine 640 for rule processing.
[0153] The rule API may be implemented as a remote procedure call (RPC), a Web service (such as REST and / or a Common Gateway Interface (CGI) post), a stored procedure, or other interface protocol.
[0154] 8.0 Training
[0155] Figure 7 700 is a block diagram depicting an example cloud 700 in an embodiment. Cloud 700 is trained with historical events to improve rules. Cloud 700 may be an implementation of cloud 100. Cloud 700 includes a machine learning engine 740, a rule engine 770, a publish-subscribe bus 720, and a DBMS (such as 711-712).
[0156] 8.1 Event Delivery
[0157] DBMS 711-712 issues events, such as 731-733. However, events are not delivered directly to consumers, such as rule engines 770 and machine learning engines 740. Instead, events are communicated by a publish-subscribe bus 720 that relays events to interested (one or more) consumers. Publish-subscribe bus 720 can be an enterprise service bus (ESB), a publish-subscribe proxy, or other forms of event queues. In an embodiment, events are classified by topics dedicated to event sources or event types, such as by pluggable databases, by container databases, by clients, or otherwise. Interested consumers (such as engines 740 and 770) can subscribe to topics of interest to receive related events. In an embodiment, each client has a dedicated rule engine (such as 770) that subscribes to client-specific event topics through publish-subscribe bus 720.
[0158] The publish-subscribe bus 720 can only relay events to (one or more) subscribers. For example, event 733 is relayed without additional communication to the rule engine 770, which can match the rules of the rule engine 770 that trigger (one or more) asynchronous jobs. The publish-subscribe bus 720 can also permanently record events in the event history 725. In an embodiment, the event history 725 is a database of recorded events. For example, the relationship table of events can have columns indicating the source and nature of each event, such as by identifying one of the following: a client, a pluggable database, a container database, an event type, an event parameter, or a timestamp.
[0159] 8.2 History Replay
[0160] Event consumers such as machine learning engine 740 can subscribe to publish-subscribe bus 720 for real-time events or retrieve recorded events in event history 725. For example, event 731 can be retrieved from event history 725 to train machine learning engine 740. Real-time events such as 732-733 can be relayed to consumers such as engines 740 and 770 in near real-time, with or without being recorded in event history 725 as they pass through publish-subscribe bus 720. Thus, machine learning engine 740 can be batch trained based on a large batch of events from event history 725, or can be trained autonomously and incrementally based on a live stream of events.
[0161] 8.3 Weighted Properties
[0162] Each event (such as 731) can have attributes (such as 791-792) that indicate separate operating conditions that exist when the event is generated. For example, attribute 791 can indicate the backlog depth, such as according to the size of the redo log. For example, attribute 792 can indicate, for example, CPU load, time of day, event type, or event source. Other events can have the same or other attributes. Some attributes can be more important than other attributes. The machine learning engine 740 has weights 751-752 that scale the values of the corresponding attributes 791-792 according to importance. Therefore, the product of the value of attribute 791 and the value of weight 751 can produce a scaled attribute value, which can be injected into the learning algorithm as an input during training. Weights 751-752 can be adjustable. For example, training can optimize weights 751-752.
[0163] 8.4 Training Confidence
[0164] The accuracy of the trained algorithm can depend on how much training has occurred. Therefore, the more events consumed during training, the more appropriate the rules (such as 782) generated by the machine learning engine 740. The confidence score 760 of the machine learning engine 740 increases as the machine learning engine 740 consumes events during training. In an embodiment, training is complete when the confidence score 760 exceeds a threshold 765. In another embodiment, training is complete when the machine learning engine 740 has consumed all training events and the iterative learning algorithm converges.
[0165] 8.5 Retraining
[0166] Threshold 765 may also be used for retraining, where machine learning engine 740 consumes additional events to perform additional tuning, such as additional optimization of weights 751-752. Retraining ends when confidence score 760 exceeds threshold 765, which may have a different value than the initial training. Each time retraining begins, confidence score 760 may be lowered or reset to zero and / or threshold 765 may be increased. In an embodiment, retraining is continuous and occurs as each new event arrives.
[0167] 8.6 Preserving Priorities
[0168] When training or retraining is complete, the machine learning engine 740 regenerates rules, such as 782. The rule engine 770 can obtain and load the regenerated rules. Subsequent loading of rules can supplement or replace previously loaded rules. Client applications such as DBA 701 or human users can load manually crafted rules such as 781 into the rule engine 770. Therefore, the rule engine 770 can have a mixture of rules of manual and automatic origin. Manually crafted rules can be given terms within the rule engine 770, so that the manually crafted rule 781 is replaced by the generated rule 782 only when the confidence score 760 exceeds a threshold (such as 765).
[0169] In an embodiment, the machine learning (ML) engine 740 subscribes to all event types required for its rule recommendation model (ML model) for training. The machine learning engine 740 uses various ML techniques, including k-nearest neighbors, neural networks, and linear regression, to build various rule recommendation models based on the attribute lists included in the subscribed events. Each rule recommendation model adjusts the weights of different attributes assigned to the subscribed events based on historical event data. Since there are multiple tenant databases in the system, the machine learning engine 740 performs cross-tenant analysis on the event data to improve its model confidence. As the number of analyzed events increases, the confidence score of the model increases over time. The recommended rules include conditions and actions / jobs, as explained above. The recommended jobs can be self-tuning jobs or self-maintenance jobs.
[0170] In an embodiment, each database tenant in the cloud 700 can specify the following two confidence score thresholds. This includes a basic confidence threshold, so that once the confidence score of the rule recommendation model exceeds this threshold, a machine-derived rule is generated and published to the rule engine 770. As the confidence increases over time, the machine-derived rule is updated. This also includes an overwrite confidence threshold, and if a conflict occurs between a machine-derived rule and a manually crafted rule, the machine-derived rule will only overwrite the manually crafted rule if its confidence exceeds the overwrite confidence threshold.
[0171] 8.7 Example Technology Stack
[0172] The following is a specific embodiment that can be an implementation of any cloud presented herein. This embodiment is tailored to clouds 400 and 700 and discussed with reference to clouds 400 and 700.
[0173] Request broker 470 includes a WebLogic Java-2 Enterprise Edition (J2EE) application server that includes a web server that accepts a REST API for request submission and management. The REST API is implemented using SmartBear's Swagger OpenAPI. Clients can use CURL or a web browser. Thus, requests can be interactive, scripted, or dynamically generated, such as request 460. Asynchronous jobs (such as 480) are implemented using either or both of the following: Java Message Service (JMS) and java.util.concurrent Java packages (e.g., thread pools, executors, and compute futures). Event distribution via publish / subscribe bus 720 is implemented using JMS publish / subscribe message topics with (one or more) DBMSs such as 711-712 as publishers and rule engine 770 as subscribers. Rule engine 770 is implemented using the Mustache template engine. This document will be discussed later in this document. Figure 8 Templating is further discussed in . The machine learning engine 740 is implemented through a machine learning library (such as Apache Spark MLlib) that has pre-installed algorithms such as logistic regression.
[0174] This particular embodiment has the following benefits. This embodiment is horizontally scalable for: a) symmetric multiprocessing (SMP) such as multi-core, and / or b) distributed computing such as cluster or cloud computing. This embodiment requires only one code base (e.g., pure Java) for heterogeneous computing (e.g., wide area network, WAN).
[0175] 9.0 Rule Interference
[0176] Figure 8 800 is a block diagram depicting an example cloud 800 in an embodiment. Cloud 800 avoids interference between rules. Cloud 800 may be an implementation of cloud 100. Cloud 800 includes a rule engine 810 and a DBMS such as 820.
[0177] Cloud 800 does not expect that individual rules (such as 861-862) will result in a reconfiguration that is globally optimal for cloud 800. Instead, each of rules 861-862 may identify a specific condition (such as 835) that is inconvenient for a specific pluggable database (such as 841-842) or tenant. Likewise, when an individual rule (such as 861-862) is applied, the rule results in a specific action that is beneficial to the pluggable database or tenant, but not necessarily beneficial to the entire cloud 800.
[0178] 9.1 Condition Identification
[0179] For example, the cause or condition 835 may be the exhaustion of storage capacity of a disk local to the container database 830 and shared by the pluggable databases 841-842. The DBMS 820 may detect that the disk is almost full and issue an event or redundant events (such as 851-852) indicating that the disk is full. In an embodiment, events 851-852 are generated for the corresponding pluggable databases 841-842 for the same occurrence of the cause or condition 835. In an embodiment, each of the pluggable databases 841-842 has its own separate set of rules in the shared or corresponding rule engine 810. For example, the rules 861-862 may be more or less similar, but associated with the corresponding pluggable databases 841-842. In an embodiment not shown, the rules 861-862 reside in a separate rule engine because the rules 861-862 are associated with separate tenants.
[0180] In various embodiments, the rule engine 810 can be a single rule engine shared by any of the following: a) some or all pluggable databases of a container database, b) some or all container databases and pluggable databases of the same DBMS, c) some or all container databases and pluggable databases of multiple DBMSs (not shown), or d) some or all container databases and pluggable databases of some or all tenants (e.g., customers and / or applications). Even when separate pluggable databases have separate rule sets, those rule sets can be combined into a combined, monolithic rule base for use in one or more rule engines. For example, each of the rules 861-862 can be maintained in a separate rule set for a separate pluggable database 841-842, and then combined into a monolithic rule base for loading into the same rule engine. In this case, some embodiments can generate only one (rather than two as shown) event from the cause or condition 835, and the single event can trigger any one or both of the rules 861-862.
[0181] In an embodiment, the rule engine 810 can be replicated (i.e., horizontally scaled) such that any event can be delivered to any replica of the rule engine and still activate (i.e., apply) the same rules against the same pluggable database. In a replicated embodiment and depending on the implementation, the pluggable database 841 may or may not be fixed to a specific instance of the rule engine 810. For example, the pluggable database 841 may have been previously managed by one rule engine instance and subsequently managed by another rule engine instance with the same rules. In an embodiment, the horizontal scaling of the rule engine 810 can be elastic, such as for reconfiguration of demand spikes, such as during an operational emergency that affects many pluggable databases simultaneously.
[0182] Each of rules 861-862 has a condition template (such as 870) that matches or does not match the current event (such as 852). For example, condition template 870 may encode predicate logic or other filter expressions that may or may not be satisfied by the attributes of event 852. For example, rule 862 may require gigabytes of free local disk space, so that condition template 870 is satisfied when event 852 indicates that the free space does not exceed a gigabyte.
[0183] 9.2 Templating
[0184] When the condition template 870 is met, the rule 862 is applied by applying the action template 880. For example, the action template 880 can generate a configuration descriptor from which an asynchronous job 890 can be constructed and submitted for final execution. In an embodiment, either or both of the templates 870 and 880 are textual. In an embodiment, the generated configuration descriptor is displayed as templated text within the action template 880, thereby achieving WYSIWIG generation of the configuration descriptor. In an embodiment, the text template is aggregated into a template engine such as Mustache or Java Server Pages (JSP). In an embodiment, the event 852 is delivered to one of the rule engines 810 using HTTP such as CGI post.
[0185] 9.3 Overall Efficiency
[0186] Thus, as described above, the cause or condition 835 may be a disk full indicated by the event 851-852 that triggers the rule 861-862. Thus, the action templates of the two rules 861-862 may be executed in response to the same disk full. For example, the rules 861-862 may both submit asynchronous jobs such as 890 that cause each corresponding pluggable database 841-842 to move out of the container database 830 and into a different container database with more local disk space. In fact, both pluggable databases 841-842 that are determined by the rule to leave the container database 830 may solve the problem of insufficient disk space. However, moving the two pluggable databases 841-842 may be globally suboptimal. For example, time and effort may be saved by moving only one of the two pluggable databases 841-842 instead. Such suboptimality caused by interference from controversial rules may be avoided in various embodiments as follows. Having multiple tenants and / or rules that manage different pluggable databases, databases of different scopes (e.g., container and pluggable databases), or databases for different tenants and / or applications can exacerbate interference between rules.
[0187] In an embodiment, asynchronous jobs such as 890 themselves avoid interference. For example, rule 861 may trigger an asynchronous job (not shown) that evicts pluggable database 841 from container database 830, thereby freeing up some local disk space, thereby causing cause or condition 835 to cease. Ultimately, asynchronous job 890 executes, and it may detect whether various preconditions are met, and if not, abort or reschedule. Asynchronous job 890 may detect whether local disk space is sufficient. If disk space is sufficient, asynchronous job 890 may terminate early, thereby leaving pluggable database 842 within container database 830. In an embodiment, asynchronous job 890 may evaluate its preconditions by re-invoking condition template 870.
[0188] In an embodiment, the asynchronous job from rule 861 may execute normally (i.e., evict pluggable database 841), and then cause another event indicating that local disk space is sufficient to be sent to rule engine 810. This additional event may cause rule engine 810 to cancel asynchronous job 890 that is no longer needed, either by rule or otherwise.
[0189] In an embodiment, the rule engine 810 does not consume event 852 until after the pluggable database 841 is evicted, in which case the condition template 870 will not be satisfied when the event 852 is finally processed. Therefore, the action template 880 will not be applied, and the asynchronous job 890 will not be created. Which of the rules 861-862 is evaluated before the other depends on the embodiment. In an embodiment, events 851-852 are processed in the same order as they arrive at the rule engine 810. In an embodiment, a level or priority can be assigned to each of the rules, pluggable databases, and tenants. In an embodiment, the rules associated with the high priority are evaluated first to avoid priority inversions that would cause the high priority database to wait for the low priority database to react. In an embodiment, the rules associated with the low priority are evaluated first, and the quality of service of the low priority database is temporarily downgraded, for example, while the low priority database is evicted, thereby minimizing the maintenance-related service degradation of the high priority database.
[0190] In an embodiment, the container database 830 may have its own rules (not shown) in the rules engine 810. In an embodiment, the rules of the container database 830 may have a high priority and be executed before the rules of the pluggable databases 841-842, thereby facilitating centralized coordination of multiple pluggable databases 841-842. For example, the rules of the container database 830 may react to disk exhaustion by selecting and evicting the pluggable database 841, thereby centrally solving the problem and avoiding the need to apply the rules 861-862 that would otherwise cause coordination problems, such as race conditions that produce suboptimal results as discussed above.
[0191] In an embodiment, the pluggable database and / or the container database may each have a more or less uniform life cycle, wherein the formalized operational states correspond to typical stages of important maintenance scenarios. For example, the pluggable database may be in a transition state representing a reconfiguration activity, such as scaling up, scaling down, or relocation. In (e.g., Java or C) embodiments, one or more enumerations declare the states of one or more life cycles. Rule templates such as 870 and 880 may check the operational state of one or more pluggable databases. For example, rule 862 of pluggable database 842 may: a) detect that another pluggable database 841 is currently being evicted (i.e., relocated), and b) refuse to activate because the ongoing relocation may quickly resolve the common problem of disk exhaustion. In some embodiments, if a rule refuses to activate due to a detected transition state, then at a later time (e.g., as planned or upon a subsequent change in the transition state), the rejected rule may be re-evaluated, and / or the original event may be reprocessed by the rule engine. Therefore, the rules of multiple pluggable databases may achieve a certain degree of coordination to achieve a global optimal result.
[0192] 9.4 Example Administration and Maintenance Tasks
[0193] Asynchronous job 890 may perform various database management tasks, such as one or more of 891-898. In an embodiment, any of tasks 891-898 are implemented by stored procedure calls in a database (such as 830 and 841-842), which may be called by asynchronous job 890. In an embodiment, any of tasks 891-898 are hard-coded into the code base or script of asynchronous job 890.
[0194] Task 891 creates a pluggable database, such as 841-842, within container database 830. The creation of the pluggable database may be templated so that a prototype pluggable database may be cloned into a container database such as 830 to create pluggable database instances such as 841-842. A rule-based technique for automatically provisioning pluggable databases using a central repository is presented in related U.S. Patent 15 / 266,902.
[0195] Task 892 creates or uses a snapshot or backup of a database, such as 830 or 841-842. Database snapshots are widely used to quickly create point-in-time virtual copies of data. Database backups protect against storage media failures. Backups or snapshots are used (consumed) during database restore or recovery. An automatic backup technique for a pluggable database is presented in related U.S. application Ser. No. 15 / 254,884.
[0196] Task 893 adjusts the maximum rate at which new sessions can connect to a database (such as 830 or 841-842). This can mitigate login storms, such as during a denial of service (DOS) attack.
[0197] Task 894 generates alerts or events based on dynamic conditions. For example, asynchronous job 890 can consume disk space to create a column index and then generate an event (such as 851) to indicate how much free disk space is left. This will cause rules such as 861 to be applied.
[0198] Task 895 adjusts the size of the redo log or fast recovery area. For example, the redo log can be expanded to accommodate more in-flight transactions, such as during demand spikes.
[0199] Task 896 applies a software upgrade or patch to a database (such as 830 or 841-842), or to a DBMS (such as 820). The patch may create or modify a file containing logic or data, may execute a shell or DDL or DML script, and / or may call a stored procedure. Techniques for automatically relocating a pluggable database from one container database to another are provided in related U.S. Patent No. 9,396,220 and U.S. Patent Application No. 15 / 215,446. Techniques for automatically upgrading a pluggable database are provided in related U.S. Patent Application No. 15 / 266,917.
[0200] Task 897 adjusts the amount of at least one resource (such as CPU, persistent storage, or RAM) allocated to a database (such as 830 or 841-842). For example, a volatile cache of disk blocks may be expanded during a demand spike. Task 897 may also affect temporary allocations such as for a specific duration or until a specific condition, after which the resource allocation reverts to a previous amount.
[0201] Task 898 relocates or clones a pluggable database from a first container database on a first computer to a second container database on the same or a different computer. For example, asynchronous job 890 may allocate more processing cores to pluggable database 841 than are available to container database 830. Thus, asynchronous job 890 may also move one of pluggable databases 841-842 to a different container database on a different computer. Techniques for cloning pluggable databases are presented in related U.S. patent application Ser. No. 15 / 215,435.
[0202] 10.0 Example of autonomous tuning process
[0203] Fig. 9 is a flow chart depicting an example database tuning process for faster query execution in an embodiment. Figure 8 discuss Fig. 9 .
[0204] 10.1 Statistics
[0205] Steps 901-903 are preparatory. They collect and analyze the required statistics to decide how to tune the database for better query performance. In step 901, measurements of data distribution and / or storage characteristics of database objects are collected. For example, DBMS 820 may monitor and record resource usage and / or data patterns associated with query statements, query expressions, tables, partitions, and / or columns.
[0206] In step 902, statistics are generated based on the measurements. For example, a histogram of values of a column or expression may be generated. A heat map of CPU, memory, or I / O usage based on a query expression or storage artifact (e.g., a column or table) may be generated. Data dependencies between columns or expressions may be detected.
[0207] 10.2 Predicate Selectivity
[0208] In step 903, the selectivity of the query predicate is calculated based on the statistics. Selectivity facilitates more or less accurate estimation of the result set size and the ratio of the result set row count to the source table row count, which improves the query plan during query processing. Selectivity is particularly helpful for reordering of cascading joins or cascading filter items to aggressively prune intermediate results. For example, even if both columns need to be filtered, filtering on the column with more distinct values first will be more efficient than filtering on the column with fewer distinct values first.
[0209] As a process of action, step 904 analyzes a subset of potential steps 906-912. Each of potential steps 906-912 is a specific reconfiguration technique for the database that can improve future query performance. Each potential reconfiguration incurs a corresponding expected reconfiguration cost and will produce a corresponding expected acceleration. The reconfiguration cost can be based on temporary resource consumption (e.g., CPU, I / O, memory) during the reconfiguration and temporary service degradation during the reconfiguration. The cost of background reconfiguration can be lower than the cost of reconfiguration of temporarily locking database objects. For example, the cost of creating a column index (906) can be higher than the cost of inserting an optimizer hint into a query (911). While a column index can be expected to speed up a query more than an optimizer hint, this can depend on the data access method involved, such as table scans and index traversals. In step 904, the respective estimated costs of each alternative database optimization mechanism are calculated based on the calculated selectivity.
[0210] 10.3 Cost-Benefit Analysis
[0211] Step 905 is deterministic. Based on a cost-benefit analysis, one (or more, if compatible) potential optimizations of 906-912 are selected. In an embodiment, any of steps 904-905 (and 906-912) may be implemented as a rule of the rule engine 810. For example, a cause or condition 835 occurs when the reuse of a particular query, expression, table, or column exceeds a threshold. For example, event 852 may indicate that a column has become hot in a usage heat map. Thus, step 904 may be triggered when the condition template 870 identifies that a hot column meets various criteria. In step 905, one or more of the potential steps 906-912 are selected and merged or not into one or more asynchronous jobs. Thus, the cloud 800 may autonomously optimize the pluggable database 841 or 842 for any or all queries.
[0212] 10.4 Example Tuning Scenario
[0213] In particular, step 906 creates or deletes a database index based on one or more columns of a table. Step 907 encodes or decodes the column, such as by using a dictionary or other compression. Step 908 adjusts the size of the table space or database file, such as by adding spare data blocks. Step 909 materializes or dematerializes the database view or virtual column. Step 910 can defragment the column or table, or preload (prefetch) the column or table into RAM. For example, the column or table can be preloaded into a cache, or loaded and fixed in a memory such as a heap. Step 911 can reserve optimizer hints for repeated insertion into subsequent queries. Step 912 can propagate optimizations (or statistics that cause optimizations) from one data dictionary to another. Therefore, optimizations can be dispersed into copies or fragments and contribute to elastic scaling without manual intervention. A technique for automatically propagating changes between pluggable databases is provided in related U.S. applications 15 / 215,443 and 15 / 266,030.
[0214] Therefore, the cloud database can be automatically optimized to improve query performance via tuning the jobs generated by the rule engine 810. The following is a specific scenario for feature tuning.
[0215] Resource parameters such as Oracle's System Global Area (SGA), Program Global Area (PGA), sessions, and maximum open cursors can be tuned. The SGA is a read / write memory area that, along with Oracle background processes, forms a database instance. All server processes executing on behalf of a user can read information in the instance's SGA. During database operations, several processes write to the SGA. Server processes and background processes do not reside in the SGA, but exist in separate memory spaces. Each database instance has its own SGA. Oracle automatically allocates memory for the SGA when the instance starts and reclaims memory when the instance shuts down. The PGA is memory specific to an operating process or thread that is not shared by other processes or threads on the system. Because the PGA is process-specific, it is never allocated in the SGA. The PGA is a memory heap that contains session-related variables required by dedicated or shared server processes. Server processes allocate the memory structures they need in the PGA. An analogy for the PGA is the temporary tabletop workspace used by a file clerk. By analogy, a file clerk is a server process (client process) that does work on behalf of a client. The clerk cleans up a portion of the work surface, uses the workspace to store details about customer requests and sort folders of customer requests, and then relinquishes the space when the work is complete. An initialization parameter can set the target maximum size of the instance PGA. Individual PGAs can grow to this target size as needed.
[0216] Asynchronous tuning jobs can rely on already collected statistics for database objects. Cost-based optimization methods use statistics to calculate the selectivity of predicates and estimate the cost of each execution plan. Selectivity is the fraction of rows in a table that are selected by the predicates of a SQL statement. The optimizer uses the selectivity of predicates to estimate the cost of a particular access method and determine the optimal join order. Statistics quantify the data distribution and storage characteristics of tables, columns, indexes, and partitions. The optimizer uses these statistics to estimate how much I / O and memory are required to execute a SQL statement using a particular execution plan. Statistics are stored in the data dictionary, and they can be exported from one database and imported into another (for example, transferring production statistics to a test system to simulate a real environment, even though the test system may only have a small sample of data). Statistics should be collected regularly to provide information about schema objects to the optimizer. New statistics should be collected after the data or structure of a schema object is modified in a way that makes the previous statistics inaccurate. For example, after a large number of rows are loaded into a table, new statistics on the number of rows should be collected. After updating the data in a table, new statistics on the average row length may be needed, but new statistics on the number of rows do not need to be collected.
[0217] Initialization parameters such as cursor_sharing can be tuned. The private SQL area holds information about parsed SQL statements and other session-specific information for processing. When a server process executes SQL or PL / SQL code, the process uses the private SQL area to store bind variable values, query execution state information, and query execution workspace. The private SQL area used for each execution of a statement is not shared and can contain different values and data. A cursor is the name or handle of a specific private SQL area. A cursor contains session-specific state information, such as bind variable values and result sets. A cursor is more or less a pointer on the client side to the state on the server side. Because cursors are closely associated with private SQL areas, these terms are sometimes used interchangeably.
[0218] A technique for automatic performance tuning of a relational database is presented in the related US Patent No. 8,903,801. A technique for automatic performance analysis of a relational database is presented in the related US Patent No. 8,341,178.
[0219] 11.0 Database System Overview
[0220] A DBMS manages one or more databases. A DBMS may include one or more database servers. A database includes database data and a database dictionary stored on a persistent storage mechanism (such as a set of hard disks). Database data may be stored in one or more data containers. Each container contains records. The data within each record is organized into one or more fields. In a relational DBMS, data containers are called tables, records are called rows, and fields are called columns. In an object-oriented database, data containers are called object classes, records are called objects, and fields are called attributes. Other database architectures may use other terms.
[0221] A user interacts with the database server of a DBMS by submitting commands to the database server that cause the database server to perform operations on the data stored in the database. A user may be one or more applications that interact with the database server running on a client computer. Multiple users may also be collectively referred to as users in this article.
[0222] Database commands may take the form of database statements that conform to a database language. The database language used to express database commands is Structured Query Language (SQL). There are many different versions of SQL, some standard, some proprietary, and with various extensions. Data Definition Language ("DDL") commands are issued to a database server to create or configure database objects, such as tables, views, or complex data types. SQL / XML is a common extension of SQL used when manipulating XML data in an object-relational database.
[0223] A multi-node database management system consists of interconnected nodes that share access to the same one or more databases. Typically, the nodes are interconnected via a network and share access to shared storage devices to varying degrees, such as shared access to a set of disk drives and data blocks stored thereon. The varying degrees of shared access between nodes can include sharing nothing, sharing everything, exclusive access to database partitions by node, or some combination thereof. The nodes in a multi-node database system can be in the form of a group of computers (e.g., workstations, personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of nodes in the form of server blades interconnected to other server blades on a rack.
[0224] Each node in a multi-node database system hosts a database server. A server, such as a database server, is a combination of integrated software components and an allocation of computing resources, such as memory, nodes, and processes on the nodes for executing the integrated software components on processors, a combination of software and computing resources dedicated to performing specific functions on behalf of one or more clients.
[0225] Resources from multiple nodes in a multi-node database system can be allocated to run the software of a particular database server. Each combination of software and allocation of resources among nodes is a server referred to herein as a "server instance" or "instance." A database server can include multiple database instances, some or all of which run on separate computers (including separate server blades).
[0226] 12.0 Overview of Pluggable Databases and Container Database Systems
[0227] Database consolidation involves allocating and sharing computing resources among multiple databases. A container database management system can be used to consolidate databases. A consolidated database, such as a multi-tenant container database (CDB), includes one or more pluggable databases (PDBs).
[0228] A container database includes a data dictionary that contains metadata that defines database objects in the container database. For example, the data dictionary for a given CDB will contain metadata that describes each PDB contained in the given CDB, including the database objects contained in each PDB. In addition, each pluggable database includes a PDB-specific database dictionary that contains metadata that defines database objects contained in the pluggable database. Database objects include tables, table columns, indexes, files, tablespaces, data types, users, user privileges, and storage structures for storing database object data, among other things.
[0229] A container database can manage multiple pluggable databases, and a given database server instance can manage and provide services to those pluggable databases in the container database. Thus, a given container database allows multiple pluggable databases to run on the same database server and / or database server instance, thereby allowing the computing resources of a single database server or instance to be shared among multiple pluggable databases. In a container database management system, each pluggable database can be opened or closed in the container database independently of other pluggable databases.
[0230] An application can access a pluggable database by establishing a database session for the pluggable database on a container database management system, where the database session represents a connection between the application and the container database management system for accessing the pluggable database. A database session for a pluggable database is initiated by, for example, transmitting a request for a new connection to the container database management system, the request specifying the pluggable database. In response to such a request, the container database management system establishes the requested database session. The container database management system can host multiple database sessions, each database session for one of the multiple pluggable databases.
[0231] A given container database is configured based on the requirements of those database management system (DBMS) features that are applicable to the container database. DBMS features that are applicable to the container database are features that interact with or affect the container database, and therefore require some configuration of the container database. DBMS features that may be applicable to a given container database include one or more of the following: the version of the DBMS that manages the container database (including major version, minor version, and / or patch level); optional features that can be installed or implemented for the container database (such as data encryption, features that allow multiple levels of data restrictions within areas of the database, localization enablement); ordinary users that exist in the container database; independently installed patches that have been installed for the DBMS that manages the container database; and so on.
[0232] The configuration of a CDB consists of aspects of the CDB that are tuned based on the characteristics of the DBMS applicable to the CDB. These aspects of the CDB include one or more of the following: the structure of data or database objects stored in the CDB's pluggable databases; the layout or contents of the CDB's underlying operating system files; the number of background processes required by the CDB; identifiers associated with the CDB; variables required for CDB functionality; initialization parameters; the character set used to encode data in the CDB; time zones supported by the CDB; standard database block sizes; tablespace settings; undo settings; services supported by the CDB; special features implemented for the CDB; database server instance clustering support for the CDB; and so on.
[0233] A pluggable database can be "plugged" into a container database and can be transferred between database servers and / or database management systems. A database server instance plugs a pluggable database into a container database by including metadata describing the pluggable database in the database dictionary of the container database and by initiating management of the pluggable database as part of the container database. Any number of aspects of the configuration of the container database into which a particular pluggable database is plugged can affect the pluggable database.
[0234] When a pluggable database is moved from a source container database to a destination container database (where the destination database and the source container database have the same configuration), the pluggable database does not need to be reconfigured before the pluggable database is made available for operation at the destination container database. However, the source container database and the destination container database from which the pluggable database is relocated are not always configured for the same set of DBMS features. When the pluggable database is moved to a destination container database that has a different configuration than the source container database of the pluggable database, the relocated pluggable database is reconfigured to conform to the configuration of the destination container database, which allows the relocated pluggable database to be configured to be compatible with the DBMS features applicable to the destination container database.
[0235] 13.0 Hardware Overview
[0236] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. Special-purpose computing devices can be hardwired to perform these technologies, or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform these technologies), or can include one or more general-purpose hardware processors that are programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. Such special-purpose computing devices can also combine customized hard-wired logic, ASICs, or FPGAs with custom programming to implement these technologies. Special-purpose computing devices can be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other devices that combine hard-wiring and / or program logic to implement these technologies.
[0237] For example, Fig.10 1 is a block diagram illustrating a computer system 1000 upon which embodiments of the present invention may be implemented. Computer system 1000 includes a bus 1002 or other communication mechanism for communicating information, and a hardware processor 1004 coupled to bus 1002 for processing information. Hardware processor 1004 may be, for example, a general purpose microprocessor.
[0238] The computer system 1000 also includes a main memory 1006, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1002 for storing information and instructions to be executed by the processor 1004. The main memory 1006 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by the processor 1004. When stored in a non-transitory storage medium accessible to the processor 1004, these instructions cause the computer system 1000 to become a special-purpose machine customized to perform the operations specified in the instructions.
[0239] Computer system 1000 also includes a read only memory (ROM) 1008 or other static storage device coupled to bus 1002 for storing static information and instructions for processor 1004. A storage device 1010, such as a magnetic or optical disk, is provided and coupled to bus 1002 for storing information and instructions.
[0240] The computer system 1000 may be coupled to a display 1012, such as a cathode ray tube (CRT), via the bus 1002 for displaying information to a computer user. An input device 1014, including alphanumeric and other keys, is coupled to the bus 1002 for communicating information and command selections to the processor 1004. Another type of user input device is a cursor control 1016, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1004 and for controlling cursor movement on the display 1012. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.
[0241] Computer system 1000 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that is combined with a computer system to make or program computer system 1000 into a special purpose machine. According to one embodiment, computer system 1000 performs the techniques described herein in response to processor 1004 executing one or more sequences of one or more instructions contained in main memory 1006. These instructions may be read into main memory 1006 from another storage medium, such as storage device 106. Execution of the sequences of instructions contained in main memory 1006 causes processor 1004 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.
[0242] The term "storage medium" as used herein refers to any non-transient medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 106. Volatile media include dynamic memory, such as main memory 1006. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or cassette tapes.
[0243] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 1002. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
[0244] Various forms of media may be involved in transmitting one or more sequences of one or more instructions to the processor 1004 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 1000 may receive the data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 1002. The bus 1002 transmits the data to the main memory 1006, from which the processor 1004 retrieves and executes the instructions. The instructions received by the main memory 1006 may optionally be stored on the storage device 106 before or after execution by the processor 1004.
[0245] The computer system 1000 also includes a communication interface 1018 coupled to the bus 1002. The communication interface 1018 provides a two-way data communication coupled to a network link 1020, wherein the network link 1020 is connected to a local network 1022. For example, the communication interface 1018 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection with a corresponding type of telephone line. As another example, the communication interface 1018 can be a local area network (LAN) card to provide a data communication connection with a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 1018 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.
[0246] The network link 1020 typically provides data communication to other data devices through one or more networks. For example, the network link 1020 can provide a connection through the local network 1022 to a host computer 1024 or to data equipment operated by an Internet Service Provider (ISP) 1026. The ISP 1026 in turn provides data communication services through the global packet data communication network, now commonly referred to as the "Internet" 1028. Both the local network 1022 and the Internet 1028 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 1020 and through the communication interface 1018, which carry the digital data to and from the computer system 1000, are example forms of transmission media.
[0247] Computer system 1000 can send messages and receive data, including program code, through the network(s), network link 1020, and communication interface 1018. In the Internet example, server 1030 can send the requested code for an application program through Internet 1028, ISP 1026, local network 1022, and communication interface 1018.
[0248] The received code may be executed by processor 1004 as it is received, and / or stored in storage device 1010 or other non-volatile storage for later execution.
[0249] 14.0 Software Overview
[0250] Fig.11 1 is a block diagram of a basic software system 1100 that may be used to control the operation of computer system 1000. Software system 1100 and its components, including their connections, relationships, and functions, are merely exemplary and are not meant to limit implementation of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.
[0251] A software system 1100 is provided for directing the operation of the computer system 1000. The software system 1100, which may be stored on system memory (RAM) 1006 and fixed storage (eg, hard disk or flash memory) 106, includes a kernel or operating system (OS) 1110.
[0252] The OS 1110 manages low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more applications, represented as 1102A, 1102B, 1102C ... 1102N, may be "loaded" (e.g., transferred from fixed storage 106 into memory 1006) for execution by the system 1100. Applications or other software intended for use on the computer system 1000 may also be stored as a downloadable set of computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, app store, or other online service).
[0253] The software system 1100 includes a graphical user interface (GUI) 1115 for receiving user commands and data in a graphical manner (e.g., "clicks" or "touch gestures"). In turn, these inputs can be operated by the system 1100 according to instructions from the operating system 1110 and / or (one or more) applications 1102. The GUI 1115 is also used to display the results of operations from the OS 1110 and (one or more) applications 1102, and the user can provide additional input or terminate the session (e.g., log out).
[0254] The OS 1110 may execute directly on the bare hardware 1120 (e.g., processor(s) 1004) of the computer system 1000. Alternatively, a hypervisor or virtual machine monitor (VMM) 1130 may be inserted between the bare hardware 1120 and the OS 1110. In this configuration, the VMM 1130 acts as a software "buffer" or virtualization layer between the OS 1110 and the bare hardware 1120 of the computer system 1000.
[0255] VMM 1130 instantiates and runs one or more virtual machine instances ("guests"). Each guest includes a "guest" operating system (such as OS 1110), and one or more applications (such as (one or more) applications 1102) designed to execute on the guest operating system. VMM 1130 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.
[0256] In some instances, VMM 1130 can allow a guest operating system to run as if it were running directly on bare hardware 1120 of computer system 1000. In these instances, the same version of the guest operating system that is configured to execute directly on bare hardware 1120 can also execute on VMM 1130 without modification or reconfiguration. In other words, VMM 1130 can provide full hardware and CPU virtualization to guest operating systems in some cases.
[0257] In other instances, the guest operating system may be specially designed or configured to execute on the VMM 1130 to improve efficiency. In these instances, the guest operating system is "aware" that it is executing on a virtual machine monitor. In other words, the VMM 1130 may provide paravirtualization to the guest operating system in certain circumstances.
[0258] A computer system process includes an allocation of hardware processor time, as well as an allocation of memory (physical and / or virtual), an allocation of memory for storing instructions executed by the hardware processor, for storing data generated by the execution of instructions by the hardware processor, and / or for storing hardware processor state (e.g., contents of registers) between allocations of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system and may run under the control of other programs executing on the computer system.
[0259] 15.0 Cloud Computing
[0260] The term "cloud computing" is used generally herein to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows resources to be rapidly provisioned and released with minimal management effort or service provider interaction.
[0261] A cloud computing environment (sometimes referred to as a cloud environment or just a cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, a private cloud environment is generally intended for use by or within a single organization. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.
[0262] In general, the cloud computing model enables some of those responsibilities that might previously have been provided by an organization's own information technology department to instead be delivered as a service layer within a cloud environment for use by consumers (either inside or outside the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer may vary, but common examples include: Software as a Service (SaaS), in which consumers use software applications running on a cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), in which consumers can use software programming languages and development tools supported by the supplier of the PaaS to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), in which consumers can deploy and run arbitrary software applications, and / or provide processes, storage, networks, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on a cloud infrastructure, while the DbaaS provider manages or controls the underlying cloud infrastructure and applications.
[0263] The above basic computer hardware and software and cloud computing environment are presented to illustrate the basic underlying computer components that can be used to implement the example embodiment(s). However, the example embodiment(s) are not necessarily limited to any particular computing environment or computing device configuration. Instead, according to the present disclosure, the example embodiment(s) can be implemented in any type of system architecture or processing environment that a person skilled in the art would understand in light of the present disclosure as being capable of supporting the features and functions of the example embodiment(s) presented herein.
[0264] In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details, which may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention, and what the applicants intend as the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application in the specific form in which such claims are issued, including any subsequent corrections.
Claims
1. A computer-implemented method comprising: training a machine learning engine based on a plurality of past events historically generated by one or more database management systems DBMS; The machine learning engine generates, and the rules engine loads, a first plurality of database rules including specific rules for generating a request indicating database configuration details; The rule engine receives events from a specific DBMS; The rule engine executes the specific rule based on the event, wherein executing the specific rule includes generating a request indicating database configuration details; The rule engine sends the request to the request agent; The request broker schedules asynchronous jobs based on the request; The asynchronous job configures the database based on the database configuration details; retraining the machine learning engine based on a second plurality of past events that are more recent than the plurality of past events; increasing a confidence score based on a count of past events in the second plurality of past events; as well as When the confidence score exceeds a threshold, the machine learning engine generates a second plurality of database rules that replace the first plurality of database rules.
2. The method of claim 1, wherein: The rule engine includes a template engine; Executing the specific rule includes the template engine selecting the specific rule based on the event; Generating a request involves the template engine generating text from a template.
3. The method of claim 1, wherein: The event indicates at least one item selected from the group consisting of: a result of an integrity check of the database, a usage of a resource of the database exceeding a threshold, or an availability of a resource of the database exceeding a threshold.
4. The method of claim 1, wherein: The specific rule is generated based on at least one of the following: a specific database tenant, or a specific service level agreement SLA.
5. The method of claim 1, wherein: The first plurality of database rules include at least: rules applicable to pluggable databases and rules applicable to container databases.
6. The method of claim 1, wherein: Generating the specific rule includes executing at least one of the following algorithms: nearest neighbor, artificial neural network, linear regression or logistic regression.
7. The method of claim 1, wherein: Each past event in the plurality of past events includes one or more attributes; Training the machine learning engine includes adjusting, for each of the one or more attributes, a weight associated with the attribute.
8. The method of claim 1, further comprising: The rule engine loads the first plurality of database rules and the plurality of manually created rules; The rules engine loads a subset of the second plurality of database rules, wherein each particular rule in the subset of the second plurality of database rules replaces a corresponding rule in: When the confidence score exceeds a second threshold, the plurality of manually crafted rules, or The first plurality of database rules.
9. The method of claim 1, wherein: The database configuration details specify tuning of the database for faster query execution.
10. The method of claim 9, wherein: The database configuration details specify adjusting at least one of: a maximum count of open database cursors, a maximum count of connected sessions, a maximum size of memory storage for session-dependent variables, a maximum size of memory storage for session-independent variables, or an amount of parallelism.
11. The method of claim 9, wherein: The database configuration details specify collecting a plurality of measurements of data distribution or storage characteristics for one or more objects of the database, the one or more objects comprising at least one of: a table, a column, an index, a partition, or a query; The method also includes generating one or more statistics based on the plurality of measurements.
12. The method of claim 11, further comprising: calculating selectivity of a query predicate based on the one or more statistics; Based on the selectivity, a respective estimated cost is calculated for each of a plurality of alternative query mechanisms.
13. The method of claim 12, wherein: The corresponding estimated cost of the query mechanism is based on an estimated amount of at least one of: memory, or input / output I / O; The query mechanism includes at least one of: a specific access method among a plurality of alternative access methods, or a specific join ordering among a plurality of alternative join orderings.
14. The method of claim 11, further comprising performing at least one of the following based on the one or more statistics: inserting optimizer hints into the query based on the one or more statistics, Create or drop database indexes, Encode or unencode columns, Defragment or preload columns or tables, Materialize or dematerialize database view objects, Resize the database file or tablespace, or The one or more statistics are copied from a data dictionary of the database to a data dictionary of a second database.
15. The method of claim 1, wherein: The asynchronous job configuration database includes: The asynchronous operation detects whether the cause or condition of the event has stopped; The asynchronous job configures the database only if it has not stopped for the stated reason or condition.
16. The method of claim 1, wherein: The database is a pluggable database that can be stored within a container database.
17. The method of claim 16, wherein: The first plurality of rules further includes a second rule applicable to a second pluggable database stored within the same container database storing the pluggable database; The event identifies the pluggable database and indicates a cause or condition; The specific rule includes a first action template and a first condition template that match the cause or condition; The second rule includes a second action template and a second condition template matching the cause or condition; The method further comprises: receiving a second event from the particular DBMS, the second event identifying a second pluggable database and indicating the cause or condition; The second action template is applied based on detecting: the event and the second event include the cause or condition, An asynchronous job has completed configuring the pluggable database, and The cause or condition has not ceased.
18. The method of claim 16, wherein: Asynchronous jobs result in at least one of the following: cloning the pluggable database from a first container database on a first computer to a second container database on a second computer, Adjust the maximum connection rate of the database, Generate alerts based on dynamic conditions, Adjust the size of the redo log or fast recovery area, Apply software upgrades or patches to the database or the specific DBMS.
19. The method of claim 1, wherein: The specific rule specifies, based on a time period, an allocation amount of at least one of the following resources: a CPU, a persistent storage device, or a RAM.
20. The method of claim 1, wherein: Executing the particular rule includes generating a second event that causes execution of a second rule.
21. The method of claim 1, wherein: The event indicates at least one of the following: execution statistics of different queries, execution statistics of data manipulation language DML commands, execution statistics of data definition language DDL commands, execution statistics of specific queries, results of integrity checks on the database, usage of resources of the database exceeding a threshold, or availability of resources of the database exceeding a threshold.
22. The method of claim 21, further comprising: After the asynchronous job configures the database, the asynchronous job initiates a database health check that generates a second event, where the second event indicates at least one of the following: execution statistics of different queries, execution statistics of DML writes, execution statistics of DDL commands, execution statistics of specific queries, results of the integrity check of the database, usage of resources of the database exceeding a threshold, or availability of resources of the database exceeding a threshold.
23. The method of claim 1, wherein: The request includes at least one of the following: a Representational State Transfer REST request, a JavaScript Object Notation JSON, an Extensible Markup Language XML, or an executable script.
24. An apparatus for storing data, comprising means for performing the method of any one of claims 1-23.
