Computer-implemented method, computer program, and system for optimally performing stress tests on big data management systems (Workload generation for optimal stress tests on big data management systems)

Abstract

To provide a method, program and system for generating a workload for an optimal stress test of big data management systems.SOLUTION: In the method, a set of random test queries is generated and compiled to determine data points of features (e.g., a table type being queried) of the set of random test queries; a distance (e.g., Mahalanobis distance) is then measured between the data points of the features and the average of a distribution of data points corresponding to each same feature of an extracted feature set; each random test query whose distance exceeds a threshold distance is then ranked; and the ranked random test queries are then executed in order of rank. Those executed random test queries which resulted in an error (e.g., system failure) are added to a log, which is used to identify those queries to perform a stress test of the big data management system.SELECTED DRAWING: Figure 4A

Description

[Technical Field]

[0001] The present disclosure relates generally to stress testing, and more particularly to generating workloads for optimal stress testing of big data management systems. [Background technology]

[0002] In the context of database management system testing, stress testing refers to testing that places more emphasis on reliability, fault tolerance, and stability under heavy load / stress conditions, rather than on what would be considered correct behavior under normal circumstances. In particular, the goal of such testing may be to ensure that the database management system does not crash under conditions of insufficient computing resources (such as memory or disk space), abnormally high concurrency, or a denial-of-service attack. Summary of the Invention [Problem to be solved by the invention]

[0003] There is currently no method for stress testing big data management systems that makes optimal use of available time and resources without compromising the quality of the stress test. [Means for solving the problem]

[0004] In one embodiment of the present disclosure, a computer-implemented method for optimally performing stress tests on a big data management system includes generating a set of random test queries. The method further includes compiling the set of random test queries to extract access plan characteristics to determine data points for features of the set of random test queries. The method additionally includes measuring a distance between the data points for features of the set of random test queries and the mean of a distribution of data points corresponding to each same feature in the extracted feature set. The method further includes ranking each random test query whose distance exceeds a threshold distance. In addition, the method additionally includes executing the ranked random test queries in rank order. In addition, the method additionally includes adding executed ranked random test queries that resulted in errors to a log. The method further includes identifying queries for performing stress tests on the big data management system based on the log.

[0005] Other aspects of the above-described computer-implemented method embodiments are in the form of a system and in the form of a computer program product.

[0006] The foregoing has outlined rather generally the features and technical advantages of one or more embodiments of the present disclosure in order that the detailed description of the disclosure that follows may be better understood. Additional features and advantages of the present disclosure will be described hereinafter and may form the subject of the claims of the present disclosure. [Brief explanation of the drawings]

[0007] The present disclosure may be better understood from the following detailed description when considered in conjunction with the following drawings.

[0008] [Figure 1] 1 illustrates a communication system embodying the principles of the present disclosure, in accordance with one embodiment of the present disclosure.

[0009] [Figure 2] FIG. 1 is a diagram of software components of a test system used to identify an optimal set of queries to be used in stress testing a big data management system that optimally utilizes available time and resources without compromising the quality of the stress test, according to one embodiment of the present disclosure.

[0010] [Figure 3] FIG. 1 illustrates one embodiment of the present disclosure of a hardware configuration of a test system representative of a hardware environment for practicing the present disclosure.

[0011] [Figure 4A] 1 is a flowchart of a method for optimally performing stress tests on a big data management system according to one embodiment of the present disclosure. [Figure 4B] 1 is a flowchart of a method for optimally performing stress tests on a big data management system according to one embodiment of the present disclosure.

[0012] [Figure 5] 1 is a flowchart of a method for calculating weights for random test queries according to one embodiment of the present disclosure.

[0013] [Figure 6] 1 is a flowchart of a method for ranking random test queries whose distance exceeds a threshold distance based on weights assigned to the test queries according to one embodiment of the present disclosure.

[0014] [Figure 7] FIG. 1 illustrates an exemplary decision tree according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] As mentioned in the Background section, in testing database management systems, stress testing refers to testing that places more emphasis on reliability, fault tolerance, and stability under heavy load / stress conditions, rather than on what would be considered correct behavior under normal circumstances. In particular, the goal of such testing may be to ensure that the database management system does not crash under conditions of insufficient computing resources (such as memory or disk space), abnormally high concurrency, or denial-of-service attacks.

[0016] Stress testing of database management systems is generally performed with a workload (e.g., a Structured Query Language (SQL) workload) in three dimensions: concurrency (the number of clients executing simultaneous queries), duration (the duration of continuous workload execution), and complexity (the complexity of the queries executed).

[0017] Typically, a test workload suite (a collection of workloads intended to test a database management system by simulating real-world conditions to identify erroneous behavior, such as software errors) may execute various queries (e.g., SQL statements) that test the database management system, such as to identify SQL statements that result in failures due to code defects in particular features of the database system. Many of the queries may run for many hours (e.g., 12 hours) to successfully complete, thereby providing little value with respect to identifying software errors while simultaneously consuming valuable test and system resources.

[0018] Stress testing of database management systems continues to be a challenge in terms of generating data, populating database objects, and generating complex queries (e.g., complex SQL queries) to provide thorough quality assurance (QA) test coverage (percentage of software that is tested) to effectively test database management system software from a system / runtime perspective.

[0019] For example, with regard to generating queries to stress test a database management system, such queries need to be complex enough to test the system for reliability, fault tolerance, and stability, while at the same time not being so complex that the queries take a lot of time to execute.

[0020] Moreover, with the transition from traditional online transaction processing (OLTP) / online analytical processing (OLAP) systems to big data clusters (clusters of containers including data stores (e.g., Hadoop® Distributed File System) that store big data) managed by big data management systems (e.g., IBM® DB2® Big SQL), the amount of data on which tests must be run has grown by several orders of magnitude, thereby significantly reducing the number of test queries that can be run within a 24-hour period. "Big data," as used herein, refers to data sets whose size exceeds the ability of commonly used software tools to capture, curate, manage, and process them within an acceptable elapsed time.

[0021] As a result, stress tests for big data management systems cannot afford to run thousands of complex queries over long periods of time to test the reliability, fault tolerance, and stability of the database management system software. Therefore, there is currently no means of stress testing big data management systems that optimally uses available time and resources without compromising the quality of the stress test.

[0022] Embodiments of the present disclosure provide a means to identify an optimal set of queries to be used in stress testing against a big data management system (e.g., IBM® DB2® Big SQL) that optimally utilizes available time and resources without compromising the quality of the stress test.

[0023] In some embodiments of the present disclosure, the present disclosure comprises a computer-implemented method, system, and computer program product for optimally performing stress tests on a big data management system. In one embodiment of the present disclosure, a set of random test queries is generated. "Test query," as used herein, refers to a query used to test a big data management system by simulating real-world conditions to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in specific features of the database system. In one embodiment, such random test queries are generated based on utilizing tables containing fields for inclusion in the test queries. The set of random test queries is then compiled to extract access plan characteristics to determine data points for the set of random test queries (e.g., column types, queried table types, access plan operator types). "Characteristics," as used herein, refers to query characteristics such as column types, queried table types, access plan operator types, timerons (cost estimates based on an assessment of resources that will be used), total complexity cost divided into buckets, temporary table usage, final statistics, etc. "Access plan," as used herein, refers to specifying an order of operations for accessing data. "Access plan characteristics," as used herein, refer to the characteristics of such an access plan. Then, the distance (e.g., Mahalanobis distance, Euclidean distance) between the data points of the features of the set of random test queries and the mean of the distribution of data points corresponding to each same feature in the extracted Feature Set is measured. In one embodiment, the Feature Set is extracted from the optimizer plan and target table metadata, where the mean of the distribution of data points corresponding to each feature in the extracted Feature Set is measured. "Optimizer plan," as used herein, refers to the output of the optimizer, which describes the optimal method of execution."Target table metadata," as used herein, refers to metadata (data about data) that provides data definitions for the source data being consumed, including query time statistics. Each random test query whose distance exceeds a threshold distance is then ranked. The ranked random test queries are then executed in rank order. Executed random test queries that resulted in errors (e.g., identified errors such as system failures) are added to a log, which is used to identify those queries for performing stress tests on the big data management system. In this way, an optimal set of queries to be used in stress tests on the big data management system (e.g., IBM® DB2® Big SQL) is identified that optimally utilizes available time and resources without compromising the quality of the stress test.

[0024] In the following description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without such specific details. In other instances, well-known circuits are shown in block diagram form so as not to obscure the present disclosure with unnecessary detail. In most cases, details concerning timing considerations and the like are omitted unless such details are necessary to obtain a complete understanding of the present disclosure and are within the skill of those skilled in the relevant art.

[0025] Referring now in detail to the drawings, Figure 1 illustrates one embodiment of the present disclosure of a communication system 100 embodying the principles of the present disclosure. The communication system 100 includes a test system 101 connected to a big data management system 102 via a network 103.

[0026] Network 103 may be, for example, a local area network, a wide area network, a wireless wide area network, a circuit-switched telephone network, a Global System for Mobile Communications (GSM) network, a Wireless Application Protocol (WAP) network, a WiFi network, an IEEE 802.11 standard network, various combinations thereof, etc. Other networks (descriptions of which are omitted herein for brevity) may also be used in conjunction with system 100 of FIG. 1 without departing from the scope of this disclosure.

[0027] The test system 101 is configured to run system tests on the big data management system 102 to evaluate the system's compliance with its specified requirements. In one embodiment, such system tests include "stress tests." As used herein, "stress tests" refers to a collection of workloads generated by the test system 101, each including queries (e.g., Structured Query Language (SQL) queries) to be executed by the big data management system 102 to test the big data management system 102 in terms of reliability, fault tolerance, and stability under heavy load / stress conditions. In one embodiment, such tests are run against a big data cluster 104 (connected to the big data management system 102) managed by the big data management system 102 (e.g., IBM® DB2® Big SQL). In one embodiment, the big data management system 102 includes an SQL engine, which corresponds to software that recognizes and interprets SQL commands for accessing and interrogating the data in the big data cluster 104.

[0028] In one embodiment, the test system 101 generates an optimal set of queries (e.g., SQL queries) to be executed against the big data cluster 104 managed by the big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test. A more detailed description of generating such an optimal set of queries is provided further below. Additionally, a description of the software components of the test system 101 is provided further below in conjunction with FIG. 2, and a description of the hardware configuration of the test system 101 is provided further below in conjunction with FIG. 3.

[0029] The big data management system 102 is configured to interact with end users, applications, databases, and a big data cluster 104 to capture and analyze data. As used herein, the big data cluster 104 refers to a cluster of containers that includes a data store (e.g., a Hadoop® distributed file system) that stores big data. "Big data," as used herein, refers to data sets whose size exceeds the ability of commonly used software tools to capture, curate, manage, and process them within an acceptable elapsed time.

[0030] In one embodiment, the queries include a class of queries enabled to query a data store, such as a Hadoop® distributed file system, using well-known SQL queries. Such queries are referred to herein as "SQL-on-Hadoop." For clarity, it should be noted that when the term "query" is used herein, such queries include not only traditional queries, such as SQL queries, but also queries that execute SQL statements in a Hadoop® environment, such as SQL-on-Hadoop queries.

[0031] System 100 is not limited in scope to any one particular network architecture and may include any number of test systems 101, big data management systems 102, networks 103, and big data clusters 104.

[0032] A discussion of the software components used by the test system 101 to identify an optimal set of queries to be used in stress testing the big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test is provided below in connection with FIG. 2.

[0033] FIG. 2 is a diagram of software components of test system 101 (FIG. 1) used to identify an optimal set of queries to be used in stress testing big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test, according to one embodiment of the present disclosure.

[0034] 2 in conjunction with FIG. 1, the test system 101 includes a feature extractor 201 configured to identify data points (e.g., column types, queried table types, access plan operator types) corresponding to features extracted from queries, access plans / optimizer plans, and environment metadata that result in successful training workloads. "Features," as used herein, refer to characteristics of queries, such as column types, queried table types, access plan operator types, timers on (cost estimates based on evaluation of resources that will be used), total complexity cost divided into buckets, temporary table usage, final statistics, etc. In this way, the characteristics of the queries are translated as features of the dataset.

[0035] A "successful query," as used herein, refers to a query executed by the big data management system 102 that does not result in an error or failure (e.g., a software error). A "workload," as used herein, refers to the ability of the big data management system 102 to handle and process work. A "training workload," as used herein, refers to a workload used to determine which queries are optimal for running stress tests on the big data management system 102, where an "optimal query" refers to a query that optimally utilizes available time and resources without compromising the quality of the stress test. In one embodiment, features from the successful queries of the training workload, such as column types, table types queried, access plan operator types, total cost of complexity divided into timer-on-bucket, temporary table usage, final statistics, etc., are extracted by the feature extractor 201.

[0036] "Optimizer plan," as used herein, refers to the output of the optimizer 202 (query optimizer) of the test system 101, which describes an optimal method of execution. Such "output" may also be referred to herein as an "execution plan," an "optimizer plan," or an "access plan." Query optimizer 202, as used herein, refers to built-in software that determines the most efficient way to execute a query, such as an SQL statement. In one embodiment, the query optimizer 202 generates a set of potential plans for an SQL statement based on available access paths and hints. The query optimizer 202 estimates the cost of each plan based on statistics in a data dictionary. The statistics include information about the data distribution and storage characteristics of the tables, indexes, and partitions accessed by the statement. In one embodiment, the cost is an estimate proportional to the expected resource usage required to execute the statement using a particular plan. In one embodiment, the query optimizer 202 calculates the cost of access paths and join orders based on estimated computer resources (e.g., input / output, processing units, memory, etc.). Sequential plans with higher costs take longer to execute than sequential plans with smaller costs. When using parallel plans, resource usage is not directly related to elapsed time.

[0037] In one embodiment, the query optimizer 202 compares plans and selects the plan with the lowest cost. The output of such analysis is an "execution plan," or "optimizer plan," or "access plan," which describes the optimal method of execution. For example, a plan may indicate the combination of steps used to execute a SQL statement.

[0038] In one embodiment, query characteristics (e.g., column types, queried table types, access plan operator types, total cost of complexity divided into timer on buckets, temporary table usage, final statistics, etc.) are included in the optimizer plan. Such features are extracted by the feature extractor 201.

[0039] Additionally, features can be extracted from "environment metadata," which describes system-wide characteristics such as base locations (including tables), data platforms, SQL toolkit locations, etc., and query time statistics. Such features, such as query time statistics, are extracted by feature extractor 201.

[0040] Additionally, in one embodiment, feature extractor 201 is configured to extract a feature set from the optimizer plan and target table metadata. "Target table," as used herein, refers to a table where data goes or where an action takes place. "Target table metadata," as used herein, refers to metadata (data about data) that provides a data definition for the source data being consumed, including query time statistics. Such features are extracted by feature extractor 201 from the target table metadata.

[0041] In one embodiment, feature extractor 201 utilizes natural language processing to identify features to be extracted. In one embodiment, feature extractor 201 extracts such features described above by identifying keywords indicative of such features, where such keywords are listed in a data structure (e.g., a table). In one embodiment, such a data structure is stored in a storage device (e.g., memory, disk drive) of test system 101.

[0042] In one embodiment, the feature extractor 201 is configured to fetch the access plan of the SQL statement and convert the fetched access plan into a set of features that it contains.

[0043] The test system 101 further comprises a data analysis tool 203 configured to determine a mean of a distribution of data points corresponding to features. For example, the feature extractor 201 extracts data points for column type, queried table type, access plan operator type, total cost of complexity divided into timer on buckets, temporary table usage, final statistics, etc. Such data points may be analyzed by the data analysis tool 203 to derive a mean of the distribution of the dataset.

[0044] Additionally, in one embodiment, the data analysis tool 203 is configured to measure the distance between a data point corresponding to each feature obtained from extracting access plan characteristics of the queries of the test suite and the mean of the distribution of data points corresponding to each such feature. In one embodiment, such distance corresponds to a multivariate distance, such as Euclidean distance. In one embodiment, the data analysis tool 203 measures the Mahalanobis distance between a data point corresponding to each feature obtained from executing the queries of the test suite and the mean of the distribution of data points corresponding to each such feature. "Test suite" or "test workload suite," as used herein, refers to a collection of workloads (including queries) intended to test a big data management system by simulating real-world conditions to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in specific features of the database system.

[0045] Additionally, in one embodiment, the data analysis tool 203 is configured to set the farthest distance between a data point corresponding to a feature (e.g., an access plan operator type) and the mean of the distribution of data points for the feature as a "threshold distance" for such feature. Such a threshold will be used to identify queries that may be utilized to perform stress tests on the big data management system 102, as discussed further below.

[0046] Examples of software tools utilized by the data analysis tool 203 to determine the mean of the distribution of data points corresponding to features include, but are not limited to, Minitab®, IBM® SPSS Statistics, RStudio®, OriginPro®, EViews®, etc.

[0047] The test system 101 additionally includes a query generator 204 configured to generate a set of random test queries. A “test query,” as used herein, refers to a query used to test a big data management system (e.g., the big data management system 102) by simulating real-world situations to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in specific features of the database system. In one embodiment, such random test queries are generated based on utilizing tables containing fields for inclusion in the test queries. In one embodiment, such fields are populated by an expert. In one embodiment, such tables correspond to data structures present in a storage device (e.g., memory, disk drive) of the test system 101. In one embodiment, the queries are randomly generated by the query generator 204 by randomly selecting fields from the tables described above for inclusion in each query of the set of random test queries. In one embodiment, each of the random test queries has the same number of fields. In one embodiment, criteria (e.g., union queries) are added to the fields by an expert.

[0048] In one embodiment, the query generator 204 generates a set of random test queries via a specification by example (SBE) method, which allows the query generator 204 to create queries based on example entity instances. For example, representative test queries can be submitted to the query generator 204, which may use an SBE method to identify "similar" queries stored in a large database. In one embodiment, such a search is based on comparing query vectors. In one such embodiment, a query may be represented as a vector of identifiers (e.g., index terms) using a vector space model.

[0049] The test system 101 further includes a query executor 205 configured to execute or compile queries. In one embodiment, the query executor 205 is configured to compile a set of random test queries to extract access plan characteristics, which are used by the feature extractor 201 to determine data points for features (e.g., column types, access plan operators) as described above. An "access plan," as used herein, refers to specifying an order of operations for accessing data. For example, an access plan allows viewing global information such as statistics for selected tables, indexes, or columns, characteristics for operators, tablespace and function statistics, and configuration parameters related to optimization. An "access plan characteristic," as used herein, refers to a characteristic of such an access plan. As described above, the feature extractor 201 is configured to extract query characteristics from an optimizer plan or access plan that includes query characteristics.

[0050] Examples of software tools that may be utilized by the query execution unit 205 to compile a set of random test queries include, but are not limited to, Microsoft® SQL Server Management Studio (SSMS), SQLGate®, Postico, PopSQL®, Admin Editor, Valentina, DBeaver®, SQuirreL SQL, etc.

[0051] Additionally, the test system 101 includes a query prioritization engine 206 configured to prioritize queries, such as random test queries, whose distance (the distance between a data point of a feature and the mean of the distribution of data points corresponding to such feature) exceeds a threshold distance. In one embodiment, such prioritization involves ranking the random test queries, such as based on expected runtime duration. In one embodiment, the runtime duration is predicted using a decision tree, where the decision tree includes variables, and elements of the variables include features, as discussed further below. Furthermore, in one embodiment, the variables include characteristics of the query, such as table type, operator, interrupt, column type, table metadata, etc.

[0052] In one embodiment, the query prioritization engine 206 ranks the random test queries based on a weight assigned to each of the random test queries, as discussed further below. In one embodiment, such weights are determined by the solution of a linear system.

[0053] Additionally, the test system 101 comprises a query selector 207 configured to identify and select an optimal set of queries (e.g., SQL queries) to be executed against the big data cluster 104 managed by the big data management system 102 to perform the stress test, where such queries optimally utilize available time and resources without compromising the quality of the stress test. In one embodiment, such optimal set of queries corresponds to ranked random test queries that produced errors when executed. In one embodiment, such queries that produced errors when executed are stored in a log (referred to herein as a "test log").

[0054] Further description of these and other features is provided below in connection with a discussion of how to optimally perform stress tests on big data management systems.

[0055] Prior to discussing how to optimally perform stress tests on a big data management system, a description of the hardware configuration of test system 101 (FIG. 1) is provided below in conjunction with FIG.

[0056] Now, referring to FIG. 3 in conjunction with FIGS. 1-2, FIG. 3 illustrates one embodiment of the present disclosure of a hardware configuration for test system 101 (FIG. 1) that represents a hardware environment for practicing the present disclosure.

[0057] Test system 101 includes a processor 301 connected to various other components by a system bus 302. An operating system 303 executes on processor 301 and provides control of and coordinates the functions of the various components of FIG. 3 . Applications 304 according to the principles of the present disclosure execute in conjunction with operating system 303 and provide calls to operating system 303, where the calls implement various functions or services to be performed by application 304. Application 304 may include, for example, feature extractor 201, query optimizer 202, data analysis tool 203, query generator 204, query executor 205, query prioritization engine 206, and query selector 207. Additionally, application 304 may include, for example, a program that generates an optimal set of queries (e.g., SQL queries) to be executed against big data cluster 104 managed by big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test, as discussed further below in connection with FIGS. 4A-4B and 5-7.

[0058] 3, read-only memory ("ROM") 305 is connected to system bus 302 and includes a basic input / output system ("BIOS") that controls certain basic functions of test system 101. Random access memory ("RAM") 306 and disk adapter 307 are also connected to system bus 302. It should be noted that software components including operating system 303 and applications 304 may be loaded into RAM 306, which may be the main memory of test system 101 for execution. Disk adapter 307 may be an integrated drive electronics ("IDE") adapter that communicates with disk unit 308, e.g., a disk drive. It should be noted that the program that generates the optimal set of queries (e.g., SQL queries) to be executed against the big data cluster 104 managed by the big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test may reside in the disk unit 308 or in the application 304, as discussed further below in connection with Figures 4A-4B and 5-7.

[0059] The test system 101 may further include a communications adapter 309 connected to the bus 302. The communications adapter 309 interconnects the bus 302 with an external network (e.g., network 103 in FIG. 1) for communication with other devices, such as the big data management system 102.

[0060] In one embodiment, the application 304 of the test system 101 includes the following software components: feature extractor 201, query optimizer 202, data analysis tool 203, query generator 204, query executor 205, query prioritization engine 206, and query selector 207. In one embodiment, such components may be implemented in hardware, in which case such hardware components would be connected to bus 302. The functions described above performed by such components are not comprehensive computer functions. As a result, the test system 101 is a specific machine that results in implementing specific, non-comprehensive computer functions.

[0061] In one embodiment, the functionality of such software components of test system 101 (e.g., feature extractor 201, query optimizer 202, data analysis tool 203, query generator 204, query executor 205, query prioritization engine 206, and query selector 207), including the functionality to generate an optimal set of queries (e.g., SQL queries) to be executed against big data cluster 104 managed by big data management system 102 that optimally utilizes available time and resources without compromising the quality of the stress test, may be embodied in an application specific integrated circuit.

[0062] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present invention.

[0063] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium in each computing / processing device for storage.

[0065] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0066] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0068] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be implemented as a single step, or may be executed concurrently, substantially concurrently, partially, or fully overlapping in time, or the blocks may even be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0070] As discussed above, stress testing of database management systems continues to be a challenge in terms of generating data, populating database objects, and generating complex queries (e.g., complex SQL queries) to provide thorough quality assurance (QA) test coverage (percentage of software being tested) to effectively test database management system software from a system / runtime perspective. For example, with regard to generating queries to stress test database management systems, such queries need to be complex enough to test the system for reliability, fault tolerance, and stability, while at the same time not being so complex that they require a significant amount of time to execute. Furthermore, with the transition from traditional online transaction processing (OLTP) / online analytical processing (OLAP) systems to big data clusters (clusters of containers that include data stores (e.g., Hadoop® distributed file systems) that store big data) managed by database management systems (e.g., IBM® DB2® Big SQL), the amount of data against which tests must be run has grown by several orders of magnitude, thereby significantly reducing the number of test queries that can be executed within a 24-hour period. "Big data," as used herein, refers to data sets whose size exceeds the ability of commonly used software tools to capture, curate, manage, and process them within an acceptable elapsed time. As a result, stress tests for big data management systems cannot afford to run thousands of complex queries over extended periods of time to test the reliability, fault tolerance, and stability of the database management system software. Therefore, there is currently no means of stress testing big data management systems that optimally uses available time and resources without compromising the quality of the stress test.

[0071] Embodiments of the present disclosure, as discussed below in connection with FIGS. 4A-4B and 5-7, provide a means for identifying an optimal set of queries to be used in a stress test on a big data management system (e.g., IBM® DB2® Big SQL) that optimally utilizes available time and resources without compromising the quality of the stress test. FIGS. 4A-4B are a flowchart of a method for optimally performing stress tests on a big data management system. FIG. 5 is a flowchart of a method for calculating weights for random test queries. FIG. 6 is a flowchart of a method for ranking random test queries whose distance exceeds a threshold distance based on weights assigned to the test queries. FIG. 7 shows an exemplary decision tree.

[0072] As mentioned above, FIGS. 4A-4B are a flowchart of a method 400 for optimally performing stress tests on a big data management system, according to one embodiment of the present disclosure.

[0073] Referring to FIG. 4A in conjunction with FIGS. 1-3, in step 401, the feature extraction unit 201 of the test system 101 identifies data points (e.g., column types, queried table types, access plan operator types) corresponding to features extracted from queries, optimizer plans, and environment metadata that result in successful training workloads.

[0074] As described above, "features," as used herein, refer to characteristics of a query, such as column type, queried table type, access plan operator type, timer on (a cost estimate based on an evaluation of the resources that will be used), total cost of complexity divided into buckets, temporary table usage, final statistics, etc. "Successful queries," as used herein, refer to queries executed by the big data management system 102 that do not result in errors or failures (e.g., software errors). "Workload," as used herein, refers to the ability of the big data management system 102 to handle and process work. "Training workload," as used herein, refers to a workload used to determine which queries are optimal for running stress tests against the big data management system 102, where "optimal queries" refer to queries that optimally utilize available time and resources without compromising the quality of the stress tests. In one embodiment, features from the queries that result in the success of the training workload, such as column types, queried table types, access plan operator types, total cost of complexity divided into timer-on-buckets, temporary table usage, final statistics, etc., are extracted by feature extractor 201.

[0075] "Optimizer plan," as used herein, refers to the output of optimizer 202 (query optimizer), which describes an optimal method of execution. Such "output" may also be referred to herein as an "execution plan," an "optimizer plan," or an "access plan." Query optimizer 202, as used herein, refers to built-in software that determines the most efficient way to execute a query, such as an SQL statement. In one embodiment, query optimizer 202 generates a set of potential plans for an SQL statement based on available access paths and hints. Query optimizer 202 estimates the cost of each plan based on statistics in a data dictionary. The statistics include information about the data distribution and storage characteristics of the tables, indexes, and partitions accessed by the statement. In one embodiment, the cost is an estimate proportional to the expected resource usage required to execute the statement using a particular plan. In one embodiment, the query optimizer 202 calculates the cost of access paths and join orders based on estimated computer resources (e.g., input / output, processing units, memory, etc.). Sequential plans with higher costs take longer to execute than sequential plans with smaller costs. When using parallel plans, resource usage is not directly related to elapsed time.

[0076] In one embodiment, the query optimizer 202 compares plans and selects the plan with the lowest cost. The output of such analysis is an "execution plan," or "optimizer plan," or "access plan," which describes the optimal method of execution. For example, a plan may indicate the combination of steps used to execute a SQL statement.

[0077] In one embodiment, query characteristics (e.g., column types, queried table types, access plan operator types, total cost of complexity divided into timer on buckets, temporary table usage, final statistics, etc.) are included in the optimizer plan. Such features are extracted by the feature extractor 201.

[0078] Additionally, features can be extracted from "environment metadata," which describes system-wide characteristics such as base locations (including tables), data platforms, SQL toolkit locations, etc., and query time statistics. Such features, such as query time statistics, are extracted by feature extractor 201.

[0079] Additionally, in one embodiment, feature extractor 201 is configured to extract feature sets from the optimizer plan and target table metadata. "Target table," as used herein, refers to a table where data goes or an action takes place. "Target table metadata," as used herein, refers to metadata (data about data) that provides a data definition for the source data being consumed, including query time statistics.

[0080] In one embodiment, feature extractor 201 utilizes natural language processing to identify features to be extracted. In one embodiment, feature extractor 201 extracts such features described above by identifying keywords indicative of such features, where such keywords are listed in a data structure (e.g., a table). In one embodiment, such a data structure is stored in a storage device (e.g., memory 305, disk drive 308) of test system 101.

[0081] In one embodiment, the feature extractor 201 is configured to fetch the access plan of the SQL statement and convert the fetched access plan into a set of features that it contains.

[0082] In step 402, the data analysis tool 203 of the test system 101 determines the mean of the distribution of data points corresponding to each feature. For example, the feature extractor 201 extracts data points for column type, table type queried, access plan operator type, total cost of complexity divided into timer on buckets, temporary table usage, final statistics, etc. Such data points may be analyzed by the data analysis tool 203 to derive the mean of the distribution of the data set.

[0083] As noted above, examples of software tools utilized by the data analysis tool 203 to determine the mean of the distribution of data points corresponding to features include, but are not limited to, Minitab®, IBM® SPSS Statistics, RStudio®, OriginPro®, EViews®, etc.

[0084] In step 403, the test system 101 receives a query for a test suite, such as from an expert. A "test suite" or "test workload suite," as used herein, refers to a collection of workloads (including queries) intended to test a big data management system by simulating real-world conditions to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in particular features of the database system.

[0085] In step 404, the query execution unit 205 of the test system 101 compiles the received queries of the test suite to extract access plan characteristics, which are used by the feature extraction unit 201 to determine data points of features (e.g., column type, queried table type, access plan operator type), whereupon the data analysis tool 203 of the test system 101 measures the distance between the data point corresponding to each feature obtained from the compilation of the queries of the test suite and the mean of the distribution of data points corresponding to each such feature (determined in step 402).

[0086] In one embodiment, the data analysis tool 203 measures the multivariate distance (e.g., Mahalanobis distance, Euclidean distance) between the data points corresponding to each feature obtained from the compilation of the test suite's queries and the mean of the distribution of data points corresponding to each such feature.

[0087] As noted above, examples of software tools that may be utilized by query execution unit 205 to compile a set of random test queries include, but are not limited to, Microsoft® SQL Server Management Studio (SSMS), SQLGate®, Postico, PopSQL®, Admin Editor, Valentina, DBeaver®, SQuirreL SQL, etc.

[0088] In step 405, the data analysis tool 203 of the test system 101 sets the farthest distance between a data point corresponding to a feature (a data point of the feature for an executed query of the test suite) and the mean of the distribution of data points for the feature as a "threshold distance" for such feature. Such a threshold may be used to identify queries that may be utilized to perform stress tests on the big data management system 102. For example, such a threshold distance may be used to identify outlier queries that have a high probability of causing a query or system failure, as discussed further below.

[0089] In step 406, the query generator 204 of the test system 101 generates a set of random test queries. As mentioned above, "test query," as used herein, refers to a query used to test a big data management system (e.g., the big data management system 102) by simulating real-world situations to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in specific features of the database system. In one embodiment, such random test queries are generated based on utilizing tables containing fields for inclusion in the test queries. In one embodiment, such fields are populated by an expert. In one embodiment, such tables correspond to data structures present in a storage device (e.g., memory 305, disk drive 308) of the test system 101. In one embodiment, the queries are randomly generated by the query generator 204 by randomly selecting fields from the tables described above for inclusion in each query of the set of random test queries. In one embodiment, each of the random test queries has the same number of fields. In one embodiment, criteria (e.g., union queries) are added to the fields by the expert.

[0090] In one embodiment, the query generator 204 generates a set of random test queries via a specification-by-example (SBE) method, which allows the query generator 204 to create queries based on example entity instances. For example, representative test queries can be submitted to the query generator 204, which may use an SBE method to identify "similar" queries stored in a large database. In one embodiment, such a search is based on comparing query vectors. In one such embodiment, a query may be represented as a vector of identifiers (e.g., index terms) using a vector space model.

[0091] In step 407, the feature extractor 201 of the test system 101 extracts a feature set from the optimizer plan and target table metadata.

[0092] As mentioned above, in one embodiment, query characteristics (e.g., column types, queried table types, access plan operator types, total cost of complexity divided into timer-on-buckets, temporary table usage, final statistics, etc.) are included in the optimizer plan. Such features are extracted by feature extractor 201.

[0093] "Target table," as used herein, refers to a table where data goes or an action takes place. "Target table metadata," as used herein, refers to metadata (data about data) that provides a data definition for the source data being consumed, including query time statistics. Such features are extracted from the target table metadata by feature extractor 201.

[0094] In step 408 , the data analysis tool 203 of the test system 101 determines the mean of the distribution of data points corresponding to each feature in the feature set extracted as described above in connection with step 402 .

[0095] In step 409, the query execution unit 205 of the test system 101 compiles random test queries to extract access plan characteristics, which are used by the feature extraction unit 201 to determine data points for the query's features, and then causes the data analysis tool 203 of the test system 101 to measure the distance between the data point corresponding to each such feature (e.g., column type, queried table type, access plan operator type) and the mean of the distribution of data points corresponding to that same feature in the extracted feature set.

[0096] In one embodiment, the query execution unit 205 is configured to compile a set of random test queries for the feature extraction unit 201 to determine data points for features (e.g., column types, queried table types, access plan operator types), as described above, such as in stage 404.

[0097] Additionally, data analysis tool 203 measures the distance between the data point corresponding to each such feature (e.g., column type, queried table type, access plan operator type) and the mean of the distribution of data points corresponding to that same feature in the extracted feature set, in the same manner as described above in connection with stage 404. In one embodiment, such distance corresponds to a multivariate distance (e.g., Mahalanobis distance, Euclidean distance).

[0098] In step 410, a determination is made by data analysis tool 203 of test system 101 as to whether the measured distance (e.g., Mahalanobis distance for a feature such as a queried table type) exceeds a threshold distance for such feature (e.g., a queried table type) (determined in step 405). Such a determination is performed to identify queries or queries that have a high probability of causing system failure.

[0099] If the measured distance does not exceed the threshold distance for such feature, then in step 411, the data analysis tool 203 of the test system 101 discards the test query. Such a query is discarded because it provides little value in terms of stress testing. Such a query provides little value in terms of stress testing because other queries that provide similar coverage (e.g., quality assurance coverage) will be executed. In other words, by discarding such a query, embodiments of the present disclosure prevent the execution of similar test queries that provide little value in terms of overload.

[0100] 4B in conjunction with FIGS. 1-3, if, on the other hand, the measured distance exceeds the threshold distance for such feature, then in step 412, the data analysis tool 203 of the test system 101 feeds the test query to the query prioritization engine 206 for priority. In one embodiment, such queries will be ranked by the query prioritization engine 206 according to coverage complexity and estimated query duration.

[0101] In step 413, a determination is made by test system 101 as to whether additional random test queries are to be compiled. If there are additional random test queries to be compiled, query executor 205 compiles another random test query in step 409.

[0102] On the other hand, if there are no additional random test queries to be compiled, then in step 414 the query prioritization engine 206 of the test system 101 ranks the random test queries to be prioritized.

[0103] In one embodiment, the query prioritization engine 206 ranks the random test queries based on expected runtime duration, where the expected runtime duration is predicted for portions of the query using a decision tree (predicting the runtime duration of the query when the query is executed against the big data clusters 104 managed by the big data management system 102), where the decision tree includes variables and elements of the variables include features. A discussion of an embodiment of the query prioritization engine 206 ranking the random test queries is discussed below in connection with FIGS. 5-7.

[0104] FIG. 5 is a flowchart of a method 500 for calculating weights for random test queries according to one embodiment of the present disclosure.

[0105] In one embodiment, the query prioritization engine 206 uses the runtime duration of a small number of selected random test queries to predict the runtime duration for other queries. In one embodiment, a decision tree is utilized to make such predictions. In one embodiment, the design of the decision tree is based on the functional specification of a particular query. For modeling classification, the input domain is divided into mutually exclusive subsets (so-called classification classes). In the decision tree definition, a query is defined by combining classification classes from different classifications. In one embodiment, a classification contains only mutually exclusive features, so a query cannot contain multiple features of one classification.

[0106] As used herein, "classification classes" are also referred to as "variables" (e.g., queried table types, operators, interrupts, etc.), and their elements are referred to as "features" (e.g., Apache® Parquet, TEMP operations, UNION operations, limit to N rows, etc.). A "query," as discussed herein, represents a set of n features, one for each variable. Query runtime prediction and query prioritization are used to determine the smallest possible subset of queries that covers all combinations of features specified by the coverage metric with at least one query.

[0107] Classification classes are used because there is a strong correlation between the characteristics of a query (e.g., an SQL query) and the code path along which the query is executed in the software of the big data management system. As a result, controlling the key characteristics of a query also means controlling which parts of the software code of the big data management system are invoked.

[0108] Examples of important characteristics of queries, such as SQL-on-Hadoop queries, that are transformed into dataset features include: table type (there is a strong correlation between table types because different table types invoke different SQL engine input / output readers to process and cover different code paths), operators (SQL query planning operators determine the exact stages of query execution, where each operator covers different functionality within the runtime engine), preemption, column type (row operations, predicate filtering, and other low-level functionality depend on column types, making SQL query execution and quality assurance coverage highly correlated), environment / table metadata (partition data and table cardinality impact runtime coverage and are indicators of query complexity), total complexity cost, and temporary table usage (key indicators of complexity and coverage for optimizer planning are correlated with the level of quality assurance coverage).

[0109] The design of such a decision tree is discussed below in connection with FIG.

[0110] 5 in conjunction with FIGS. 1-3 and 4A-4B, in step 501, the query prioritization engine 206 of the test system 101 selects a template query. For example, a query including features y1, y2, and y3 is selected, where, for example, y1="Relational", y2="GRPBY", and y3="Fetch first N rows". In one embodiment, the query prioritization engine 206 selects the template query from a set of template queries stored in a data structure (e.g., a table) populated by an expert. In one embodiment, the data structure is stored in a storage device (e.g., memory 305, disk drive 308) of the test system 101. Such a template query is designed to include features that are included in most queries utilized to perform stress testing of big data management systems.

[0111] In step 502, the query prioritization engine 206 of the test system 101 sets the class weights of the classification classes (variables) in the template query to be 1. For example, the query prioritization engine 206 sets the class weights (cw) of the classification classes in the template query to be 1, i.e., y1 (cw=1), y2 (cw=1), and y3 (cw=1). "Class weight," as used herein, refers to a value that indicates the importance of a classification class. For example, the higher the class weight, the more important that classification class is. In one embodiment, the class weights are normalized between values ​​of 0 and 1.

[0112] In step 503, the query prioritization engine 206 of the test system 101 generates a query (referred to herein as a "sigma query") by replacing the reference classification class of the template query with another classification class c of the same modeled classification. For example, a reference classification class (e.g., y3) is replaced with another classification class (e.g., y4) of the same modeled classification (e.g., σc) generating a sigma σ query (e.g., δY2=(y1, y2, y4)).

[0113] In step 504, the query prioritization engine 206 of the test system 101 measures the query runtime t_qt of the template query. In one embodiment, the query prioritization engine 206 utilizes any of the following tools to measure query runtime, such as SolarWinds® Database Performance Analyzer, AppOptics™ APM, ManageEngine® Applications Manager, Redgate® SQL Monitor, EverSQL, Idera® DB Optimizer, dbForge Studio, etc.

[0114] In step 505, the query prioritization engine 206 of the test system 101 measures the query runtime t_qc for each sigma query σc (e.g., δY2 = (y1, y2, y4)). As mentioned above, in one embodiment, the query prioritization engine 206 utilizes any of the following tools to measure query runtime, such as SolarWinds® Database Performance Analyzer, AppOptics™ APM, ManageEngine® Applications Manager, Redgate® SQL Monitor, EverSQL, Idera® DB Optimizer, dbForge Studio, etc.

[0115] In step 506, the query prioritization engine 206 of the test system 101 sets the weight of c equal to t_qc / t_qt. For example, if t_qt=4s and t_Y2(t_qc)=8s, then the weight of y4 is cw=2.

[0116] In step 507, the query prioritization engine 206 of the test system 101 calculates the total weight of a random query as the product of the weights of its classification classes. As a result, queries can be prioritized based on their total weight.

[0117] In one embodiment, the calculation of the weights is determined by the solution of a linear system with N=("total classification classes" - "total modeled classifications") solutions. In one embodiment, such N solutions correspond to weights of classification classes outside the reference classification class. A discussion regarding calculating weights in such a method is provided below in connection with FIG. 6.

[0118] FIG. 6 is a flowchart of a method 600 for ranking random test queries whose distance exceeds a threshold distance based on weights assigned to the test queries according to one embodiment of the present disclosure.

[0119] Referring to FIG. 6 in conjunction with FIGS. 1-3 and 4A-4B, in step 601, the query generator 204 of the test system 101 generates N+1 queries.

[0120] In one embodiment, such queries are generated by the query generator 204 based on utilizing tables containing fields for inclusion in the query. In one embodiment, such fields are populated by an expert. In one embodiment, such tables correspond to data structures residing in a storage device (e.g., memory 305, disk drive 308) of the test system 101. In one embodiment, the queries are generated by the query generator 204 by randomly selecting fields from the tables described above for inclusion in each of the N+1 queries. In one embodiment, each of the queries has the same number of fields. In one embodiment, criteria (e.g., union queries) are added to the fields by the expert.

[0121] In one embodiment, the query generator 204 generates a set of N+1 queries via a specification-by-example (SBE) method, which allows the query generator 204 to create queries based on example entity instances. For example, representative test queries can be submitted to the query generator 204, and the query generator 204 may use an SBE method to identify "similar" queries stored in a large database. In one embodiment, such a search is based on comparing query vectors. In one such embodiment, a query may be represented as a vector of identifiers (e.g., index terms) using a vector space model.

[0122] Examples of software tools that may be utilized by the query generator 204 to generate queries include, but are not limited to, Microsoft® SQL Server Management Studio (SSMS), SQLGate®, Postico, PopSQL®, Admin Editor, Valentina, DBeaver®, SQuirreL SQL, and the like.

[0123] In step 602, the query prioritization engine 206 of the test system 101 sets the first query to be a template query, where all classification classes in the query are assigned a weight of one.

[0124] In step 603, the query prioritization engine 206 of the test system 101 obtains a weight for the remaining query based on its query runtime (the query runtime of the remaining query). In one embodiment, the shorter the query runtime, the higher the weight. As mentioned above, in one embodiment, the query prioritization engine 206 utilizes any of the following tools to measure query runtime, such as SolarWinds® Database Performance Analyzer, AppOptics™ APM, ManageEngine® Applications Manager, Redgate® SQL Monitor, EverSQL, Idera® DB Optimizer, dbForge Studio, etc.

[0125] In step 604, the query prioritization engine 206 of the test system 101 uses the obtained weights to calculate a determinant that defines the linear system, as discussed further below in connection with FIG.

[0126] In step 605, a determination is made by the query prioritization engine 206 of the test system 101 as to whether the uniqueness of the linear system has been confirmed, as discussed further below in connection with FIG.

[0127] If the uniqueness of the linear system is confirmed, then in step 606, the query prioritization engine 206 of the test system 101 stores the query in a list of queries. In one embodiment, such a list corresponds to a data structure (e.g., a table) stored in a storage device (e.g., memory 305, disk drive 308) of the test system 101.

[0128] However, if the uniqueness of the linear system is not confirmed or upon storing the query in the list of queries, in step 607 a determination is made by the query prioritization engine 206 of the test system 101 as to whether there are any further queries to be analyzed.

[0129] If there are more queries to be analyzed, the query prioritization engine 206, in step 603, obtains a weight for the next remaining query based on its query runtime.

[0130] However, if there are no further queries to be analyzed, then in step 608, the query prioritization engine 206 of the test system 101 ranks the queries in the list of queries based on query runtime. For example, queries with smaller runtime durations are ranked higher than queries with larger runtime durations.

[0131] 7, which illustrates an exemplary decision tree according to one embodiment of the present disclosure, decision tree pattern set 700 includes layers of classification classes (variables) 701A-701C (identified as "table type variables," "operator variables," and "interrupt variables," respectively). Classification classes 701A-701C may each be referred to collectively as classification classes 701 or individually as classification classes 701.

[0132] 7, each classification class 701 may include one or more elements (features). For example, classification class 701A includes elements y1 702A, y6 702B, and y8 702C. Classification class 701B includes elements y2 702D, y7 702E, and y9 702F. Furthermore, classification class 701C includes elements y3 702G and y5 702H.

[0133] Referring to FIG. 7 in conjunction with FIG. 6, the decision tree in FIG. 7 implies the following: the first 6=(8-3)+1 queries (assuming N=5 to generate N+1 queries) in the set of all pair-sorted sequences are given by QR=(y1,y2,y3), Q1=(y1,y4,y5), Q2=(y6,y2,y5), Q3=(y6,y4,y3), Q4=(y8,y9,y3), and Q5=(y8,y4,y5).

[0134] In one embodiment, the class weight (cw) for y1 is given by cw_y1, the class_weight(cw) for y2 is given by cw_y2, and so on, where all classification classes in the query are assigned a weight of 1, e.g., (cw_y1=cw_y2=cw_3)=1. Then, by considering the query runtime as t_QR, t_Q1, ..., t_Q5, Q1 gets the weight (cw_y1*cw_y2*cw_y3)=(t_Q1 / t_QR), and so on. To solve the linear system in exponential form, by taking into account that (cw_y1=cw_y2=cw_y3)=1, we get the following equation: log(cw_y4)+log(cw_y5)=log(t_Q1 / t_QR), and so on. The linear system is then defined by the following matrices: [ 0 0 1 0 1 | log(t_Q1 / t_QR) 1 0 0 0 1 | log(t_Q2 / t_QR) 1 0 1 0 0 | log(t_Q3 / t_QR) 0 1 0 1 0 | log(t_Q4 / t_QR) 0 1 1 0 0 | log(t_Q5 / t_QR) ]

[0135] The calculated determinant of the right-hand side is 2. As a result, the uniqueness of the linear system solution is confirmed. For example, we measure the runtimes as t_QR=1s, t_Q1=2s, t_Q2=109s, t_Q3=12s, t_Q4=82s, and t_Q5=60s, and predict the same results as above for weights. Then, query prioritization is achieved from the minimum query runtime value to the maximum query runtime value.

[0136] Therefore, queries can be executed from minimum to maximum query runtimes, as Quality Assurance (QA) coverage allows.

[0137] Furthermore, as shown in Figure 7, the runtime duration is predicted based on the importance of the classification tree elements (i.e., the weights of the selected test cases), which are calculated via a linear system as described above.

[0138] Returning to FIG. 4B in conjunction with FIGS. 1-3 and 5-7, in step 415, query execution unit 205 of test system 101 executes the test queries in rank order. Examples of software tools utilized by query execution unit 205 to execute queries include, but are not limited to, Microsoft® SQL Server Management Studio (SSMS), SQLGate®, Postico, PopSQL®, Admin Editor, Valentina, DBeaver®, SQuirreL SQL, etc. The following steps discuss the execution of one such test query.

[0139] At step 416, a determination is made by query execution unit 205 of test system 101 as to whether the test query execution was successful (ie, no errors such as system failures were identified).

[0140] If the test query execution is successful, then in step 417, the query execution unit 205 of the test system 101 updates the training set of queries. Such training set is updated to reflect that no errors were identified and is used to influence future query generation. As discussed above, the principles of the present disclosure are directed to generating an optimal set of queries for testing the big data management system 102 that have a high probability of causing a query or system failure while optimizing the use of available time and resources. As a result, successful test queries are less likely to be utilized in subsequently generated sets of random test queries.

[0141] However, if the execution of the test query is not successful (i.e., an error such as a system failure is identified), then in step 418, the query executor 205 of the test system 101 adds the query to a log (referred to herein as a "test log") that maintains a list of queries that resulted in errors. Such a list of queries includes queries that have a high probability of forcing an error, such as a system or query failure.

[0142] In step 419, a determination is made by query execution unit 205 of test system 101 as to whether additional test queries need to be executed. If additional test queries need to be executed, query execution unit 205 of test system 101 executes the next test query in rank order in step 415.

[0143] However, if there are no additional test queries to be executed, then in step 420, the query selector 207 of the test system 101 uses the list of queries stored in the test log to identify an optimal set of queries for performing stress tests on the big data management system 102. As previously discussed, the test log contains a list of queries that have a high probability of forcing an error when the big data management system 102 executes such queries against the big data cluster 104.

[0144] By limiting the number of queries for performing stress tests against a big data management system, such as utilizing queries that have a high probability of forcing an error, time and resources are optimally used without compromising the quality of the stress test.

[0145] In this way, an optimal set of queries is identified for use in stress testing against a big data management system (e.g., IBM® DB2® Big SQL) that optimally utilizes available time and resources without compromising the quality of the stress test.

[0146] As a result of the foregoing, embodiments of the present disclosure generate a set of outlier queries that have a high probability of causing a query or system failure. Furthermore, embodiments of the present disclosure predict the runtime duration of each outlier query and prioritize their execution so that the shortest running queries can be executed first, thereby optimizing the use of available time and resources.

[0147] Furthermore, the principles of the present disclosure improve upon techniques or fields involving stress testing. As discussed above, stress testing of database management systems continues to be a challenge in terms of generating data, populating database objects, and generating complex queries (e.g., complex SQL queries) to provide thorough quality assurance (QA) test coverage (percentage of software being tested) to effectively test database management system software from a system / runtime perspective. For example, with regard to generating queries to stress test database management systems, such queries need to be complex enough to test the system for reliability, fault tolerance, and stability, while at the same time not being so complex that they require a significant amount of time to execute. Furthermore, with the transition from traditional online transaction processing (OLTP) / online analytical processing (OLAP) systems to big data clusters (clusters of containers that include data stores (e.g., Hadoop® distributed file systems) that store big data) managed by database management systems (e.g., IBM® DB2® Big SQL), the amount of data that tests need to run against has grown by several orders of magnitude, thereby significantly reducing the number of test queries that can be executed within a 24-hour period. "Big data," as used herein, refers to data sets whose size exceeds the ability of commonly used software tools to capture, curate, manage, and process them within an acceptable elapsed time. As a result, stress tests for big data management systems cannot afford to run thousands of complex queries over extended periods of time to test the reliability, fault tolerance, and stability of the database management system software. Therefore, there is currently no means of stress testing big data management systems that optimally uses available time and resources without compromising the quality of the stress test.

[0148] Embodiments of the present disclosure improve upon such techniques by generating a set of random test queries. A "test query," as used herein, refers to a query used to test a big data management system by simulating real-world conditions to identify erroneous behavior, such as SQL statements, that result in failures due to code defects in specific features of the database system. In one embodiment, such random test queries are generated based on utilizing tables containing fields for inclusion in the test queries. The set of random test queries is then compiled to extract access plan characteristics to determine data points for the set of random test queries (e.g., column types, queried table types, access plan operator types). An "access plan," as used herein, refers to specifying an order of operations for accessing data. An "access plan characteristic," as used herein, refers to the characteristics of such an access plan. A "characteristic," as used herein, refers to characteristics of a query, such as column types, queried table types, access plan operator types, timers (cost estimates based on an evaluation of the resources that will be used), total complexity cost divided into buckets, temporary table usage, final statistics, etc. Next, the distance (e.g., Mahalanobis distance, Euclidean distance) between the data points of the features of the set of random test queries and the mean of the distribution of data points corresponding to each same feature in the extracted Feature Set is measured. In one embodiment, the Feature Set is extracted from the optimizer plan and target table metadata, where the mean of the distribution of data points corresponding to each feature in the extracted Feature Set is measured. "Optimizer plan," as used herein, refers to the output of the optimizer, which describes the optimal method of execution. "Target table metadata," as used herein, refers to metadata (data about the data) that provides data definitions for the source data consumed, including query time statistics. Next, each random test query whose distance exceeds a threshold distance is ranked.The ranked random test queries are then executed in rank order. Executed random test queries that resulted in errors (e.g., identified errors such as system failures) are added to a log, which is used to identify those queries for performing stress tests on the big data management system. In this way, an optimal set of queries to be used in stress testing the big data management system (e.g., IBM® DB2® Big SQL) is identified that optimally utilizes available time and resources without compromising the quality of the stress test. Further, in this way, improvements are made in technical fields including stress testing.

[0149] The technical solutions provided by the present disclosure cannot be implemented in the human mind or by a human using pen and paper, i.e., the technical solutions provided by the present disclosure cannot be realized in the human mind or by a human using pen and paper, in any reasonable amount of time, and with any reasonable expectation of accuracy, without the use of a computer.

[0150] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications of, or technical improvements to, the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for optimally performing stress tests on a big data management system, comprising: generating a set of random test queries; compiling the set of random test queries to extract access plan characteristics to determine characteristic data points for the set of random test queries; measuring the distance between the data points of the features of the set of random test queries and the mean of the distribution of data points corresponding to each same feature of the extracted Feature Set; ranking each random test query whose distance exceeds a threshold distance; executing the ranked random test queries in rank order; adding the executed ranked random test queries that resulted in errors to a log; identifying queries to perform stress testing on the big data management system based on the log; A method comprising:

2. The method of claim 1 , wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on expected runtime duration.

3. The method of claim 2 , wherein the runtime duration is predicted using a decision tree, the decision tree including variables, elements of the variables including features.

4. The method of claim 3 , wherein the variables include one or more characteristics of a query selected from the group consisting of a table type, an operator, an interrupt, a column type, and table metadata.

5. 5. The method of claim 1, wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on a weight assigned to each of the random test queries whose distance exceeds the threshold distance, the weight being determined by the solution of a linear system.

6. identifying data points corresponding to features extracted from queries that result in a successful training workload; determining a mean of a distribution of data points from the identified data points corresponding to each feature of the extracted features; receiving a query for a test suite; compiling the received queries of the test suite to extract access plan characteristics to determine characteristic data points; measuring the distance between the data point for each feature of the compiled queries of the test suite and the mean of the distribution of data points from the identified data points corresponding to each such feature; setting the farthest distance between a data point corresponding to a feature of the compiled query of the test suite and the mean of the distribution of data points for the feature as a threshold distance for the feature; The method of claim 1 , further comprising:

7. The method of claim 6 , wherein the distance comprises a Mahalanobis distance.

8. A computer program for optimally performing stress tests on a big data management system, the computer program comprising: generating a set of random test queries; compiling the set of random test queries to extract access plan characteristics to determine characteristic data points for the set of random test queries; measuring the distance between said data points of the features of said set of random test queries and the mean of the distribution of data points corresponding to each same feature of the extracted feature set; ranking each random test query whose distance exceeds a threshold distance; executing the ranked random test queries in rank order; Adding the ranked random test queries that were executed and that resulted in errors to the log; and identifying queries to perform stress testing on the big data management system based on the log; A computer program for executing

9. The computer program product of claim 8 , wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on expected runtime duration.

10. The computer program product of claim 9 , wherein the runtime duration is predicted using a decision tree, the decision tree including variables, elements of the variables including features.

11. The computer program product of claim 10 , wherein the variables include one or more characteristics of a query selected from the group consisting of a table type, an operator, an interrupt, a column type, and table metadata.

12. 12. The computer program product of claim 8, wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on a weight assigned to each of the random test queries whose distance exceeds the threshold distance, the weight being determined by the solution of a linear system.

13. the processor, identifying data points corresponding to features extracted from queries that result in a successful training workload; determining a mean of a distribution of data points from the identified data points corresponding to each feature of the extracted features; receiving queries for the test suite; compiling the received queries of the test suite to extract access plan characteristics to determine characteristic data points; measuring the distance between the data point for each feature of the compiled queries of the test suite and the mean of the distribution of data points from the identified data points corresponding to each such feature; setting a threshold distance for a feature as the farthest distance between a data point corresponding to the feature in the compiled query of the test suite and the mean of the distribution of data points for the feature; 12. The computer program of claim 8, further comprising:

14. The computer program product of claim 13 , wherein the distance comprises a Mahalanobis distance.

15. 1. A system comprising: a memory storing a computer program for optimally performing stress tests on a big data management system; a processor coupled to the memory, the processor comprising: generating a set of random test queries; compiling the set of random test queries to extract access plan characteristics to determine characteristic data points for the set of random test queries; measuring the distance between the data points of the features of the set of random test queries and the mean of the distribution of data points corresponding to each same feature of the extracted Feature Set; ranking each random test query whose distance exceeds a threshold distance; executing the ranked random test queries in rank order; adding the executed ranked random test queries that resulted in the error to a log; and Identifying queries to perform stress testing on the big data management system based on the log; a processor configured to execute program instructions of the computer program, A system comprising:

16. The system of claim 15 , wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on expected runtime duration.

17. The system of claim 16 , wherein the runtime duration is predicted using a decision tree, the decision tree including variables, elements of the variables including features.

18. 20. The system of claim 17, wherein the variables include one or more characteristics of a query selected from the group consisting of a table type, an operator, an interrupt, a column type, and table metadata.

19. 19. The system of claim 15, wherein each of the random test queries whose distance exceeds the threshold distance is ranked based on a weight assigned to each of the random test queries whose distance exceeds the threshold distance, the weight being determined by the solution of a linear system.

20. The program instructions of the computer program include: identifying data points corresponding to features extracted from queries that result in a successful training workload; and determining a mean of a distribution of data points from the identified data points corresponding to each feature of the extracted features; receiving a test suite query; compiling the received queries of the test suite to extract access plan characteristics to determine characteristic data points; measuring the distance between the data point for each feature of the compiled queries of the test suite and the mean of the distribution of data points from the identified data points corresponding to each such feature; setting the farthest distance between a data point corresponding to a feature of the compiled query of the test suite and the mean of the distribution of data points for the feature as a threshold distance for the feature; 19. The system of claim 15, further comprising:

Images