Using query clustering to identify abnormal queries

Abstract

This document discloses techniques relating to managing database queries. In some implementations, a server system receives a query from a computer system and determines a set of aspects of the query, which at least include the number of columns specified in the query and the computational cost of executing the query. The system generates a query vector based on the set of aspects determined for the query. The system then compares the query vector with multiple clusters, where each cluster includes two or more previously generated query vectors, which are generated based on aspects of the query previously received by the server system. Based on the comparison, specifically the distances between the query vectors and multiple clusters of previously generated query vectors, the system classifies the query. Based on the classification of the query determined during the classification process, the system manages the query.

Description

Technical Field

[0001] The publicly available text deals with database systems as a whole, and more specifically with database optimization. Background Technology

[0002] When a query is submitted to the database, it indicates what the query result should be. The result is then ingested by the system operating the database to determine one or more ways to execute the query (this can be done using several methods). For example, a query requesting a join of tables A, B, and C can be executed as: 1) join A and B, then join the result and C, or 2) join B and C, then join A and the result. Modern relational database systems typically employ a query optimizer, which receives parsed queries and evaluates different query execution plans to determine the plan used to execute the query on the database. However, such systems often receive too many queries per second, per minute, or per day, making query optimization difficult and time-consuming. Attached Figure Description

[0003] Figure 1 This is a block diagram illustrating a server system configured to classify queries based on multiple query aspects according to some implementation schemes.

[0004] Figure 2 This is a block diagram illustrating exemplary vector and clustering modules according to some implementation schemes.

[0005] Figure 3 This is a diagram illustrating exemplary queries, query vector formats, and exemplary query vectors according to some implementation schemes.

[0006] Figure 4 This is a diagram illustrating exemplary clustering according to some implementation schemes, and exemplary distance measurements between a query vector and two different clusters.

[0007] Figure 5 This is a block diagram illustrating an exemplary classification module according to some implementation schemes.

[0008] Figure 6 This is a flowchart illustrating a method for classifying queries according to some implementation schemes.

[0009] Figure 7 This is a block diagram illustrating the components of a multi-tenant system according to some implementation schemes.

[0010] Figure 8 This is a block diagram illustrating an exemplary computer system according to some implementation schemes. Detailed Implementation

[0011] Generally, database systems (such as Salesforce.com™) typically store large amounts of customer data and receive a massive number of queries requesting access to this data every second, every minute, every day, etc. Therefore, these database systems are often difficult to design and optimize, for example, receiving millions of queries from different computer systems. Furthermore, a large number of queries often creates security risks (these queries may be received by computer systems operated by authorized users inside the database system and by computer systems operated by users outside the database system). For example, a database administrator might accidentally submit a query requesting the deletion of a large collection of data stored in the database. In this example, because the administrator has a high level of privilege within the database system, the query may be authorized, leading to the loss of a large amount of data. As another example, multiple similar queries are often submitted from different computer systems within a given second, minute, hour, etc., resulting in inefficient use of database system resources (i.e., performing the same database operation on the database multiple times within a minute of each other).

[0012] To prevent the execution of unwanted queries on the database, the disclosed techniques use clustering to encode and classify newly received queries. Such techniques identify anomalous or suspicious queries entering a database system (e.g., Salesforce.com), which, as described above, can be millions of queries per minute. The disclosed techniques classify queries (e.g., whether a query is risky, an outlier, slow, computationally expensive, etc.) by encoding the queries according to various characteristics, and then clustering the queries based on the similarity of their encoded characteristics (e.g., via k-means clustering). Query characteristics may also be referred to herein as query "aspects." The encoding forms query vectors, each including several characteristics determined during encoding.

[0013] Query clustering forms groups of similar queries, where different groups represent common or expected queries. These clusters are used to categorize new incoming queries submitted to, for example, a web application. For instance, if the vector of a new query lies outside the various generated query clusters, the query may be somewhat anomalous (and potentially risky or even malicious). In the example, if the vector of a new query is more than a threshold distance from the various generated clusters, the exposed system can automatically flag the new query and perform one or more preventative actions relative to the query.

[0014] In some cases, if the encoding of an input query is closer to a cluster consisting of "risky" or "abnormal" queries, the disclosed technique will likely also classify the input query as risky and may completely block the query (or send the query for further review by an administrator). In other cases, the query may be classified as a "slow" query (based on its proximity to the centroid query being classified into a computationally expensive cluster). In such cases, the query is marked as needing modification or improvement and sent back to the computer system that submitted the query for revision.

[0015] In the disclosed techniques, encoding queries in a multidimensional aspect space allows for comparison of queries at internal hierarchies (e.g., comparisons are not distorted by the query syntax). The internal comparisons that generate queries provide an understanding of the queries, which can then be advantageously used to improve database performance and security. For example, the disclosed techniques can prevent the execution of suspicious (and often malicious) queries on the database. Furthermore, the disclosed techniques can advantageously reduce the number of queries executed on the database. As an example, given 500,000 newly submitted queries, the disclosed database system might break these queries down into 30,000 unique queries (e.g., by identifying and removing duplicates), which correspond to approximately 80 different clusters. This reduction in the number of queries approved for execution on the database further reduces the amount of storage capacity required for the database itself (e.g., for storing new data submitted in write queries).

[0016] Query categorization and execution

[0017] Now go to Figure 1 This diagram depicts a server system configured to classify queries based on multiple query aspects. In the illustrated embodiment, database system 10 (an example of a server system configured to classify queries) includes a query optimizer 120, a vector module 140, a clustering module 160, a classification module 170, an execution engine 130, and a database 145. In the illustrated embodiment, database system 10 receives query 102 from one or more computer systems 105. In some embodiments, database system 10 may be implemented differently than shown. For example, system 10 may include more components, query 102 may be expressed using any syntax from a variety of syntaxes, query optimizer 120 may generate a single execution plan for each query, and several components included in database system 10 (e.g., vector module 140, clustering module 160, and classification module 170) may be included in another server system that performs query vectorization, clustering, and classification, discussed in more detail below, and transmits the results generated by these components to database system 10 for input to execution engine 130, etc.

[0018] Database system 10 can correspond to any suitable database system. In some implementations, system 10 is a relational database management system (RDBMS), which can be implemented using, for example, Oracle™, MySQL™, Microsoft™ SQL Server, PostgreSQL™, IBM™ DB2, etc. Therefore, system 10 can be configured to store data in one or more data tables 145A to serve query 102. Furthermore, system 10 can be implemented as a distributed database with multiple distributed instances. System 10 can also maintain one or more indexes 145B that can be used to facilitate data retrieval from data table 145A, and can generate one or more temporary tables 145C in response to serving query 102. In some implementations, structured query language (SQL) is used to express query 102; in other implementations, other query declarative languages ​​may be supported. In some implementations, database system 10 includes a multi-tenant database, in which each of the multiple tenants stores a corresponding set of data in the database, as referenced below. Figure 6 This will be discussed in more detail. For example, a multi-tenant database may include a first set of data belonging to a non-profit organization (e.g., the first tenant) and a second set of data belonging to an individual (e.g., the second tenant). In such implementations, database system 10 may employ various security measures to ensure that one tenant's dataset is isolated from another tenant's dataset, in order to prevent one tenant from accessing the other tenant's data.

[0019] In the illustrated implementation, database system 10 receives multiple queries 102 from one or more computer systems 105. In some implementations, in addition to receiving queries 102 from one or more computer systems 105, database system 10 also retrieves previously received queries by automatically ingesting queries from the database control record infrastructure. For example, in addition to receiving new queries 102, database system 10 may also retrieve previous queries via clustering module 160 to analyze query patterns, as referenced below. Figure 2 and Figure 3 This will be discussed in more detail. After receiving query 102, database system 10 inputs query 102 into both vector module 140 and query optimizer 120.

[0020] In the illustrated implementation, query optimizer 120 receives query 102 from one or more computer systems 105. In various implementations, query optimizer 120 is operable to generate an execution plan 122 (also referred to herein as a “query plan”) for a given query 102, which includes evaluating various execution plans 122 and selecting one execution plan to implement the given query 102. Query optimizer 120 may use any suitable algorithm to generate and evaluate execution plan 122. In some implementations, query optimizer 120 may use heuristic algorithms, where the execution plan 122 is evaluated based on a set of rules provided to optimizer 120. In other implementations, optimizer 120 uses cost-based algorithms, where optimizer 120 performs cost analysis, which includes assigning scores to execution plan 122 based on estimated processor consumption, estimated memory consumption, estimated execution time, etc. These estimates may also be based on various metrics, such as the number of distinct values ​​in table columns, predicate selectivity (the score of rows that satisfy the predicate), cardinality of the accessed table 140A (e.g., row count), etc. Based on these scores, query optimizer 120 can then select execution plan 122 with the best score. In some implementations, query optimizer 120 may use a combination of heuristic algorithms and cost-based algorithms.

[0021] As described above, in various implementations, query optimizer 120 may also operate to evaluate execution plan 122 based on constraints included in query 102 (also referred to herein as query “criteria”) and select one or more plans 122 that meet the constraints. For example, in some implementations, query optimizer 120 may assign an unfavorable score to any execution plan 122 that does not meet the query constraints (or may even not score it) to prevent it from being selected.

[0022] In various implementations, if the query optimizer 120 cannot select an execution plan 122 that satisfies the constraints of a given query 102, the query optimizer 120 is operable to provide a corresponding indication that an error has occurred. In some implementations, the error may not only indicate that no plan 122 satisfies the constraints, but may also identify a specific constraint that cannot be satisfied, in the case where multiple constraints are specified in query 102. In some implementations, the query optimizer 120 may still select an execution plan 122 (albeit one that does not meet the constraints) and provide it to the execution engine 130—therefore, the user can still receive the query results, but is aware that the results were obtained in a manner inconsistent with the provided constraints. However, in other implementations, the query optimizer 120 may provide an error and not select any plan 122 to implement query 102. In the implementation shown, the query optimizer 120 sends one or more selected execution plans 122 to the execution engine 130 for executing query 102, as discussed in more detail below.

[0023] In the illustrated implementation, query optimizer 120 sends query estimate 124 to vector module 140. Query estimate 124 indicates the amount of time and computational resources required to execute different queries 102 based on the execution plan 122 determined by query optimizer 120 for query 102. For example, query optimizer 120 may send query estimate 124 for a given query 102 to vector module 140, indicating that the plan will take a given number of seconds to execute when utilizing a given amount of resources. In some implementations, query optimizer 120 sends two different query estimates 124 for two different execution plans 122 for a single query 102. In other implementations, query optimizer 120 sends query estimate 124 for a single execution plan 122 for each query 102, provided to vector module 140. Query optimizer 120 generates these query estimates 124 by estimating the total number of tuples or rows (or both) required for the selected query execution plan 122 for a given query 102 and the cost of processing each of these tuples and rows. The query optimizer 120 can generate an estimate 124 by estimating any of the various metrics. When determining which of the multiple query plans to execute for a given query, the query optimizer 120 estimates the cost and various other metrics for each of the multiple query plans. Thus, the query optimizer 120 is able to provide these metrics (query estimates 124) to the vector module 140 for encoding query aspects.

[0024] In the illustrated implementation, vector module 140 receives query 102 from one or more computer systems 105 and query estimate 124 from query optimizer 120. In various implementations, vector module 140 generates a query vector 142 for query 102. Vector module 140 determines a set of aspects of each query 102 received from computer system 105 and uses these set of aspects to generate query vector 142. For example, vector module 140 may determine the following set of aspects for a given query 102: the number of columns specified in the query, the computational cost of executing the query, the number of tables specified in the query, the number of modifications specified in the query, the query type of the query, one or more table object identifiers referenced in the query, etc. References will follow. Figure 2 The determination of the query is discussed in more detail. In the illustrated implementation, vector module 140 sends query vector 142 to clustering module 160.

[0025] In various implementations, clustering module 160 generates clusters of query vectors generated for previously received queries and compares the query vector 142 of the current query 102 with these clusters. For example, the clusters of previously received queries indicate patterns in typically received queries. In various cases, if the query vector 142 of the current query is far from the clusters of previously received queries in the vector space, the query corresponding to the vector may be somewhat anomalous (e.g., it may be a risky query). See below for reference. Figure 2 In more detail, clustering module 160 utilizes one or more clustering algorithms to generate multiple clusters. After comparing the query vector 142 of the current query 102 with the clusters of vectors corresponding to previously received queries, clustering module 160 sends the comparison result 162 to classification module 170. The comparison result 162 can indicate the similarity between the query vector of the current query and the query vector of a previously received query. For example, if the distance between the clusters of the query vector of the current query 102 and the query vector of a previously received query is greater than a distance threshold requirement, then the current query 102 is not similar to the previously submitted query.

[0026] In the illustrated embodiment, classification module 170 receives comparison results 162 from clustering module 160. Classification module 170 assigns a category corresponding to the comparison results to the current query 102. In the illustrated embodiment, classification module 170 generates and sends a query decision 172 (which is generated based on the classification of the current query 102). For example, query decision 172 indicates whether the execution engine 130 should execute the current query 102 based on the execution plan 122 corresponding to the current query. See below for reference. Figure 5 The query decision 172 for query 102 is discussed in more detail.

[0027] In various implementations, execution engine 130 is operable to execute a selected execution plan 122 based on a query decision 172 received from classification module 170 for the query 102 for which the execution plan 122 was generated. Therefore, engine 130 can perform various actions listed in execution plan 122, which may include accessing one or more data tables 145A, indexes 145B, or temporary tables 145C. Engine 130 can then return any results to serve query 102. For example, execution engine 130 may cause the results of the executed query plan to be transmitted to a computer system in computer system 105. In some implementations, execution engine 130 does not execute execution plan 122. For example, if query decision 172 indicates that the query corresponding to the plan should be rejected, execution engine 130 will not execute execution plan 122 for the query. In some cases, execution engine 130 and query optimizer 120 may also be referred to as “modules”.

[0028] In the published text, various "modules" (e.g., database statistics module 140, selector module 150, etc.) operable to perform specified functions are shown and described in detail in the figures. As used herein, "module" refers to software or hardware operable to perform a specified set of operations. A module can refer to a set of software instructions that can be executed by a computer system to perform a set of operations. A module can also refer to hardware configured to perform a set of operations. Hardware modules can constitute general-purpose hardware and non-transitory computer-readable media storing program instructions, or special-purpose hardware such as custom ASICs. Therefore, a module described as "capable of performing" to perform operations refers to a software module, while a module described as "configured" to perform operations refers to a hardware module. A module described as "operable" to perform operations refers to a software module, a hardware module, or some combination thereof. Furthermore, with respect to any discussion herein of modules "capable of performing" to perform specific operations, it should be understood that in other embodiments, these operations may be implemented by hardware modules "configured" to perform operations, and vice versa.

[0029] Exemplary vectorization and clustering

[0030] Now go to Figure 2 A diagram illustrating exemplary vectors and clustering modules is provided. Figure 2 The detailed configurations of the vector module 140 and the clustering module 160 are shown in the figure. For example, the vector module 140 includes an aspect module 210 and a vectorization module 220, while the clustering module 160 includes a clustering cache 250 and a distance module 240.

[0031] In the illustrated embodiment, aspect module 210 receives query 102 and identifies multiple aspects 212 for each query. For example, aspect module 210 identifies two or more of the following aspects for each query: the number of columns specified in the query, the number of tables specified in the query, the number of modifications specified in the query, the query type of the query, one or more table object identifiers referenced in the query, the number of joins specified in the query, the number of scans specified in the query, one or more subqueries included in the query, the number of operator nodes in the query, etc. In the illustrated embodiment, aspect module 210 provides different groups of aspects 212 of query 102 to vectorization module 220.

[0032] In the illustrated implementation, the vectorization module 220 receives the query estimate 124 (from... Figure 1 The query optimizer 120 shown in the diagram receives aspects 212 from aspect module 210. In various embodiments, vectorization module 220 generates a query vector 142 for each query 102 based on aspects 212 and query estimates 124. For example, the query estimate 124 for each query 102 specifies the computational cost of executing the query and the amount of time required to execute the query. Vectorization module 220 selects one or more aspects from aspects 212 of a given query 102 and query estimates 124, and encodes these values ​​in a vector to generate the query vector 142 for the given query 102. In some embodiments, vectorization module 220 places all values ​​of aspects 212 and query estimates 124 in the query vector 142 for the given query. In other embodiments, vectorization module 220 selects only a subset of aspects and query estimates 124 to include in the query vector 142.

[0033] In the illustrated implementation, clustering module 160 manages clustering cache 250, which stores clusters previously generated by clustering module 160 based on multiple query vectors of previously submitted queries. For example, clustering module 160 retrieves multiple previously received queries from a database (e.g., database 145) and generates multiple clusters based on these queries. In this example, clustering module 160 then stores the clusters in clustering cache 250 for later use. Furthermore, in this example, clustering module 160 identifies different query vectors located at the "center" of each cluster and assigns these vectors to the centroid of each cluster. This type of clustering is called centroid clustering. In other cases, clustering module 160 implements other types of clustering, including density-based clustering or distribution-based clustering. In some implementations, the clustering performed by clustering module 160 is k-means clustering. In various implementations, clustering module 160 uses one or more of the following to perform clustering: density-based noisy applied spatial clustering (DBSCAN), hierarchical clustering, spectral clustering, etc.

[0034] In the illustrated implementation, distance module 240 retrieves existing clusters and centroids 252 from clustering cache 250. Distance module 240 determines the distance between the query vector 142 of the current query and the centroids 252 of the clusters retrieved from clustering cache 250. In some implementations, distance module 240 performs a distance measurement algorithm to determine the distance between a given query vector 142 and one or more centroids 252 of the clusters retrieved from clustering cache 250. Distance module 240 performs one or more distance measurement algorithms of the following types: Euclidean distance (e.g., determining the square root of the sum of the squared differences between two vectors), cosine similarity (e.g., determining the dot product of two vectors divided by the product of the norms of the two vectors), or inner product (e.g., multiplying each dimension by the corresponding dimension of another vector and summing the products). After determining the distance between the query vector of the current query and one or more centroids, distance module 240 transmits distance 262 to classification module 170. Classification module 170 classifies the query vector based on the measured distances, as referenced below. Figure 5 To be discussed in more detail.

[0035] Now go to Figure 3 The diagram depicts an exemplary query 302, a query vector format 310, and an exemplary query vector 342. Figure 3 The exemplary query 302 shown includes two different queries 302A and 302B, which reference different tables and columns but have the same general query structure. In the disclosed art, due to their similar structure, the database system 10 (via clustering module 160 and classification module 140) places both queries 302A and 302B in the same cluster. For example, the two exemplary queries 302 have similar query vectors 342A and 342B, such as... Figure 3 As shown at the bottom. Example query 302A includes the following three criteria: "Delete from table_1", "Where column_1 = $1", and "Where column_2 = $2". Similarly, example query 302B includes the same three criteria, but for different tables and columns: "Delete from table_2", "Where column_3 = variable-length character '1'", and "Where column_4 = $1".

[0036] exist Figure 3The example vector format 310 is shown, which contains the following entries within a query vector: [Query type, table name, table object identifier (ID), table count, column reference count, query cost estimate, modification count, ..., query aspect N]. It should be noted that the list of aspect values ​​included in query vector format 310 is not an exhaustive list, and other query aspects can be included in the query vector. The "query aspect N" entry in query format 310 indicates that any number of query aspects can be included in the query vector. For example, a query vector may include ten or more aspects from its corresponding query.

[0037] exist Figure 3 The bottom part shows (by) Figure 1 and Figure 2 The vector module 140 shown generates two exemplary query vectors 342 for two queries 302A and 302B and assigns them to the same cluster. For example, query vector 342A generated for query 302A includes the following values: [delete, table_1, "table_1", 1 second, 2 seconds, 30 seconds and 5 CPUs, 2, ..., empty]. The exemplary query vector 342B generated for query 302B includes the following values: [delete, table_2, "table_2", 1 second, 2 seconds, 30 seconds and 5 CPUs, 2, ..., empty]. In the example, as described above, although the two queries 302 refer to different tables and columns, they have very similar query vectors 342, and therefore (e.g., by clustering module 160) they are added to the same cluster.

[0038] In some implementations, queries that are visually different and refer to different tables and columns, but have similar multidimensional aspect spaces (i.e., similar query vectors), are grouped together in the same cluster. For example, if exemplary query 302 looks different (e.g., has different modification actions, such as having "read from" instead of "delete from") but still has similar query vectors, these queries will still be grouped in the same cluster within the multidimensional aspect space. In some cases, visually similar queries but with different multidimensional aspect spaces will not be in the same cluster.

[0039] Figure 4 This is a diagram illustrating an exemplary clustering and an exemplary distance measurement between a query vector and two different clusters. In the illustrated embodiment, the exemplary cluster 400 shown in the top portion of the diagram includes ten different query vectors 452A, wherein one of these query vectors is a centroid 402A assigned to cluster 400. In the example, clustering module 160 (such as...) Figure 1 and Figure 2(As shown) Cluster 400 is generated based on existing query vector 452A, or clusters are retrieved from cluster cache 250, and centroid 402A is assigned as the "center point" of cluster 400. For example, clustering module 160 selects centroid 402A based on summing the positions of each query vector within cluster 400 and dividing the sum by the number of query vectors included in cluster 400. In the example, the quotient indicates a position (e.g., a point) within cluster 400. Clustering module 160 assigns the query vector in the cluster closest to the position indicated by the quotient as the centroid (i.e., 402A).

[0040] exist Figure 4 The bottom portion shows an example of distance measurement 410. In the exemplary distance measurement 410, two distinct clusters are shown: cluster 400 (also shown in...) Figure 4 The top portion is shown), and another cluster 404 is composed of the existing query vector 452B. In the example, distance module 240 (as shown) Figure 2 (As shown) Determine the distance between the new query vector 442 and the cluster centroids 402A and 402B. Figure 4 The exemplary distance measurement 410 shown illustrates a first line between centroid 402A and vector 442, and a second line between centroid 402B and vector 442. These lines respectively represent the shortest distances in the multidimensional query space between query vector 442 and query vectors at centroids 402A and 402B. Figure 4 The distance measurements shown are performed using Euclidean distance measurement techniques.

[0041] Exemplary distance measurement 410 shows that the new query vector 442 is closer to the centroid 402A of cluster 400 than the centroid 402B of another cluster 404, as... Figure 4 As shown. Therefore, in the example, classification module 170 (such as...) Figure 1 (As shown) the query corresponding to vector 442 may be assigned to cluster 400, as shown in the following reference. Figure 5 This will be discussed in more detail. For example, classification module 170 assigns the category or label associated with cluster 400 to the query corresponding to vector 442. However, if query vector 442 is not close to the centroid of any cluster, classification module 170 may mark the query corresponding to vector 442 as an anomaly (e.g., to indicate that the query may be suspicious or risky to some extent).

[0042] In some implementations, if a query vector falls into a "grey area" between two threshold distance requirements, queries targeting that vector are classified differently from queries whose vectors are very close to or very far from the cluster centroid. For example, see the following reference. Figure 5In more detail, queries whose vectors fall within the “grey area” of the multidimensional aspect space can be classified differently, such that these queries require and trigger different types of preventative actions based on the centroid of the clusters in the multidimensional aspect space that are close to or far from their vectors.

[0043] In some cases, after determining that the new query vector 442 is closer to cluster 400 (i.e., closer to centroid 402A rather than centroid 402B), the clustering module 160 adds query vector 442 to cluster 400. In this way, the new query vector 442 modifies cluster 400, which can be used for future query evaluation. Alternatively, the clustering module 160 can update cluster 400 by determining whether centroid 402A is still located at the center of cluster 400. If centroid 402A is no longer near the center of cluster 400, the clustering module 160 determines and assigns a different query vector within cluster 400 instead of centroid 402A as the centroid.

[0044] Exemplary classification

[0045] Now go to Figure 5 The diagram illustrates an exemplary classification module. In the illustrated embodiment, the classification module 170 includes a decision module 510, a machine learning classifier 520, and a preventative action module 530. In the illustrated embodiment, the classification module 170 receives comparison results 162 (from...) Figure 1 The clustering module 160 shown outputs a query decision 172 for query 102. In the illustrated implementation, the classification module 170 executes one or both of the decision module 510 and the machine learning classifier 520 to determine the classification of query 102 corresponding to the comparison result 162. For example, sometimes the classification module 170 executes only the decision module 510 for query 102. In other cases, the classification module 170 executes only the machine learning classifier 520. In still other cases, the classification module 170 executes both module 510 and classifier 520 and compares the results of these two different executions.

[0046] In the illustrated implementation, the decision module 510 receives a comparison result 162 (which includes results from...) Figure 2 The distance module 240 (shown) measures one or more distances and compares the distances with one or more distance thresholds 512. For example, the decision module 510 will... Figure 4The distance between the new query vector 442 and centroids 402A and 402B is measured and compared with a distance threshold 512. In the example, decision module 510 determines that the distance between the new query vector 442 and centroid 402A is less than the distance threshold 512 requirement. Based on comparing the distance with one or more rules or heuristics 514, decision module 510 assigns a classification to the query corresponding to vector 442 (or to vector 442 itself). Rules and heuristics 514 may specify, for example, that if the distance measurement included in comparison result 162 is below a first distance threshold 512, then the query vector corresponding to result 162 is not anomalous (this is a normal, expected query), and decision module 530 labels the vector accordingly. Decision module 510 inputs the classification of the query to preventive action module 530, as shown in the example illustrated.

[0047] In various implementations, clustering module 160 can generate clusters from previously received queries that have been classified as suboptimal queries. For example, clustering module 160 can generate clusters from a query training set that includes slow queries, anomalous queries, etc. In the example, when a new query is compared with clusters generated using suboptimal queries, decision module 510 can classify query vectors that are close to these clusters as anomalous; query vectors that are not close to any of these clusters can be classified as non-anomalous.

[0048] In the illustrated implementation, machine learning classifier 520 receives comparison results 162 (which include the distance between the query vector and one or more centroids of the cluster) and outputs a classification of the query vector based on the input distance. Machine learning classifier 520 is, for example, one or more machine learning models of the following types: neural networks, logistic regression classifiers, random forest classifiers, linear regression classifiers, etc. The classification indication output by machine learning classifier 520 (and input to preventative action module 530) is one or more of the following classifications for the query vector: non-anomaly query, anomaly query, gray area query (e.g., when the vector is neither close to nor far from any given cluster), unknown query, slow query, security risk query, duplicate query, etc.

[0049] In the illustrated implementation, the preventative action module 530 receives output from the decision module 510 or the machine learning classifier 520, or both. Based on the received output, the preventative action module 530 generates a query decision 172, indicating at least whether the execution engine 130 should execute the newly received query 102. In other words, the query decision 172 indicates how the newly received query 102 should be managed. In an implementation where the preventative action module 530 receives output from both module 510 and classifier 520, module 530 compares the two outputs before making the query decision 172. As an example, the query decision 172 could indicate that the query is rejected, approved (and executed), requires additional authentication or approval, requires revision, requires additional review, requires deduplication, etc. For example, if the distance between the query vectors of two queries within a cluster is 0, the preventative action module 530 determines that these queries are duplicates of each other. In this example, the preventative action module 530 outputs a query decision 172 indicating that only one of these queries should be executed (to avoid wasting time and computational resources by executing the same query twice).

[0050] In various implementations, in addition to generating query decision 172, preventative action module 530 generates and executes one or more preventative actions relative to query decision 172. For example, if query decision 172 indicates that a given query requires additional authentication or approval, preventative action module 530 performs preventative actions such as sending an authentication request (e.g., one or more authentication factors) to the computer system 105 that submitted the given query. Alternatively, preventative action module 530 may cancel a query based on query decision 172 indicating that it is an anomalous query. In the example, preventative action module 530 may transmit a notification indicating that the query is anomalous to a system administrator, who can then manually cancel the query. This type of preventative action can be helpful where queries generated by a new system application in response to a new pattern change appear “different” (e.g., different) from existing queries. In such cases, although the query is not anomalous, it may appear anomalous based on the new pattern and thus be classified as anomalous by the publicly disclosed system; however, since the query is not actually anomalous, it is sent for review by the system administrator to prevent the query from being completely cancelled (e.g., the administrator approves the query for execution after receiving an alert from the preventive action module 530).

[0051] Exemplary methods

[0052] Figure 6 This is a flowchart illustrating an exemplary method for classifying queries according to some implementation schemes. Figure 6The methods illustrated herein can be used in conjunction with any of the computer circuits, systems, devices, elements or components, and other devices disclosed herein. In various embodiments, they may be executed simultaneously, in a different order than shown, or some of the method elements shown may be omitted. Additional method elements may also be executed as needed. In one embodiment, method 600 is executed by a query optimizer of a database (such as a database implemented by database system 10) capable of receiving optimizer constraints. In some instances, executing method 600 allows for the potential selection of a better execution plan relative to previous query plan evaluation and selection techniques.

[0053] Method 600 begins at element 610, whereby the server system receives queries from the computer system. In some embodiments, prior to receiving queries from the computer system, the server system generates multiple query vectors for multiple previously received queries, wherein generating the multiple query vectors includes identifying two or more aspects of the corresponding queries. In some embodiments, prior to receiving queries from the computer system, the server system further performs a clustering operation on the multiple query vectors, wherein the output of the clustering operation is multiple clusters representing two or more of the multiple query vectors of the multiple previously received queries. In some embodiments, the server system further assigns the query codes located at the centers of the clusters within the multiple clusters as the centroids of the corresponding clusters of these query codes.

[0054] At element 620, the server system determines grouped aspects of the query, which include at least the number of columns specified in the query and the computational cost of executing the query. In some implementations, the server determines one or more additional aspects of the grouped aspects of the query, including: the number of tables specified in the query, the number of modifications specified in the query, the number of columns specified in the query, the computational cost of executing the query, the query type of the query, and one or more table object identifiers referenced in the query.

[0055] At element 630, the server system generates a query vector based on the grouped aspects of the query. In some implementations, the query vector includes the same number of values ​​as the number of aspects included in the grouped aspects of the query. In other implementations, the query vector includes values ​​corresponding only to a subset of the grouped aspects.

[0056] At element 640, the server system compares the query vector with multiple clusters, which include two or more previously generated query vectors based on aspects of queries previously received by the server system.

[0057] At element 650, the server system classifies queries based on comparisons, where the classification is based on the distances between the query vector and multiple clusters of previously generated query vectors. In some implementations, the distances between the query vector and the multiple clusters of previously generated query vectors are determined by measuring the Euclidean distance between the query vector and the centroids of the corresponding clusters in the multiple clusters of previously generated queries. In some implementations, the classification includes assigning an anomalous classification to queries received from the computer system based on the distance between the query vector and the multiple clusters being greater than a distance threshold requirement. For example, if the distance between the query vector and the centroid of the nearest cluster is greater than or equal to the distance threshold requirement, the query corresponding to the query vector may be anomalous in some way (e.g., the query is suspicious and potentially harmful). In the example, the distance between the query vector and the centroid could be three inches, while the distance threshold requirement could be two inches.

[0058] In some implementations, the server system determines the distance between the query vector and the multiple clusters of the previously generated query vector by measuring the distance between the query vector and the centroids of the corresponding clusters in the multiple clusters of the previously generated query vector using cosine similarity. In some implementations, the server system determines the distance between the query vector and the multiple clusters of the previously generated query vector by measuring the distance between the query vector and the centroids of the corresponding clusters in the multiple clusters of the previously generated query vector using an inner product algorithm.

[0059] At element 660, the server system manages queries based on a classification of the query determined during classification. In some embodiments, managing queries based on their classification includes the server system transmitting an alert for the query to the computing system, wherein the alert specifies the classification of the query and one or more preventative actions for the query. In some embodiments, management includes rejecting queries received from the computer system, including transmitting a notification to the computer system indicating that the query is unauthorized. In some embodiments, management includes transmitting a request for one or more authentication factors to the computer system. In some embodiments, management includes determining whether to execute the query based on the response to one or more authentication factors received from the computer system.

[0060] In some implementations, management includes transmitting a request for one or more authentication factors to a computer system. In some implementations, management includes rejecting a query based on a response from the computer system to the request for one or more authentication factors. In some implementations, management includes progressively escalating one or more preventative actions based on the distance between a query vector and multiple clusters of a previously generated query vector, wherein progressive escalation includes performing more restrictive preventative actions on query vectors that are at greater distances to one or more clusters than on preventative actions performed on query vectors that are at smaller distances to one or more clusters. In some implementations, management includes transmitting a notification requesting a change to the query to the computer system from which the query is received, wherein the notification specifies one or more defects in the query corresponding to the requested change.

[0061] In some implementations, management includes deduplicating queries by classifying them into categories that instruct them to perform the same operation (even if they are not the same query). This can advantageously improve the performance of the disclosed database system by reducing the amount of computational resources required to execute queries (the system will not waste resources by executing the same query twice) and reducing or preventing database errors caused by implementing duplicate queries.

[0062] Exemplary multi-tenant database system

[0063] Now go to Figure 7 An exemplary multi-tenant database system (MTS) 700 in which various technologies for implementing public text are shown is illustrated; for example, database system 10 may be an MTS 700. Figure 7 In this embodiment, MTS 700 includes a database platform 710, an application platform 720, and a network interface 730 connected to a network 740. Similarly, as shown, database platform 710 includes a data storage device 712 and a set of database servers 714A-N that interact with the data storage device 712, and application platform 720 includes a set of application servers 722A-N with a corresponding environment 724. In the illustrated embodiment, MTS 700 is connected to various user systems 750A-N via network 740. The inclusion of the disclosed multitenant system for illustrative purposes is not intended to limit the scope of the disclosed text. In other embodiments, the techniques of the disclosed text are implemented in non-multitenant environments such as client / server environments, cloud computing environments, clustered computers, etc.

[0064] In various implementations, the MTS 700 is a group of computer systems that together provide various services to users (alternatively referred to as "tenants") interacting with the MTS 700. In some implementations, the MTS 700 implements a Customer Relationship Management (CRM) system, which provides tenants (e.g., companies, government agencies, etc.) with mechanisms for managing their relationships and interactions with customers and prospects. For example, the MTS 700 can enable tenants to store customer contact information (e.g., customer websites, email addresses, phone numbers, and social media data), identify opportunities, record service issues, and manage marketing campaigns. The MTS 700 can also enable these tenants to identify previous communications with customers, what customers have purchased, when customers last purchased items, and the amount customers paid. To provide CRM system services and / or other services, as shown in the figure, the MTS 700 includes a database platform 710 and an application platform 720.

[0065] In various implementations, database platform 710 is a combination of hardware components and software routines that implement database services for storing and managing data (including tenant data) of MTS 700. As shown, database platform 710 includes data storage device 712. In various implementations, data storage device 712 includes a set of storage devices (e.g., solid-state drives, hard disk drives, etc.) connected together on a network (e.g., a storage attached network (SAN)) and configured to redundantly store data to prevent data loss. In various implementations, data storage device 712 is used to implement a database (e.g., database 145) comprising a series of information organized in a manner that allows access, storage, and manipulation of information. Data storage device 712 can implement a single database, a distributed database, a series of distributed databases, a database with redundant online or offline backups, or other redundancies. As part of implementing the database, data storage device 712 can store files including those with corresponding data payloads (e.g., database tables such as... Figure 1 The data table 145A shown here contains the values ​​of its fields and one or more database records containing metadata such as key values, timestamps, table identifiers of the tables associated with the records, tenant identifiers of the tenants associated with the records, etc.

[0066] In various implementations, database records may correspond to rows in a table. A table typically contains one or more data categories, which are logically arranged as columns or fields in a viewable schema. Therefore, each record in a table may contain a data instance defined by fields for each category. For example, a database may include a table describing customers, with fields for basic contact information such as name, address, phone number, fax number, etc. Therefore, a record for the table may include a value for each field in the table (e.g., a name for the name field). Another table may describe purchase orders, including fields for information such as customer, product, sales price, date, etc. In various implementations, standard entity tables are provided for use by all tenants, such as tables for account, contact, prospect, and opportunity data, each containing predefined fields. The MTS 700 may store database records for one or more tenants in the same table; that is, tenants may share tables. Therefore, in various implementations, database records include a tenant identifier indicating the owner of the database record. Thus, one tenant's data is kept secure and separate from other tenants' data, preventing one tenant from accessing another tenant's data unless such data is explicitly shared.

[0067] In some implementations, data stored at data storage device 712 is organized as part of a Log Structure Merge Tree (LSM tree). The LSM tree typically comprises two high-level components: an in-memory buffer and persistent storage. In operation, database server 714 may initially write these records to a local in-memory buffer before later dumping them to persistent storage (e.g., data storage device 712). As part of dumping the database records, database server 714 may write the database records to a new file included in the “top” level of the LSM tree. Over time, as database records move down the hierarchy of the LSM tree, database server 714 may rewrite the database records to new files included in lower levels. In various implementations, as database records age and move down the LSM tree, they are moved to increasingly slower storage devices (e.g., from solid-state drives to hard disk drives) of data storage device 712.

[0068] When database server 714 wants to access a database record for a specific key, it can traverse different levels of the LSM tree to find files that may contain the database record for that key. If database server 714 determines that a file may contain the relevant database record, it can retrieve the file from data storage device 712 into its own memory. The retrieved file can then be examined to find the database record with the specific key. In various implementations, database records are immutable once written to data storage device 712. Therefore, if database server 714 wants to modify the value of a row in a table (which can be identified from the accessed database records), it writes the new database record to the top level of the LSM tree. Over time, database records are merged down the levels of the LSM tree. Thus, the LSM tree can store various database records for a database key, with older records for the key located at lower levels of the LSM tree compared to newer records.

[0069] In various implementations, database server 714 is a hardware component, software routine, or a combination thereof capable of providing database services such as data storage, data retrieval, and / or data manipulation. Database server 714 may correspond to one of a plurality of database nodes included in a database system (e.g., database system 10). Such database services can be provided by database server 714 to components within MTS 700 (e.g., application server 722) and components outside MTS 700. As an example, database server 714 may receive from application server 722 a database transaction request that requests to write data to data storage device 712 or read data from data storage device. The database transaction request may specify an SQLSELECT command to select one or more rows from one or more database tables. The contents of the rows may be qualified in database records, and thus database server 714 may locate and return one or more database records corresponding to the selected one or more table rows. In various cases, the database transaction request may instruct database server 714 to write one or more database records against an LSM tree—database server 714 maintains an LSM tree implemented on database platform 710. In some implementations, database server 714 implements a relational database management system (RDMS) or an object-oriented database management system (OODBMS) that facilitates the storage and retrieval of information for data storage device 712. In various cases, database servers 714 can communicate with each other to facilitate transaction processing. For example, database server 714A can communicate with database server 714N to determine whether database server 714N has written a database record to its in-memory buffer for a specific key.

[0070] In various implementations, application platform 720 is a combination of hardware components and software routines that implement and execute CRM software applications, and provide relevant data, code, forms, web pages, and other information to and from user system 750, and store relevant data, objects, web page content, and other tenant information via database platform 710. In various implementations, to facilitate these services, application platform 720 communicates with database platform 710 to store, access, and manipulate data. In some instances, application platform 720 may communicate with database platform 710 via different network connections. For example, one application server 722 may be connected via a local area network, and another application server 722 may be connected via a direct network link. Transmission Control Protocol (TCP) and Internet Protocol (TCP / IP) are exemplary protocols for communication between application platform 720 and database platform 710; however, it will be apparent to those skilled in the art that other transport protocols may be used depending on the network interconnection used.

[0071] In various implementations, application server 722 is a hardware element, software routine, or combination thereof capable of providing services of application platform 720, including processing requests received from tenants of MTS 700. In various implementations, application server 722 can create environment 724 that can be used for various purposes, such as providing developers with functionality (e.g., business logic) for developing, executing, and managing applications. Data can be transferred from another environment 724 and / or from database platform 710 to environment 724. In some cases, environment 724 cannot access data from other environments 724 unless such data is explicitly shared. In some implementations, multiple environments 724 may be associated with a single tenant.

[0072] Application platform 720 can provide user system 750 with access to multiple different hosted (standard and / or custom) applications, including CRM applications and / or applications developed by tenants. In various implementations, application platform 720 can manage application creation, application testing, application storage in database objects at data storage device 712, application execution in environment 724 (e.g., a virtual machine in process space), or any combination thereof. In some implementations, application platform 720 can add and remove application server 722 from the server pool at any time for any reason, so there may be no server kinship with a specific application server 722 for users and / or organizations. In some implementations, an interface system (not shown) implementing load balancing (e.g., an F5 Big-IP load balancer) is located between application server 722 and user system 750 and is configured to distribute requests to application server 722. In some implementations, the load balancer uses a least-connections algorithm to route user requests to application server 722. Other examples of load balancing algorithms, such as round-robin and observation of response time, may also be used. For example, in some implementations, three consecutive requests from the same user may hit three different servers 722, and three requests from different users may hit the same server 722.

[0073] In some implementations, the MTS 700 provides security mechanisms (such as encryption) to keep each tenant's data separate unless the data is shared. If more than one server 714 or 722 is used, they can be located close to each other (e.g., in a server farm located in a single building or campus), or they can be distributed in locations far apart from each other (e.g., one or more servers 714 located in city A and one or more servers 722 located in city B). Therefore, the MTS 700 can include one or more logically and / or physically connected servers that are locally distributed or distributed across one or more geographic locations.

[0074] One or more users (e.g., via user system 750) can interact with MTS 700 via network 740. User system 750 may correspond to, for example, a tenant of MTS 700, a provider of MTS 700 (e.g., an administrator), or a third party. Each user system 750 may be a desktop PC, workstation, laptop computer, PDA, cellular phone, or any wireless access protocol (WAP) enabled device, or any other computing device capable of being directly or indirectly connected to the Internet or other network connections. User system 750 may include dedicated hardware configured to intersect with MTS 700 via network 740. User system 750 can execute a graphical user interface (GUI) corresponding to MTS 700, an HTTP client (e.g., a browser program such as Microsoft's Internet Explorer™ browser, Netscape's Navigator™ browser, Opera's browser, or a WAP-enabled browser in the case of cellular phones, PDAs, or other wireless devices), or both, thereby allowing users of user system 750 (e.g., subscribers of a CRM system) to access, process, and view information and pages available from MTS 700 via network 740. Each user system 750 may include one or more user interface devices, such as a keyboard, mouse, touchscreen, pen, etc., for interacting with the graphical user interface (GUI) provided by MTS 700 or other systems or servers in conjunction with pages, forms, and other information provided by MTS 700 or other systems or servers, on a display monitor screen, LCD display, etc. As described above, the disclosed embodiments are suitable for use with the Internet, which refers to a specific global network, the Internet of Things. However, it should be understood that other networks may be used instead of the Internet, such as intranets, extranets, virtual private networks (VPNs), non-TCP / IP-based networks, any LAN or WAN, etc.

[0075] Because users of User System 750 can have varying capabilities, the capabilities of a specific User System 750 can be determined based on one or more permission levels associated with the current user. For example, when a user is interacting with MTS 700 using a specific User System 750, User System 750 may have the capabilities assigned to the user (e.g., user privileges). However, when an administrator is interacting with MTS 700 using the same User System 750, User System 750 may have the capabilities assigned to the administrator (e.g., administrator privileges). In a system with a hierarchical role model, a user at one permission level can access applications, data, and database information that are accessible to users at lower permission levels, but cannot access certain applications, database information, and data that are accessible to users at higher permission levels. Therefore, different users can have different capabilities regarding access to and modification of application and database information, depending on the user's security or permission level. Similarly, some data structures may be managed by MTS 700 and allocated at the tenant level, while other data structures are managed at the user level.

[0076] In some embodiments, the user system 750 and its components are configurable using an application (such as a browser) that includes computer code executable on one or more processing elements. Similarly, in some embodiments, the MTS 700 (and additional instances of the MTS, where more than one additional instance exists) and its components are configurable by an operator using an application that includes computer code executable on a processing element. Thus, the various operations described herein can be performed by executing program instructions stored on a non-transitory computer-readable medium and executed by the processing element. The program instructions may be stored on a non-volatile medium (such as a hard disk), or may be stored in any other known volatile or non-volatile storage medium or device (such as ROM or RAM), or provided on any medium capable of booting the program code (such as a compressed optical disc (CD) medium, a digital versatile disc (DVD) medium, a floppy disk, etc.). Furthermore, the entire program code or portions thereof may be transferred and downloaded from a software source or from another server as is known (e.g., via the Internet), or transmitted via any other known conventional network connection (e.g., extranet, VPN, LAN, etc.) using any communication medium and protocol (e.g., TCP / IP, HTTP, HTTPS, Ethernet, etc.). It should also be understood that the computer code used to implement aspects of the disclosed embodiments may be implemented in any programming language that can execute on a server or server system, such as C, C++, HTML, Java, JavaScript, or any other scripting language (e.g., VBScript).

[0077] Network 740 can be a LAN (Local Area Network), WAN (Wide Area Network), wireless network, point-to-point network, star network, token ring network, hub network, or any other suitable configuration. The global network commonly referred to as the "Internet," with its capital "I," is an example of a TCP / IP (Transmission Control Protocol and Internet Protocol) network. However, it should be understood that the disclosed implementation can utilize any of a variety of other network types.

[0078] User system 750 can communicate with MTS 700 using TCP / IP, and at a higher network level, it can communicate using other public internet protocols such as HTTP, FTP, AFS, and WAP. For example, when using HTTP, user system 750 can include an HTTP client, commonly referred to as a "browser," to send and receive HTTP messages to and from the HTTP server at MTS 700. This server can be implemented as the sole network interface between MTS 700 and network 740, but other technologies can be used or alternatively. In some implementations, the interface between MTS 700 and network 740 includes load-sharing features, such as a round-robin HTTP request dispatcher, to balance the load and distribute incoming HTTP requests evenly across multiple servers.

[0079] In various implementations, user system 750 communicates with application server 722 to request and update system-level and tenant-level data from MTS 700, which may require one or more queries to data storage device 712. In some implementations, MTS 700 automatically generates one or more SQL statements (SQL queries) designed to access desired information. In some cases, user system 750 may generate a request with a specific format corresponding to at least a portion of MTS 700. As an example, user system 750 may request the use of object notation describing an object-relational mapping (e.g., a JavaScript object notation mapping) to move data objects into a specific environment 724.

[0080] Exemplary computer system

[0081] Now go to Figure 8A block diagram depicts an exemplary computer system 800 that can implement database system 10. Computer system 800 includes a processor subsystem 880 connected to system memory 820 and I / O interface 840 via interconnect 860 (e.g., system bus). I / O interface 840 is connected to one or more I / O devices 850. Computer system 800 can be any of various types of devices, including but not limited to server systems, personal computer systems, desktop computers, laptop or notebook computers, mainframe computers, tablet computers, handheld computers, workstations, network computers, and consumer devices such as mobile phones, music players, or personal digital assistants (PDAs). Although for convenience, Figure 8 The diagram shows a single computer system 800, but system 1400 can also be implemented as two or more computer systems operating together.

[0082] Processor subsystem 880 may include one or more processors or processing units. In various embodiments of computer system 800, multiple instances of processor subsystem 880 may be coupled to interconnect 860. In various embodiments, processor subsystem 880 (or each processor unit within 880) may include cache or other forms of onboard memory.

[0083] System memory 820 can be used to store program instructions that can be executed by processor subsystem 880 to cause system 800 to perform the various operations described herein. System memory 820 can be implemented using different physical memory media, such as hard disk storage devices, floppy disk storage devices, removable disk storage devices, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), and so on. The memory in computer system 800 is not limited to main storage devices such as memory 820. Instead, computer system 800 may also include other forms of storage devices, such as cache memory in processor subsystem 880 and auxiliary storage devices (e.g., hard disk drives, storage arrays, etc.) on I / O devices 850. In some embodiments, these other forms of storage devices may also store program instructions that can be executed by processor subsystem 880. In some embodiments, a portion of the database system 10 described above may include (or be included therein) system memory 820.

[0084] According to various implementations, I / O interface 840 can be any type of interface configured to connect to and communicate with other devices. In one implementation, I / O interface 840 is a bridge chip (e.g., a southbridge) from a front end to one or more back end buses. I / O interface 840 can be connected to one or more I / O devices 850 via one or more corresponding buses or other interfaces. Examples of I / O devices 850 include storage devices (hard disk drives, optical disk drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or wide area network), or other devices (e.g., graphics, user interface devices, etc.). In one implementation, computer system 800 is connected to a network (e.g., configured to communicate via Wi-Fi, Bluetooth, Ethernet, etc.) via network interface device 850.

[0085] Various articles of art that store instructions (and optionally, data) executable by a computing system to implement the techniques disclosed herein are also envisioned. The computing system may use one or more processing elements to execute the instructions. Articles of art include non-transitory computer-readable storage media. Envisioned non-transitory computer-readable storage media include portions of the memory subsystem of a computing device and storage media or memory media, such as magnetic media (e.g., disks) or optical media (e.g., CDs, DVDs, and related technologies). Non-transitory computer-readable media can be volatile or non-volatile memory.

[0086] Public texts include references to “one implementation” or a group of “implementations” (e.g., “some implementations” or “various implementations”). An implementation is a different way of realizing or being implemented of the disclosed concepts. References to “implementation,” “one implementation,” “specific implementation,” etc., do not necessarily refer to the same implementation. A large number of possible implementations are envisioned, including those specifically disclosed, as well as modifications or alternatives that fall within the spirit or scope of the public text.

[0087] The disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all implementations of these embodiments will necessarily exhibit any or all of these potential advantages. Whether an advantage is realized for a particular implementation depends on many factors, some of which are outside the scope of the disclosure. In fact, there are multiple reasons why an implementation falling within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of the disclosure, which, in combination with an embodiment of the disclosed embodiments, negates or diminishes one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular implementation (e.g., the implementation technique or tool) may also negate or diminish the disclosed advantages. Even assuming a skilled implementation, the realization of an advantage can still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, the inputs provided to a particular implementation may prevent one or more problems addressed in the disclosure from occurring in a particular context, resulting in the benefits of its solution potentially not being realized. Given the existence of possible factors outside the disclosure, this invention expressly aims that any potential advantages described herein should not be construed as requiring the satisfaction of claims limitations to prove infringement. Rather, the identification of such potential advantages is intended to indicate the types of improvements available to the designer that benefit from the disclosure. The permissive description of such advantages (e.g., stating that a particular advantage "may occur") is not intended to express doubt about whether such advantages can actually be realized, but rather to recognize the technological reality that the realization of such advantages often depends on additional factors.

[0088] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims drafted based on the disclosure, even where only a single example is described with respect to a particular feature. The disclosed embodiments are intended to be illustrative rather than restrictive unless there is any statement to the contrary in the disclosure. Therefore, this application is intended to allow the claims to cover the disclosed embodiments, as well as such alternatives, modifications, and equivalents of the disclosure that would be apparent to those skilled in the art and would benefit from.

[0089] For example, features in this application can be combined in any suitable manner. Therefore, during the examination of this application (or an application claiming priority thereto), new claims can be formulated for any such combination of features. Specifically, referring to the appended claims, features from dependent claims can be combined with features from other dependent claims (including claims dependent on other independent claims) where appropriate. Similarly, features from individual independent claims can be combined where appropriate.

[0090] Therefore, while the appended dependent claims can be drafted such that each dependent claim depends on a single other claim, additional dependent relationships are also contemplated. Any combination of features in the dependent claims consistent with the published text is contemplated, and such combinations can be claimed in this application or another application. In short, the combinations are not limited to those specifically enumerated in the appended claims.

[0091] Where appropriate, it is also envisioned that claims drafted in one format or statutory type (e.g., apparatus) are intended to support corresponding claims in another format or statutory type (e.g., method).

[0092] Because the published text is a legal document, various terms and phrases may be subject to administrative and judicial interpretations. Therefore, it is hereby announced that the following paragraphs, along with the definitions provided throughout the published text, will be used to determine how to interpret claims drafted based on the published text.

[0093] Unless the context clearly specifies otherwise, references to the singular form of an item (i.e., nouns or noun phrases beginning with "an," "a," or "the") are intended to mean "one or more." Therefore, references to "an / a type" in a claim do not exclude additional instances of that item in the absence of accompanying context. "A plurality of" items refer to a collection of two or more of that item.

[0094] The word “can” is used in this text in a permissive sense (i.e., having the possibility, being able) rather than in a mandatory sense (i.e., having to).

[0095] The terms “include” and “including” and their forms are open-ended, meaning “including but not limited to”.

[0096] When the term “or” is used in public text in relation to a list of options, it is generally understood to be used in an inclusive sense, unless the context otherwise specifies. Therefore, a statement of “x or y” is equivalent to “x or y, or both,” and thus encompasses: 1) x but not y; 2) y but not x; and 3) both x and y. On the other hand, phrases such as “either x or y, but not both” clearly indicate that “or” is used in an exclusive sense.

[0097] The statements “w, x, y, or z or any combination thereof” or “...at least one of w, x, y, and z” are intended to cover all possibilities involving a single element up to all elements in the set. For example, given the set [w, x, y, z], these terms cover any single element of the set (e.g., w, but not x, y, or z), any two elements (e.g., w and x, but not y or z), any three elements (e.g., w, x, and y, but not z), and all four elements. Therefore, the phrase “...at least one of w, x, y, and z” refers to at least one element in the set [w, x, y, z], thus covering all possible combinations of the list of elements. This phrase should not be interpreted as requiring the existence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0098] In public text, various “labels” may precede nouns or noun phrases. Unless the context otherwise specifies, different labels used for features (e.g., “first circuit,” “second circuit,” “specific circuit,” “given circuit,” etc.) refer to different instances of the feature. Furthermore, unless otherwise stated, the labels “first,” “second,” and “third” do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) when applied to features.

[0099] The phrase “based on” is used to describe one or more factors that influence a determination. This term does not exclude the possibility that additional factors may influence the determination. That is, a determination may be based solely on the specified factor or on the specified factor along with other unspecified factors. Consider the phrase “A is determined based on B.” This phrase specifies that B is a factor used to determine A or that influences the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on another factor, such as C. This phrase is also intended to cover implementations in which A is determined solely based on B. As used herein, the phrase “based on” is synonymous with the phrase “at least partially based on.”

[0100] The phrases “in response to” and “as a response to” describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect, either jointly with or independently of the specified factor. That is, the effect may respond only to those factors, or it may respond to the specified factor as well as other unspecified factors. Consider the phrase “A is executed in response to B.” This phrase specifies that B is a factor that triggers the execution of A or triggers a specific result of A. This phrase does not exclude that the execution of A may also respond to another factor, such as C. This phrase also does not exclude that the execution of A may jointly respond to B and C. This phrase is also intended to cover implementations in which A is executed only in response to B. As used herein, the phrase “as a response to” is synonymous with the phrase “at least partially as a response to.” Similarly, the phrase “in response to” is synonymous with the phrase “at least partially in response to.”

[0101] Within the publicly available text, different entities (which may be referred to differently as “units,” “circuits,” other components, etc.) may be described or claimed to be “configured” to perform one or more tasks or operations. This expression—[entity] configured to [perform one or more tasks]—is used herein to refer to a structure (i.e., a physical thing). More specifically, this expression is used to indicate that the structure is arranged to perform one or more tasks during operation. A structure may be referred to as being “configured” to perform a particular task even if the structure is not currently being operated. Therefore, an entity described or stated as being “configured” to perform a particular task refers to physical things such as devices, circuits, systems having processor units and memory storing program instructions that can be executed to perform that task, etc. This phrase is not used herein to refer to intangible things.

[0102] In some cases, various units / circuits / components may be described herein as performing a set of tasks or operations. It should be understood that, even if not specifically stated otherwise, these entities are "configured" to perform these tasks / operations.

[0103] The term "configured as" is not intended to mean "configurable as". For example, an unprogrammed FPGA would not be considered "configured as" to perform a specific function. However, the unprogrammed FPGA can be "configurable as" to perform that function. After proper programming, the FPGA can then be referred to as "configured as" to perform that specific function.

[0104] For the purposes of a U.S. patent application based on a published text, a statement structure in the claims that is “configured” to perform one or more tasks is expressly intended not to invoke 35 USC § 112(f) for that claim element. If an applicant wishes to invoke 112(f) during the examination of a U.S. patent application based on a published text, it will use the “means for [performing a function]” structure to state the claim element.

Claims

1. A method comprising: The server system receives queries from the computer system. The server system determines the grouped aspects of the query, which include at least the number of columns specified in the query and the computational cost of executing the query; The server system generates a query vector based on the grouped aspects of the query; The server system compares the query vector with multiple clusters, where each cluster includes two or more previously generated query vectors, which are generated based on aspects of queries previously received by the server system. The server system classifies the query based on the comparison, wherein the classification is based on the distance between the query vector and the plurality of clusters of the previously generated query vector; as well as The server system manages the query based on the classification of the query determined during the classification process.

2. The method of claim 1, further comprising, before receiving the query from the computer system: The server system generates multiple query vectors for multiple previously received queries, wherein generating the multiple query vectors includes identifying two or more aspects of the corresponding queries; and The server system performs a clustering operation on the plurality of query vectors, wherein the output of the clustering operation is a plurality of clusters representing two or more of the plurality of query vectors from the plurality of previously received queries.

3. The method according to claim 2, further comprising: The server system assigns the query code located at the center of the cluster in the plurality of clusters as the centroid of the corresponding cluster of the query code.

4. The method of claim 1, wherein the distance between the query vector and the plurality of clusters of the previously generated query vector is determined by: Measure the Euclidean distance between the query vector and the centroid of the corresponding cluster in the plurality of clusters of the previously generated query.

5. The method of claim 4, wherein the classification includes assigning anomaly classification to the query received from the computer system based on the distance between the query vector and the plurality of clusters being greater than a distance threshold requirement.

6. The method of claim 1, wherein managing the query based on the classification of the query comprises: The server system transmits an alert for the query to the computing system, wherein the alert specifies the category of the query and one or more preventative actions for the query.

7. The method of claim 1, wherein the management includes rejecting the query received from the computer system, including transmitting to the computer system a notification indicating that the query is unauthorized.

8. A non-transitory computer-readable medium having instructions stored thereon that enable a computing device to perform operations, the operations including: Receive queries from the computer system; Determine the grouped aspects of the query, the grouped aspects including at least the number of tables specified in the query and the number of modifications specified in the query; Generate a query vector based on the grouped aspects of the query; The query vector is compared with multiple clusters, the clusters of which include two or more previously generated query vectors, the previously generated query vectors being generated based on aspects of queries previously received by the computing device; The query is classified based on the comparison, wherein the classification is based on the distance between the query vector and the plurality of clusters of the previously generated query vector; as well as The query is managed based on the classification of the query determined during the classification process.

9. The non-transitory computer-readable medium of claim 8, wherein the operation further comprises, prior to receiving the query from the computer system: Generate multiple query vectors for multiple previously received queries, wherein generating the multiple query vectors includes identifying two or more aspects of the corresponding queries; and Perform a clustering operation on the plurality of query vectors, wherein the output of the clustering operation is a plurality of clusters representing two or more query vectors from the plurality of previously received queries, and The query code located at the center of the cluster in the plurality of clusters is assigned as the centroid of the corresponding cluster of the query code.

10. The non-transitory computer-readable medium of claim 8, wherein the aspect of determining the grouping of the query further includes determining the number of columns specified in the query and the computational cost of performing the query.

11. The non-transitory computer-readable medium of claim 8, wherein the management comprises: Transmit a request for one or more authentication factors to the computer system; as well as The query is rejected based on the computer system's response to the request for one or more authentication factors.

12. The non-transitory computer-readable medium of claim 8, wherein the management comprises: One or more preventative actions are progressively escalated based on the distance between the query vector and the plurality of clusters of the previously generated query vector, wherein the progressive escalation includes performing more restrictive preventative actions on the query vector that is at a greater distance from one or more of the plurality of clusters, compared to performing preventative actions on another query vector that is at a smaller distance from one or more of the plurality of clusters.

13. The non-transitory computer-readable medium of claim 8, wherein the distance between the query vector and the plurality of clusters of the previously generated query vector is determined by: Cosine similarity is used to measure the distance between the query vector and the centroid of the corresponding cluster in the plurality of clusters of the previously generated query.

14. The non-transitory computer-readable medium of claim 8, wherein the management comprises: A notification requesting a change to the query is transmitted to the computer system from which the query is received, wherein the notification specifies one or more defects in the query corresponding to the requested change.

15. A system comprising: At least one processor; and A memory, having stored instructions that can be executed by the at least one processor to cause the system to: Receive queries from the computer system; Determine the grouped aspects of the query, the grouped aspects including at least the query type of the query and one or more table object identifiers referenced in the query; Generate a query vector based on the grouped aspects of the query; The query vector is compared with multiple clusters, the clusters of which include two or more previously generated query vectors, the previously generated query vectors being generated based on aspects of queries previously received by the system; The query is classified based on the comparison, wherein the classification is based on the distance between the query vector and the plurality of clusters of the previously generated query vector; as well as The query is managed based on the classification of the query determined during the classification process.

16. The system of claim 15, wherein the instructions are further executable by the at least one processor to cause the system to: Generate multiple query vectors for multiple previously received queries, wherein generating the multiple query vectors includes identifying two or more aspects of the corresponding queries; and A clustering operation is performed on the plurality of query vectors, wherein the output of the clustering operation is a plurality of clusters representing two or more of the plurality of query vectors of the plurality of previously received queries.

17. The system of claim 16, wherein the instructions are further executable by the at least one processor to cause the system to: The query code located at the center of the cluster in the plurality of clusters is assigned as the centroid of the corresponding cluster of the query code.

18. The system of claim 15, wherein the classification includes assigning anomaly classification to the query received from the computer system based on the distance between the generated query vector and the plurality of clusters being greater than a distance threshold requirement.

19. The system of claim 15, wherein the management includes: Transmit a request for one or more authentication factors to the computer system; as well as Whether to execute the query is determined based on the response to the one or more authentication factors received from the computer system.

20. The system of claim 15, wherein the distance between the query vector and the plurality of clusters of the previously generated query vector is determined by: The inner product algorithm is used to measure the distance between the query vector and the centroid of the corresponding cluster in the plurality of clusters of the previously generated query.

Images