Data query method and device, electronic equipment and storage medium

Abstract

The application discloses a data query method and device, electronic equipment and storage medium. The method comprises the following steps: determining at least one dimension in an aggregation group as a dimension of a set type based on a first frequent item set and / or a first directed graph corresponding to each aggregation group of a set data model; generating at least one materialized view corresponding to each aggregation group according to the at least one dimension of the set type; and performing data query based on the generated materialized view; wherein the first directed graph is constructed based on at least one first association rule and a confidence degree of each first association rule in the at least one first association rule. In the above scheme, the dimension type of the aggregation group is identified based on the corresponding frequent item set and / or directed graph, the computing power and time consumed in the dimension type identification in the data query process are reduced, more SQL information can be processed, and the accuracy of the data query is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data query method, apparatus, electronic device and storage medium. Background Technology

[0002] A data warehouse (DW) is a relational database built to store data in a specific schema for easy multidimensional analysis and multi-perspective visualization. Its data is based on Online Transaction Processing (OLTP) systems. The data in a data warehouse is detailed, integrated, and subject-oriented, designed to meet the analytical needs of Online Analytical Processing (OLAP) systems. OLAP queries require pre-computed materialized view sets (also known as data cubes). Among related technologies, OLAP queries have low precision. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data query method, apparatus, electronic device, and storage medium to at least solve the problem of low query accuracy in OLAP in related technologies.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data query method, the method including:

[0006] Based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group of the defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type;

[0007] Based on at least one dimension of the defined type, generate at least one materialized view for each aggregation group;

[0008] Data queries are performed based on the generated materialized views; among which...

[0009] The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph.

[0010] In the above scheme, the first frequent itemset and / or the first directed graph corresponding to each aggregation group based on the defined data model, determining at least one dimension in the aggregation group as a dimension of a defined type, includes:

[0011] At least one dimension in the aggregation group that satisfies the first predefined condition is determined as a required dimension type; wherein...

[0012] The first set condition indicates that the corresponding dimension appears more than a first set threshold in the set SQL information and belongs to the first frequent itemset; the set SQL information is used to construct the set data model.

[0013] In the above scheme, the first frequent itemset and / or the first directed graph corresponding to each aggregation group based on the defined data model, determining at least one dimension in the aggregation group as a dimension of a defined type, includes:

[0014] Each dimension of the first dimension set is determined as a dimension of the joint dimension type; where...

[0015] All dimensions of the first dimension set can form a first path and a second path that are opposite to each other, and the corresponding first difference is less than a second set threshold; the first difference represents the absolute value of the difference between the weight of the first path and the weight of the second path.

[0016] In the above scheme, the first frequent itemset and / or the first directed graph corresponding to each aggregation group based on the defined data model, determining at least one dimension in the aggregation group as a dimension of a defined type, includes:

[0017] Traverse the first directed graph, and while traversing the first directed graph, add the first dimension in the first directed graph that satisfies the second set of conditions to the second dimension set. Then traverse the first directed graph again until there are no dimensions in the first directed graph that satisfy the second set of conditions. Then, determine each dimension in the second dimension set as a dimension of the hierarchical dimension type.

[0018] The second setting condition indicates that for any dimension of the second dimension set, there exists a unidirectional path pointing to the first dimension, and the relationship between any dimension of the second dimension set and the first dimension satisfies the third setting condition.

[0019] In the above scheme, the third setting condition includes:

[0020] There is one and only one unidirectional path between the second dimension set and the first dimension;

[0021] And / or,

[0022] When a third and fourth path exist between the second dimension and the first dimension, the second difference is greater than a third predetermined threshold; where,

[0023] The third path and the fourth path are opposite paths to each other; the second difference represents the absolute value of the difference between the weight of the third path and the weight of the fourth path.

[0024] In the above scheme, before determining at least one dimension in the aggregation group as a dimension of a set type, the method further includes:

[0025] Construct the first undirected graph based on the defined data model;

[0026] Based on the connectivity relationships in the first undirected graph, at least one aggregation group of the defined data model is determined.

[0027] In the above scheme, when determining at least one aggregation group of the set data model based on the connection relationship in the first undirected graph, the method includes:

[0028] The set of the third dimension corresponding to the first undirected graph is determined as the aggregation group of the defined data model; wherein...

[0029] The third dimension set represents the dimension set of the maximum connected graph corresponding to the first undirected graph.

[0030] This application also provides a data query device, including:

[0031] The first processing unit is used to determine at least one dimension in the aggregation group as a dimension of a specified type based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group of the specified data model.

[0032] A generation unit is configured to generate at least one materialized view for each aggregation group based on at least one dimension of the defined type.

[0033] The query unit is used to perform data queries based on the generated materialized view; among which,

[0034] The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph.

[0035] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0036] The processor is used to execute the steps of the above-described data query method when running the computer program.

[0037] This application embodiment also provides a storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-described data query method.

[0038] This application provides a data query method, apparatus, electronic device, and storage medium. Based on a first frequent itemset and / or a first directed graph corresponding to each aggregation group in a defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type. At least one materialized view is generated for each aggregation group according to the at least one dimension of the defined type. Data query is performed based on the generated materialized views. The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule. Each first association rule represents the association rule between two dimensions of the aggregation group corresponding to the first directed graph. In this scheme, the dimension type identification of the aggregation group is based on the corresponding frequent itemset and / or directed graph, reducing the computational power and time consumed in dimension type identification during data querying, supporting the processing of more SQL information, and thus improving the accuracy of data querying. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating the implementation process of the data query method provided in the embodiments of this application;

[0040] Figure 2 This is a schematic diagram of the required dimension type identification process provided in the embodiments of this application;

[0041] Figure 3 This is a schematic diagram of the joint dimension type identification process provided in the embodiments of this application;

[0042] Figure 4 This is a schematic diagram of a path in a directed graph provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the hierarchical dimension type identification process provided in the embodiments of this application;

[0044] Figure 6 A path diagram in a directed graph provided for another embodiment of this application;

[0045] Figure 7 A schematic diagram illustrating the implementation process of the dimension type identification method provided in the application embodiments of this application;

[0046] Figure 8 This is a schematic diagram of the structure of a data query device provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] A data cube (also known as a multidimensional model, materialized view set, or Cube) allows for multidimensional data modeling, defined by dimensions and facts. Data cubes are not limited to three-dimensional models and can combine more dimensions. For each dimension combination, the measures are aggregated, and the result is saved as a materialized view (also known as a Cuboid). The materialized views of all dimension combinations are called the materialized view set, which is a collection of materialized views aggregated by dimensions.

[0049] A data warehouse is a relational database established to store data in a specific schema for easy multidimensional analysis and multi-perspective presentation. Its data is based on OLTP systems. The data in a data warehouse is detailed, integrated, and subject-oriented, designed to meet the analytical needs of OLAP systems. OLAP queries require pre-computed data cubes. Among related technologies, OLAP queries have low precision.

[0050] Based on this, in various embodiments of this application, based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group of the defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type; according to the at least one dimension of the defined type, at least one materialized view is generated for each aggregation group; data query is performed based on the generated materialized view; wherein, the first directed graph is constructed based on at least one first association rule and the confidence of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph. In the above scheme, compared with dimension type identification through the cardinality calculation result of the dimension, dimension type identification of the aggregation group based on the corresponding frequent itemset and / or directed graph can reduce the computing power and time consumed by dimension type identification during the data query process, and the saved computing power supports electronic devices to process more SQL information, thereby improving the accuracy of data query.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] Figure 1 This is a schematic diagram illustrating the implementation flow of the data query method provided in this application embodiment. This application embodiment provides a data query method applied to electronic devices, wherein the electronic devices include, but are not limited to, servers, terminals, and other electronic devices. It includes:

[0053] Step 101: Based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group of the defined data model, determine at least one dimension in the aggregation group as a dimension of the defined type.

[0054] The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph.

[0055] In the process of generating the materialized view set, the defined data model corresponds to at least one aggregation group, and each aggregation group includes at least two dimensions of the defined data model. For aggregation groups that do not include any of the dimension types of joint dimension type, hierarchical dimension type, or required dimension type, those skilled in the art can use corresponding construction methods to construct the data model, which is not limited here; for aggregation groups that include at least one dimension type of joint dimension type, hierarchical dimension type, or required dimension type, algorithms can be used to determine the first frequent itemset and at least one first association rule corresponding to each aggregation group, and then a first directed graph can be constructed based on the determined at least one first association rule and the confidence of each first association rule. The defined data model represents the defined data model constructed based on the source data. In a scenario where defined SQL information is used as the source data, the statistical information of each SQL statement can be transformed into a query pattern to construct the data model. Here, the dimension can represent the data dimension of the defined data model. The aggregation group represents the set of defined dimensions in the defined data model. Here, frequent pattern mining algorithms, such as Apriori, Eclat, or FP-Growth, can be used to extract the frequent itemsets and association rules corresponding to the statistical information of the SQL statements involved in the defined data model.

[0056] Based on the first frequent itemset and / or the first directed graph corresponding to the aggregation group, dimension type identification is performed on the dimensions in the aggregation group, and at least one dimension is determined to be a dimension of a specified type. The specified type, which can be used as a condition for filtering materialized views, includes at least one of the following: joint dimension type, hierarchical dimension type, and required dimension type. In practical applications, determining a dimension as a required dimension type can halve the number of materialized views to be generated.

[0057] Before determining at least one dimension in an aggregation group as a dimension of a given type, at least one aggregation group can be set up for a given data model, and then a corresponding first frequent itemset and / or first directed graph can be determined for each aggregation group in the set at least one aggregation group. As an embodiment of this application, before determining at least one dimension in an aggregation group as a dimension of a given type, the method further includes:

[0058] Construct the first undirected graph based on the defined data model;

[0059] Based on the connectivity relationships in the first undirected graph, at least one aggregation group of the defined data model is determined.

[0060] Based on the statistical information of the SQL statements used to construct the defined data model, a first undirected graph is constructed for the defined data model, and at least one aggregation group of the defined data model is determined according to whether there are connections between the dimensions in the first undirected graph.

[0061] Using some or all of the SQL statement statistics used to construct the defined data model, a first undirected graph is built for the defined data model. During the construction of the first undirected graph, if two dimensions of the defined data model appear in the same defined field of the SQL statement, an edge is created between these two dimensions in the first undirected graph; that is, a connection is established between these two dimensions in the first undirected graph.

[0062] Here, the connectivity relationships in the first undirected graph can form a corresponding connected graph, and the aggregation group of the set data model is determined based on some or all of the nodes in the connected graph. As an embodiment of this application, when determining at least one aggregation group of the set data model based on the connectivity relationships in the first undirected graph, the method includes:

[0063] The set of the third dimension corresponding to the first undirected graph is determined as the aggregation group of the defined data model; wherein...

[0064] The third dimension set represents the dimension set of the maximum connected graph corresponding to the first undirected graph.

[0065] All dimensions in the maximally connected graph corresponding to the first undirected graph are defined as an aggregation group. Alternatively, at least two discrete dimensions in the first undirected graph that are not part of the connected graph can also be defined as an aggregation group.

[0066] As an embodiment of this application, determining whether a dimension in an aggregation group is a required dimension type, see [link to documentation]. Figure 2 Based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group in the defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type, including:

[0067] Step 201: Determine at least one dimension in the aggregation group that satisfies the first set condition as a dimension of the required dimension type.

[0068] Wherein, the first set condition indicates that the corresponding dimension appears more than a first set threshold in the set SQL information and belongs to the first frequent itemset; the set SQL information is used to construct the set data model.

[0069] The algorithm determines whether the dimensions in the aggregation group meet the set conditions, and identifies the dimension types of dimensions that meet the first set condition as mandatory dimension types. Specifically, dimensions that meet the first set condition belong to the first frequent itemset and appear more than a first set threshold in the set SQL information used to construct the set data model. Here, determining the number of times a dimension appears in the set SQL information can be done by counting the number of times the dimension appears in each SQL statement within the set SQL information.

[0070] Here, to determine whether the dimensions in the aggregation group meet the set conditions, we can first filter out the dimensions that belong to the first frequent itemset in the aggregation group, and then determine the dimensions that appear more than the first set threshold in the set SQL information among these dimensions.

[0071] Compared to identifying dimension types by calculating the cardinality of dimensions, identifying the required dimension types of aggregation groups based on the corresponding frequent itemsets can reduce the computational effort required for identifying required dimension types.

[0072] As an embodiment of this application, determining whether a dimension in an aggregation group is a joint dimension type, see [link to documentation]. Figure 3 Based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group in the defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type, including:

[0073] Step 301: Determine each dimension of the first dimension set as a dimension of the joint dimension type.

[0074] Wherein, all dimensions of the first dimension set can form a first path and a second path that are opposite to each other, and the corresponding first difference is less than a second set threshold; the first difference represents the absolute value of the difference between the weight of the first path and the weight of the second path.

[0075] Traverse the first directed graph. The set of dimensions in the first directed graph that satisfies the set of predetermined conditions is called the first dimension set. The dimension type of each dimension in the first dimension set is determined as a joint dimension type. Specifically, all dimensions in the first dimension set that satisfy the predetermined conditions can form a bidirectional loop path (opposite paths) in the first directed graph, namely the first path and the second path, where the absolute value of the difference between the weights of the first path and the second path is less than a second predetermined threshold. The path weights can be determined based on the sum of the confidence scores of the first association rules corresponding to every two adjacent dimensions in the path.

[0076] by Figure 4 The diagram shown below illustrates the path diagram of the joint dimension in a directed graph. The solid line represents the first path, and the dashed line represents the second path. The first path and the second path are opposite paths to each other.

[0077] Compared to identifying dimension types by calculating the cardinality of dimensions, identifying joint dimension types based on the corresponding directed graph and the aggregation group can reduce the computational effort required for joint dimension type identification.

[0078] As an embodiment of this application, determining whether the dimension in the aggregation group is a hierarchical dimension type, see [link to documentation]. Figure 5 Based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group in the defined data model, at least one dimension in the aggregation group is determined as a dimension of a defined type, including:

[0079] Step 501: Traverse the first directed graph, and while traversing the first directed graph, add the first dimension in the first directed graph that satisfies the second set condition to the second dimension set, and then traverse the first directed graph again until there are no dimensions in the first directed graph that satisfy the second set condition, and determine each dimension of the second dimension set as a dimension of the hierarchical dimension type.

[0080] The second setting condition indicates that for any dimension of the second dimension set, there exists a unidirectional path pointing to the first dimension, and the relationship between any dimension of the second dimension set and the first dimension satisfies the third setting condition.

[0081] When traversing the first directed graph, determine if the first directed graph contains a first dimension that satisfies the second condition. If the first directed graph contains a first dimension, add this first dimension to the second dimension set, and traverse the first directed graph again until there are no dimensions in the first directed graph that satisfy the second condition. Then, determine each dimension in the second dimension set as a hierarchical dimension. Specifically, during the current traversal of the first directed graph, if any dimension in the current second dimension set has a unidirectional path pointing to another dimension, and the relationship between any dimension in the current second dimension set and the first dimension satisfies the third condition, then this dimension is determined as the first dimension that satisfies the second condition.

[0082] As one embodiment of this application, the third setting condition includes:

[0083] There is one and only one unidirectional path between the second dimension set and the first dimension;

[0084] And / or,

[0085] When a third and fourth path exist between the second dimension and the first dimension, the second difference is greater than a third predetermined threshold; where,

[0086] The third path and the fourth path are opposite paths to each other; the second difference represents the absolute value of the difference between the weight of the third path and the weight of the fourth path.

[0087] The third condition can be set as follows: there are no two unidirectional paths that are opposite to each other between the dimensions of the second dimension set and the first dimension. In other words, there is one and only one unidirectional path between the dimensions of the second dimension set and the first dimension.

[0088] by Figure 6 Taking the path diagram of the hierarchical dimensions in the directed graph as an example, the black-filled shape represents the first dimension, and the white-filled shape represents the dimensions of the second dimension set. There are no two paths that are opposite to each other between any dimension of the second dimension set and the first dimension. There is one and only one unidirectional path from the dimensions of the second dimension set to the first dimension.

[0089] The third condition can be set as follows: there are two unidirectional paths that are opposite to each other between the dimensions of the second dimension set and the first dimension, namely the third path and the fourth path, but the absolute value of the difference between the weights of these two paths is greater than the third threshold.

[0090] The third condition can also be a combination of the two conditions mentioned above. The third condition is satisfied when the dimensions of the second dimension set and the first dimension satisfy one of the two conditions mentioned above.

[0091] Compared to identifying dimension types by calculating the cardinality of dimensions, identifying hierarchical dimension types based on the corresponding directed graph and the aggregation group can reduce the computational effort required for hierarchical dimension type identification.

[0092] Step 102: Generate at least one materialized view for each aggregation group based on at least one dimension of the defined type.

[0093] Based on at least one defined dimension, a subset of materialized views to be generated is selected from multiple materialized views, and the selected materialized views are generated based on a defined data model. This reduces the number of materialized views that need to be generated, thus achieving pruning of the materialized view set.

[0094] Step 103: Perform data query based on the generated materialized view.

[0095] Data querying based on materialized views generated from pre-calculated data models improves the speed of data querying for set data models.

[0096] In this embodiment, the dimension type identification of the aggregation group is performed based on the corresponding frequent itemsets and / or directed graphs, which reduces the computing power and time consumed by dimension type identification during the data query process. The saved computing power supports electronic devices to process more SQL information, thereby improving the accuracy of data query.

[0097] The present application will be further described in detail below with reference to application examples.

[0098] Data in a data warehouse is detailed, integrated, and subject-oriented, designed to meet the analytical needs of OLAP systems. OLAP queries require pre-computed data cubes. During data warehouse index modeling, to prune the data cubes, it's necessary to statistically analyze the cardinality of relevant columns in the query history based on the designed data model. This cardinality is then used to identify and classify dimension types; for example, C(A) = C(A,B) can be used to determine if A and B are hierarchical dimensions. This cardinality calculation step consumes significant cluster computing power; for instance, calculating the cardinality for over 100 dimensions involves processing tens of millions of data points, and consequently, building the data cubes also requires a considerable amount of time.

[0099] Based on this, this application example proposes a dimension type identification scheme based on frequent pattern mining. A data model is designed based on the statistical information obtained by parsing the given user's historical SQL input. Then, according to the SQL information contained in the data model, frequent itemsets and association rules are extracted using a frequent pattern mining algorithm. The dimensions are distinguished based on the extracted information. Figure 7 The following is a schematic diagram illustrating the implementation flow of the dimension type recognition method provided in the application embodiment of this application. The specific steps are as follows:

[0100] 1. Set up aggregation groups.

[0101] Using graph algorithms, dimensions appearing in the data model are treated as points. If any two points appear in the same SQL statement's select, groupby, or where fields, an edge is added between them, thus constructing an undirected graph. Finally, all maximally connected elements in the undirected graph are returned as an aggregation group. Furthermore, all discrete points are grouped together as another aggregation group.

[0102] Here, an aggregation group is a category that a user can categorize based on the dimensions they are interested in. These categories are called aggregation groups in Apache Kylin.

[0103] 2. Extract association rules and frequent itemsets.

[0104] Based on the SQL information involved in each data model, frequent itemsets and association rules are extracted using frequent pattern mining algorithms (such as Apriori, Eclat, FP-Growth, etc.). A directed graph is then constructed based on the association rules and their confidence levels. Some paths in the directed graph can be used to distinguish the types of dimensions.

[0105] Among them, the extracted association rules can be strong association rules, that is, association rules that satisfy minimum support and minimum confidence.

[0106] Here, the frequent pattern mining algorithm has two concepts: support and confidence. These are set before the algorithm is executed. After each iteration, itemsets with a support greater than the required support are retained as frequent itemsets. The final generated rules are composed of these frequent itemsets. Support: support(A) = P(A), representing the probability of A occurring. support(A=>B) = P(A∪B), representing the probability of A and B occurring simultaneously. Confidence: confidence(A=>B) = support(A∪B) / support(A), representing the ratio of the probability of A and B occurring simultaneously to the probability of A occurring. Frequent itemsets: Let U = {u1, u2, ..., un} be a set of items, and UI... For a given minimum support min_sup, if the support of itemset U, support(U), ​​is ≥ min_sup, then U is called a frequent itemset; otherwise, U is a non-frequent itemset. Association rules: Association rules are implications of the form X => Y, where X and Y are proper subsets of I, and... X is called the premise of the rule, and Y is called the result of the rule. Association rules reflect the pattern that when an item in X appears, an item in Y also appears.

[0107] 3. Determine the necessary dimensions.

[0108] The input consists of the aggregation group containing dimensions, frequent itemsets, and a set threshold. First, the dimensions must be in the frequent itemsets. Second, for each dimension in the aggregation group, a score is calculated based on its frequency in the SQL statement. If the score exceeds a certain threshold, the corresponding dimension is set as a required dimension and removed from the directed graph. The remaining dimensions are returned as ordinary dimensions.

[0109] Here, users may be particularly interested in one or several dimensions, and the group by dimension appears in all query requests. This dimension is called a required dimension.

[0110] 4. Calculate the joint dimension.

[0111] The input consists of the normal dimensions returned by the required dimensions, a directed graph generated according to the association rules, and a set threshold. The directed graph is traversed to check for paths with bidirectional cycles, where the absolute difference between the weights of opposite paths is less than the threshold. If such a cycle exists and is included in the normal dimensions returned by the required dimensions, it is set as the joint dimension, and the set dimension is removed from the directed graph; the remaining dimensions are returned as normal dimensions.

[0112] Here, users sometimes do not care about the various detailed combinations of dimensions. For example, if a user's query statement contains "group by A, B, C", it will only contain "group by A, B, C" and not "group by A, B" or "group by C", etc. These detailed combinations of dimensions are called joint dimensions.

[0113] 5. Calculate the hierarchical dimensions.

[0114] The input consists of the ordinary dimensions returned by the joint dimension, a directed graph generated according to the association rules, and a set threshold. The task is to traverse the directed graph to check if the following exists: Figure 6 The specified path refers to a unidirectional path that exists directly between any dimension already added to the dimension set and the specified dimension, and where there is no opposite path between the dimensions in the dimension set and the specified dimension, or where there is an opposite path but the absolute value of the difference in the corresponding weights is greater than a set threshold. If there is a specified dimension that meets the conditions, and the specified dimension is included in the ordinary dimensions returned by the joint dimension, the specified dimension is added to the dimension combination until it is traversed to the point where no specified dimension exists. The updated dimension combination is then set as a hierarchical dimension, and the rest are returned as ordinary dimensions.

[0115] Here, the dimensions selected by the user often include those with hierarchical relationships; these dimensions are called hierarchical dimensions. For example, for the three dimensions of country, province, and city, there is a one-to-many relationship between country / province / city from top to bottom.

[0116] This application example aims to differentiate dimensions using frequent pattern mining algorithms, employing frequent itemsets and association rules to partition dimensions, avoiding time-consuming cardinality calculations, and significantly improving the speed of building data models and data cubes. The solution in this example constructs the data cube in an extremely short time; the rule extraction time for 10,000 SQL statements is only 0.1323 seconds. In contrast, calculating cardinality for single or double columns involving hundreds of dimensions typically takes over ten minutes, representing a speed improvement of four orders of magnitude. This allows for processing a larger amount of SQL information in a single index recommendation, improving the accuracy of data queries.

[0117] To implement the method of the embodiments of this application, the embodiments of this application also provide a data query device, such as... Figure 8 As shown, the device includes:

[0118] The first processing unit 801 is used to determine at least one dimension in the aggregation group as a dimension of a specified type based on the first frequent itemset and / or the first directed graph corresponding to each aggregation group of the specified data model.

[0119] The generation unit 802 is configured to generate at least one materialized view for each aggregation group based on at least one dimension of the set type.

[0120] Query unit 803 is used to perform data queries based on the generated materialized view; wherein,

[0121] The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph.

[0122] In one embodiment, the first processing unit 801 is configured to:

[0123] At least one dimension in the aggregation group that satisfies the first predefined condition is determined as a required dimension type; wherein...

[0124] The first set condition indicates that the corresponding dimension appears more than a first set threshold in the set SQL information and belongs to the first frequent itemset; the set SQL information is used to construct the set data model.

[0125] In one embodiment, the first processing unit 801 is configured to:

[0126] Each dimension of the first dimension set is determined as a dimension of the joint dimension type; where...

[0127] All dimensions of the first dimension set can form a first path and a second path that are opposite to each other, and the corresponding first difference is less than a second set threshold; the first difference represents the absolute value of the difference between the weight of the first path and the weight of the second path.

[0128] In one embodiment, the first processing unit 801 is configured to:

[0129] Traverse the first directed graph, and while traversing the first directed graph, add the first dimension in the first directed graph that satisfies the second set of conditions to the second dimension set. Then traverse the first directed graph again until there are no dimensions in the first directed graph that satisfy the second set of conditions. Then, determine each dimension in the second dimension set as a dimension of the hierarchical dimension type.

[0130] The second setting condition indicates that for any dimension of the second dimension set, there exists a unidirectional path pointing to the first dimension, and the relationship between any dimension of the second dimension set and the first dimension satisfies the third setting condition.

[0131] In one embodiment, the third setting condition includes:

[0132] There is one and only one unidirectional path between the second dimension set and the first dimension;

[0133] And / or,

[0134] When a third and fourth path exist between the second dimension and the first dimension, the second difference is greater than a third predetermined threshold; where,

[0135] The third path and the fourth path are opposite paths to each other; the second difference represents the absolute value of the difference between the weight of the third path and the weight of the fourth path.

[0136] In one embodiment, the apparatus further includes:

[0137] The second processing unit is configured to construct a first undirected graph based on a set data model before the first processing unit 801 determines at least one dimension in the aggregation group as a dimension of a set type; and to determine at least one aggregation group of the set data model according to the connection relationship in the first undirected graph.

[0138] In one embodiment, the second processing unit is configured to:

[0139] The third dimension set corresponding to the first undirected graph is determined as the aggregation group of the set data model;

[0140] in,

[0141] The third dimension set represents the dimension set of the maximum connected graph corresponding to the first undirected graph.

[0142] In practical applications, the first processing unit 801, the generation unit 802, the query unit 803, and the second processing unit can be implemented by processors in the data query device, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0143] It should be noted that the data query device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data query device and the data query method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0144] Based on the hardware implementation of the above program modules, and in order to implement the data query method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 9 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, as shown below. Figure 9 As shown, the electronic device includes:

[0145] Communication interface 1 enables information exchange with other devices, such as network devices;

[0146] Processor 2 is connected to communication interface 1 to enable information interaction with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running computer programs. The computer program is stored in memory 3.

[0147] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 9 The general will label all buses as Bus System 4.

[0148] The memory 3 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0149] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 2 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] The methods disclosed in the embodiments of this application can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0151] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0152] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 that stores a computer program, which can be executed by a processor 2 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0156] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0157] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0158] It is understood that, in the embodiments of this application, data involving user information, when applied to specific products or technologies, requires user permission or consent, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0159] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict. Unless otherwise stated and limited, the term "connection" should be interpreted broadly. For example, it can refer to an electrical connection, or the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above term according to the specific circumstances.

[0160] Furthermore, in the examples of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first," "second," and "third" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein.

[0161] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0163] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A data query method, characterized in that, The method includes: Based on the first directed graph corresponding to each aggregation group of the defined data model, at least one dimension in the aggregation group is determined to be a dimension of a defined type. Based on at least one dimension of the defined type, generate at least one materialized view for each aggregation group; Data queries are performed based on the generated materialized views; among which... The first directed graph is constructed based on at least one first association rule and the confidence level of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph; Specifically, the first directed graph corresponding to each aggregation group based on the defined data model determines at least one dimension in the aggregation group as a dimension of a defined type, including: In the first directed graph, the dimensions of paths containing bidirectional cycles where the absolute value of the difference between the weights of opposite paths is less than a threshold are defined as joint dimension types. And / or, Traverse the first directed graph. If there exists a specific dimension in the first directed graph that has a unidirectional path with any dimension in the dimension set and no opposite path with any dimension, or has an opposite path with any dimension and the absolute value of the difference between the weights of the opposite paths is greater than a set threshold, and the specific dimension is not a dimension of the joint dimension type or a dimension of the required dimension type, add the specific dimension to the dimension set. Continue traversing until no specific dimension exists. Then, determine the dimensions in the updated dimension set as dimensions of the hierarchical dimension type. The dimensions of the set type include at least one of the dimensions of the joint dimension type and the dimensions of the hierarchical dimension type.

2. The method according to claim 1, characterized in that, The first directed graph corresponding to each aggregation group based on the defined data model determines at least one dimension in the aggregation group as a dimension of a defined type, including: Traverse the first directed graph, and while traversing the first directed graph, add the first dimension in the first directed graph that satisfies the second set of conditions to the second dimension set. Then traverse the first directed graph again until there are no dimensions in the first directed graph that satisfy the second set of conditions. Then, determine each dimension in the second dimension set as a dimension of the hierarchical dimension type. The second setting condition indicates that for any dimension of the second dimension set, there exists a unidirectional path pointing to the first dimension, and the relationship between any dimension of the second dimension set and the first dimension satisfies the third setting condition; the third setting condition includes that there is one and only one unidirectional path between the dimensions of the second dimension set and the first dimension.

3. The method according to claim 2, characterized in that, The third setting condition also includes: When a third and fourth path exist between the second dimension and the first dimension, the second difference is greater than a third predetermined threshold; where, The third path and the fourth path are opposite paths to each other; the second difference represents the absolute value of the difference between the weight of the third path and the weight of the fourth path.

4. The method according to any one of claims 1 to 3, characterized in that, Before determining at least one dimension in the aggregation group as a dimension of a given type, the method further includes: Construct the first undirected graph based on the defined data model; Based on the connectivity relationships in the first undirected graph, at least one aggregation group of the defined data model is determined.

5. The method according to claim 4, characterized in that, When determining at least one aggregation group of the defined data model based on the connectivity relationships in the first undirected graph, the method includes: The set of the third dimension corresponding to the first undirected graph is determined as the aggregation group of the defined data model; wherein... The third dimension set represents the dimension set of the maximum connected graph corresponding to the first undirected graph.

6. A data query device, characterized in that, include: The first processing unit is used to determine at least one dimension in each aggregation group as a dimension of a specified type based on the first directed graph corresponding to each aggregation group of the specified data model. A generation unit is configured to generate at least one materialized view for each aggregation group based on at least one dimension of the defined type. The query unit is used to query data based on the generated materialized view; The first processing unit is further configured to: determine the dimensions in the first directed graph that have bidirectional loops and whose absolute difference in weights between opposite paths is less than a threshold as dimensions of the joint dimension type; and / or, traverse the first directed graph, and if there exists a specific dimension in the first directed graph that has a unidirectional path with any dimension in the dimension set and no opposite path with any dimension, or has an opposite path with any dimension and whose absolute difference in weights between opposite paths is greater than a set threshold, and the specific dimension is not a dimension of the joint dimension type or a dimension of the mandatory dimension type, add the specific dimension to the dimension set until no specific dimension exists, and determine the dimensions in the updated dimension set as dimensions of the hierarchical dimension type; wherein, the first directed graph is constructed based on at least one first association rule and the confidence of each of the at least one first association rule; each first association rule represents the association rule of two dimensions of the aggregation group corresponding to the first directed graph; the dimension of the set type includes at least one of the dimensions of the joint dimension type and the dimensions of the hierarchical dimension type.

7. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images