Data processing method and apparatus, computer-readable storage medium, and electronic device

Abstract

The present disclosure provides a data processing method and device, a computer readable storage medium and an electronic device. The method comprises: obtaining dimension fields in a historical query statement of a target table and query co-occurrence frequencies between different dimension fields; taking the dimension fields as nodes, determining first-order edge weights between different nodes according to the query co-occurrence frequencies between different dimension fields, and forming a dimension co-occurrence graph; determining a first core node in the dimension co-occurrence graph according to degrees of each node in the dimension co-occurrence graph; obtaining a first connected subgraph after the first core node in the dimension co-occurrence graph is deleted; obtaining a split tree of the dimension co-occurrence graph according to the dimension co-occurrence graph, the first core node and the first connected subgraph; determining an aggregated group of the dimension fields according to a target leaf node of the split tree; and constructing a data cube for the target table according to the aggregated group of the dimension fields. The constructed data cube can be stored in a database to improve query efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the development of mobile internet, the Internet of Things, and other technologies, the accumulated data is growing explosively, ushering in the era of big data. Collecting massive amounts of data is only the first step in big data technology; the ultimate goal of the big data field is to generate value from that data. Hadoop (Haydup, a distributed system infrastructure) solved the data storage problem, but a satisfactory solution for real-time querying of massive amounts of data has yet to be found. In most cases, queries need to respond to user actions in real time, and the response time of query engines in related technologies, often several minutes or even tens of minutes, is clearly insufficient to meet these requirements.

[0003] In some data platform products, data analysis, or reporting operations, it is often necessary to choose an OLAP (Online Analytical Processing) data query engine. Apache Kylin, an open-source OLAP engine built on the Hadoop big data platform, employs multidimensional cube (also known as data cube) pre-computation technology. By trading space for time, compared to traditional MPP (Massive Parallel Processing) architecture OLAP query engines, it improves query speed to sub-second levels and offers very high concurrency capabilities, greatly enhancing the efficiency of data analysis. The emergence of Apache Kylin not only effectively solves the problem of fast querying of massive amounts of data but also avoids a series of troubles caused by manually developing and maintaining pre-computation programs.

[0004] When using Apache Kylin as the OLAP query engine as a technical solution, we face the problem of computing complex cubes: the pre-computation of cubes with ultra-large dimension combinations generally uses Apache Spark as the building engine. However, the computing tasks of large cubes will put a lot of pressure on computing resources. Not only will they consume a lot of computing resources, but they will also have extremely long build times. This is not friendly to some businesses with high data timeliness. Moreover, ultra-large Spark computing tasks may fail due to resource issues, leading to more waste of resources. Summary of the Invention

[0005] This disclosure provides a data processing method and apparatus, a computer-readable storage medium, and an electronic device, which can solve the technical problems in the above-mentioned related technologies where Cube construction takes a long time and consumes a lot of computing resources.

[0006] This disclosure provides a data processing method, comprising: obtaining dimension fields and query co-occurrence frequencies between different dimension fields in historical query statements of a target table; using dimension fields as nodes, determining the initial edge weights between different nodes based on the query co-occurrence frequencies between different dimension fields to form a dimension co-occurrence graph; determining the first core node in the dimension co-occurrence graph based on the degree of each node; obtaining a first connected subgraph after deleting the first core node in the dimension co-occurrence graph; obtaining a split tree of the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and the first connected subgraph; determining the aggregation group of dimension fields based on the target leaf nodes of the split tree; and constructing a data cube oriented towards the target table based on the aggregation group of dimension fields.

[0007] In some exemplary embodiments of this disclosure, determining the associated dimension group pairs that satisfy the merging conditions in each associated dimension group based on the relevance and merging expansion rate between each associated dimension group includes: if the relevance between the associated dimension groups is greater than a first relevance threshold; or if the relevance between the associated dimension groups is greater than a second relevance threshold and the merging expansion rate is less than a first expansion rate threshold; or if the relevance between the associated dimension groups is greater than a third relevance threshold and the merging expansion rate is less than a second expansion rate threshold, then determining the corresponding associated dimension group as an associated dimension group pair that satisfies the merging conditions; wherein, the first relevance threshold is greater than the second relevance threshold, the second relevance threshold is greater than the third relevance threshold, and the first expansion rate threshold is greater than the second expansion rate threshold.

[0008] This disclosure provides a data processing apparatus, comprising: a co-occurrence frequency acquisition unit, configured to acquire dimension fields and the query co-occurrence frequencies between different dimension fields in historical query statements of a target table; a dimension co-occurrence graph construction unit, configured to use dimension fields as nodes, determine the initial edge weights between different nodes based on the query co-occurrence frequencies between different dimension fields, and form a dimension co-occurrence graph; a core node determination unit, configured to determine the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph; a connected subgraph acquisition unit, configured to acquire the first connected subgraph after deleting the first core node in the dimension co-occurrence graph; a graph split tree acquisition unit, configured to acquire the split tree of the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and the first connected subgraph; a field aggregation group determination unit, configured to determine the aggregation groups of dimension fields based on the target leaf nodes of the split tree; and a data cube construction unit, configured to construct a data cube oriented towards the target table based on the aggregation groups of dimension fields.

[0009] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described in the above embodiments.

[0010] This disclosure provides an electronic device, including: at least one processor; and a storage device configured to store at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the data processing method as described in the above embodiments.

[0011] In some embodiments of this disclosure, the technical solutions provided include, on the one hand, analyzing historical query statements of the target table, constructing a dimension co-occurrence graph based on the dimension fields in the historical query statements and the frequency of co-occurrence between different dimension fields, determining the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph, obtaining the first connected subgraph after deleting the first core node in the dimension co-occurrence graph, splitting the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and the first connected subgraph, obtaining a split tree of the dimension co-occurrence graph, thereby determining the aggregation group of the dimension fields in the aforementioned historical query statements based on the target leaf nodes of the split tree, and when using the aggregation group of the dimension fields to construct a data cube (Cube) for the target table, the large Cube is split into aggregation groups, and each aggregation group can be used to construct each smaller Cube. That is, by splitting the dimension co-occurrence graph, the data scale is compressed, the computing resources consumed in the Cube construction process are reduced, and the Cube construction time is shortened, achieving a balance between construction resources and construction time. Meanwhile, since the dimension co-occurrence graph in this embodiment is constructed based on the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, splitting this dimension co-occurrence graph allows the construction task of a large Cube to be divided into multiple smaller Cube construction tasks. This ensures that each smaller Cube can meet the query requirements of actual business operations without increasing the amount of redundant calculations. On the other hand, by analyzing historical query statements, the Cube construction is automated, simplifying the difficulty for users to design Cubes and optimizing Cube design. Attached Figure Description

[0012] Figure 1 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 2 A schematic diagram of a dimensional co-occurrence graph according to an embodiment of the present disclosure is shown.

[0014] Figure 3 The illustration shows the deletion. Figure 2A schematic diagram of the connected subgraph following the node with the highest degree in the dimensional co-occurrence graph.

[0015] Figure 4 The illustration shows the deletion. Figure 3 A schematic diagram of the connected subgraph following the node with the highest degree in the connected subgraph.

[0016] Figure 5 The illustration schematically shows a process diagram of dimensional co-occurrence graph splitting according to an embodiment of the present disclosure.

[0017] Figure 6 The diagram illustrates a structural schematic of a split tree of a dimensional co-occurrence graph according to an embodiment of the present disclosure.

[0018] Figure 7 The illustration shows a schematic diagram of a pruning operation on a split tree according to an embodiment of the present disclosure.

[0019] Figure 8 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown.

[0020] Figure 9 A schematic flowchart of a data processing method according to yet another embodiment of the present disclosure is shown.

[0021] Figure 10 The diagram illustrates a system architecture diagram of the data processing method provided in the embodiments of this disclosure.

[0022] Figure 11 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0025] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0026] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0029] In related technologies, allowing users to design their own Cube aggregation groups presents challenges. When faced with highly complex business scenarios, users need to spend a significant amount of time understanding the business before designing the Cube. They typically encounter problems such as determining which dimension combinations can be grouped into an aggregation group; which dimension combinations within an aggregation group can form a joint dimension; and how to design the RowKey to improve query speed. The dimension combinations of user-designed Cubes are often enormous, and user-designed Cubes do not utilize historical query data, resulting in Cubes that cannot adequately meet actual query needs.

[0030] Furthermore, designing a good cube demands a high level of expertise from developers. They need to understand the data, be familiar with the business logic, and be proficient in the principles of Kylin and Spark. The design process is not straightforward; it involves continuous adjustments to arrive at the final result. Therefore, automating cube design is a crucial task.

[0031] In view of the technical problems existing in the above-mentioned related technologies, the present disclosure proposes a data processing method to at least partially solve the above problems. The methods provided in the embodiments of the present disclosure can be executed by any electronic device, such as a server, or a terminal, or interaction between a server and a terminal, and the present disclosure does not limit this.

[0032] The server mentioned in this disclosure can be an independent server, a server cluster or distributed system composed of multiple servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0033] The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication, and this disclosure does not impose any restrictions.

[0034] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0035] Figure 1 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically. Figure 1 As shown, the method provided in this disclosure embodiment may include the following steps.

[0036] In step S110, the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields are obtained.

[0037] In this embodiment, the target table refers to the table from which the Cube is to be built. The target table can come from the same data source. In this embodiment, the data source can be Hive (a data warehouse tool based on Hadoop) or Kafka (a distributed publish-subscribe messaging system), etc. This disclosure does not limit the specific data source; in the following embodiments, Hive is used as an example. Therefore, the same data source refers to the same table or the same combination of fact table and dimension table on Hive. When the target table comes from different data sources, multiple Cubes are built that interact with multiple data sources.

[0038] A fact table is a table that stores factual records, such as system logs, sales records, and sensor values. A dimension table, also known as a lookup table, is a table that corresponds to a fact table; it stores the attribute values ​​of dimensions and can be associated with the fact table. Essentially, it extracts and standardizes frequently repeated attributes from the fact table and manages them in a single table. Common dimension tables include: date tables (storing attributes such as week, month, and quarter corresponding to dates) and region tables (containing attributes such as country, province / state, and city).

[0039] In this embodiment of the disclosure, the historical query statements of the target table can refer to the SQL (Structured Query Language) statements that the user has queried on Hive against the same data source in the past. These historical SQL statements can be stored in the query log.

[0040] In this embodiment of the disclosure, a dimension is the perspective from which data is observed, and is generally a set of discrete values, such as each independent date in the time dimension, each independent product in the product dimension, each independent device in the device dimension, etc. A metric, on the other hand, is an aggregated statistical value, that is, the result of aggregation calculations (such as summation, mean, maximum, minimum, etc.), and is generally a continuous value, such as sales revenue, average selling price, total number of items sold, etc.

[0041] By analyzing the fields involved in historical query statements, we can categorize them, determine whether they are dimension fields, and determine the query co-occurrence frequency of different dimension fields. Query co-occurrence frequency refers to the number of times different dimension fields appear simultaneously in the same historical query statement across all historical queries on the target table.

[0042] In this embodiment of the disclosure, an SQL Parser can be used to convert the SQL statements in the historical query statements into an AST (Abstract Syntax Tree). Then, by traversing the edges of the AST, the fields of the physical table (a table in a data source) corresponding to each SQL statement involved in all historical query statements can be extracted, and the field type of each field can be labeled. For example, the labeled field type can include aggregation fields (group), filter fields (filter), and indicator fields, etc.

[0043] For example, the following SQL statement: SELECT user_id,user_name,sum(value)as valueFROM user_data WHERE partition_time=20200101GROUP BY user_id,user_name, will produce the following result:

[0044] [(user_data, user_id, ("group")),

[0045] (user_data, user_name, ("group")),

[0046] (user_data, partition_time, ("filter")),

[0047] (user_data, value, ("sum"))]

[0048] Assuming the target table for the historical query is named `user_data`, the SQL statement above extracts the fields `user_id` (user identifier), `user_name` (username), `partition_time` (partition time), and `value`. The `user_id` and `user_name` fields are of aggregate type, the `partition_time` field is of filter type, and the `value` field is of metric type. The `value` field, being a metric field, will not be included in the query network. The `user_id`, `user_name`, and `partition_time` fields are then designated as dimension fields.

[0049] In step S120, the dimension fields are used as nodes, and the initial edge weights between different nodes are determined based on the query co-occurrence frequency between different dimension fields to form a dimension co-occurrence graph.

[0050] In this embodiment of the disclosure, a dimension co-occurrence graph can be constructed using the dimension fields and query co-occurrence frequencies of different dimension fields obtained in step S110 above. The initial edge weight represents the query co-occurrence frequency between the two dimension fields corresponding to the two nodes of the connecting edge.

[0051] For example, taking the target table user_data from the historical query above as an example, based on the three dimension fields user_id, user_name, and partition_time determined above, we can obtain the following information such as nodes, edges, and initial edge weights, thus forming the dimension co-occurrence graph of the target table user_data:

[0052] [(user_name,user_id,1),

[0053] (partition_time,user_id,1),

[0054] (partition_time,user_name,1)]

[0055] In this context, (user_name,user_id,1) indicates that user_name and user_id are two nodes, and there is an edge between them. Since the dimension fields user_name and user_id both appear once in the above SQL statement, the initial edge weight is 1. (partition_time,user_id,1) and (partition_time,user_name,1) express similar meanings.

[0056] After analyzing all historical query statements, the weights of the initial edges on the same edge are accumulated. That is, the weights of the initial edges are determined based on the query co-occurrence frequency between the two dimension fields corresponding to the two nodes connected by each edge. This results in an undirected weighted dimension co-occurrence graph G containing all dimension fields in all historical query statements.

[0057] In step S130, the first core node in the dimensional co-occurrence graph is determined based on the degree of each node in the dimensional co-occurrence graph.

[0058] In this embodiment of the disclosure, the degree of each node in the dimensional co-occurrence graph refers to the number of edges connected to each node. For example, assuming that user_id is connected to partition_time and user_name respectively, the degree of the node corresponding to the dimension field user_id is 2.

[0059] In this embodiment of the disclosure, the node with the highest degree in the dimensional co-occurrence graph G can be determined as the first core node of the dimensional co-occurrence graph G.

[0060] In step S140, the first connected subgraph after the first core node in the deleting dimension co-occurrence graph is obtained.

[0061] In this embodiment, after obtaining the first core node with the highest degree in the dimensional co-occurrence graph G, the first core node can be deleted from the dimensional co-occurrence graph, correspondingly deleting the edges connecting the first core node to all other nodes. Then, it is determined whether any nodes still exist in the dimensional co-occurrence graph after deleting the first core node. If nodes exist, it is further determined whether any two remaining nodes are still connected. If at least two of the remaining nodes are not connected, it indicates that there are two or more first connected subgraphs in the dimensional co-occurrence graph after deleting the first core node. Any two nodes in each first connected subgraph are connected.

[0062] In an undirected graph, nodes i and j are said to be connected if there is a path connecting them, where i and j are both positive integers greater than or equal to 1. A graph is considered connected if any two nodes are connected, and otherwise disconnected.

[0063] In step S150, the split tree of the dimension co-occurrence graph is obtained based on the dimension co-occurrence graph, its first core node, and its first connected subgraph.

[0064] In an exemplary embodiment, obtaining a split tree of the dimensional co-occurrence graph based on the dimensional co-occurrence graph, its first core node, and its first connected subgraph may include: combining nodes in the dimensional co-occurrence graph to form the root node of the split tree; determining the second core node in the first connected subgraph based on the degree of each node in the first connected subgraph; if deleting the second core node in the first connected subgraph leaves no nodes in the first connected subgraph, then combining the first core node and the second core node as the initial leaf node of the split tree.

[0065] In an exemplary embodiment, obtaining a split tree of the dimensional co-occurrence graph based on its first core node and first connected subgraph may further include: if, after deleting the second core node in the first connected subgraph, nodes exist in the first connected subgraph, and the nodes after deleting the second core node are connected, then a second connected subgraph after deleting the second core node is obtained; if the current splitting level of the split tree is less than or equal to a splitting level threshold, then a third core node in the second connected subgraph is determined based on the degree of each node in the second connected subgraph; if, after deleting the third core node in the second connected subgraph, no nodes exist in the second connected subgraph, then the first core node, the second core node, and the third core node are combined as the initial leaf nodes of the split tree. The specific method for splitting the dimensional co-occurrence graph to obtain a split tree can be referred to below. Figures 2 to 7 .

[0066] In step S160, the aggregation group of the dimension field is determined based on the target leaf node of the split tree.

[0067] In an exemplary embodiment, determining the aggregation group of dimension fields based on the target leaf nodes of the split tree may include: obtaining the splitting reward of each branch node in the split tree; performing a pruning operation on the split tree based on the size of the splitting reward of each branch node and the leaf node threshold; using the primary leaf nodes retained after the split tree pruning operation as target leaf nodes, and the number of target leaf nodes is less than or equal to the leaf node threshold; and using the target leaf nodes as the aggregation group of dimension fields.

[0068] In this embodiment of the disclosure, the threshold value of the leaf node can be set according to actual needs, and this disclosure does not limit it.

[0069] In an exemplary embodiment, each branch node in the split tree may include a first branch node, and each first branch node may include each first child node of a first parent node. Obtaining the splitting benefit of each branch node in the split tree may include: obtaining the dimensional combination size of each dimension field in the first parent node; obtaining the query co-occurrence frequency among each dimension field in the first parent node; obtaining the query co-occurrence frequency among each dimension field in the root node; obtaining the query construction cost of the first parent node based on the dimensional combination size of each dimension field in the first parent node, the query co-occurrence frequency among each dimension field in the first parent node, and the query co-occurrence frequency among each dimension field in the root node; obtaining the dimensional combination size of each dimension field in each first child node; obtaining the query co-occurrence frequency among each dimension field in each first child node; obtaining the query construction cost of each first child node based on the dimensional combination size of each dimension field in each first child node, the query co-occurrence frequency among each dimension field in each first child node, and the query co-occurrence frequency among each dimension field in the root node; and obtaining the splitting benefit of the first branch node based on the query construction cost of the first parent node and the query construction cost of each first child node. For details on calculating splitting rewards and pruning the split tree based on those rewards, please refer to [link to relevant documentation]. Figure 7 Example.

[0070] In step S170, a data cube oriented towards the target table is constructed based on the aggregation group of the dimension fields.

[0071] In this embodiment, the Cube, also known as a multidimensional cube or data cube, is a multidimensional space constructed based on dimensions. It contains the basic data to be analyzed, and data aggregation operations are performed on the Cube. Specifically, for each combination of dimensions, the metrics are aggregated, and the result is saved as a materialized view called a Cuboid. All Cuboids of dimension combinations are considered as a whole and referred to as the Cube.

[0072] The data processing method provided in this disclosure, on the one hand, analyzes historical query statements of a target table, constructs a dimension co-occurrence graph based on the dimension fields in the historical query statements and the co-occurrence frequency between different dimension fields, and determines the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph. Then, it obtains the first connected subgraph after deleting the first core node in the dimension co-occurrence graph. Based on the dimension co-occurrence graph, its first core node, and the first connected subgraph, the dimension co-occurrence graph is split to obtain a split tree of the dimension co-occurrence graph. Thus, the aggregation group of dimension fields in the aforementioned historical query statements can be determined based on the target leaf node of the split tree. When the aggregation group of dimension fields is used to construct a data cube (Cube) for the target table, it realizes the splitting of a large Cube into aggregation groups. Each aggregation group can be used to construct a smaller Cube. That is, by splitting the dimension co-occurrence graph, the data scale is compressed, the computing resources consumed in the Cube construction process are reduced, and the Cube construction time is shortened, achieving a balance between construction resources and construction time. Meanwhile, since the dimension co-occurrence graph in this embodiment is constructed based on the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, splitting this dimension co-occurrence graph allows the construction task of a large Cube to be divided into multiple smaller Cube construction tasks. This ensures that each smaller Cube can meet the query requirements of actual business operations without increasing the amount of redundant calculations. On the other hand, by analyzing historical query statements, the Cube construction is automated, simplifying the difficulty for users to design Cubes and optimizing Cube design.

[0073] Figure 2 A schematic diagram of a dimensional co-occurrence graph according to an embodiment of the present disclosure is shown.

[0074] like Figure 2As shown, assume the historical query statement includes the dimension fields city_name (city name), city_id (city identifier), user_name, user_id, partition_time, Y (year), M (month), and D (day), and assume that partition_time is connected to city_name, city_id, user_name, user_id, Y, M, and D by an edge, and the initial edge weights of each edge are 56, 106, 70, and 1, respectively. 00, 300, 60, and 80; the initial weight of the edge connecting the nodes user_name and user_id is 61, the initial weight of the edge connecting the nodes city_name and city_id is 200, the initial weight of the edge connecting the nodes city_id and Y is 77, the initial weight of the edge connecting the nodes Y and M is 53, the initial weight of the edge connecting the nodes D and M is 96, the initial weight of the edge connecting the nodes city_id and Y is 89, and the initial weight of the edge connecting the nodes Y and D is 21.

[0075] In this embodiment of the disclosure, after obtaining the dimensional co-occurrence graph G, the following graph decomposition algorithm can be used to partition the dimensional co-occurrence graph to obtain a split tree:

[0076] The first step is to set the core node list cores and the child node list children, and initialize the core node list to be empty, that is, cores = []. The child node list is initialized to include all dimension fields in the co-occurrence graph G of this dimension.

[0077] The second step is to calculate the degree of all nodes in the dimensional co-occurrence graph G, and sort them in reverse order according to the degree, and determine the node with the largest degree as the first core node.

[0078] The third step is to remove the first core node with the highest degree from the co-occurrence graph G and add the first core node to the core node list cores.

[0079] The fourth step is to determine whether any nodes remain in the co-occurrence graph G after deleting the first core node with the highest degree. If no nodes remain, the co-occurrence graph is split. Otherwise, if nodes remain, determine whether any two remaining nodes are connected. If they are connected, return to the second step and perform similar processing on all remaining nodes in the co-occurrence graph G after deleting the first core node. This involves calculating the degree of all remaining nodes in G and sorting them in reverse order of degree, identifying the node with the highest degree as another first core node. Then, delete this other first core node with the highest degree from the co-occurrence graph G and set it as the next first core node. A first core node is added to the core node list `cores`. Then, it is determined whether there are any nodes left in the dimensional co-occurrence graph G after deleting the other first core node with the highest degree. If there are no nodes left, the dimensional co-occurrence graph is split. Otherwise, if there are nodes left, it is determined whether any two remaining nodes after deleting the other first core node are connected. If any two remaining nodes after deleting the other first core node are connected, the process returns to step two above, and steps two through four are performed on all remaining nodes in the dimensional co-occurrence graph G after deleting the other first core node, until there are no nodes left in the dimensional co-occurrence graph after deleting a certain first core node, or at least two of the remaining nodes in the dimensional co-occurrence graph after deleting a certain first core node are not connected.

[0080] Fifth step: If, after deleting the first core node, there are at least two nodes among the remaining nodes that are not connected, then add all the first connected subgraphs G' in the co-occurrence graph of the dimension after deleting the first core node to the children list.

[0081] Next, each first connected subgraph G' in the child node list children is split. The splitting method can be referenced from the dimensional co-occurrence graph G mentioned above, that is, it is split in a tree-like manner, and finally a split tree can be obtained.

[0082] Take the splitting of any first connected subgraph G' as an example:

[0083] 1) Set a core node list cores and a child node list children, and initialize the core node list cores to include the first core node in the above dimension co-occurrence graph, and initialize the child node list to include all dimension fields in the first connected subgraph G'.

[0084] 2) Calculate the degree of all nodes in the first connected subgraph G', and sort them in reverse order of degree. The node with the largest degree is determined as the second core node.

[0085] 3) Delete the second core node with the highest degree from the first connected subgraph G', and add the second core node to the core node list cores.

[0086] 4) Determine if there are any nodes left in the first connected subgraph G' after deleting the second core node with the highest degree. If there are no nodes left, complete the splitting of the first connected subgraph G'. Otherwise, if there are nodes left, determine if any two remaining nodes after deleting the second core node are connected. If any two remaining nodes after deleting the second core node are connected, return to step 2) above and perform similar processing on all remaining nodes in the first connected subgraph G' after deleting the second core node, i.e., continue to calculate the degree of all remaining nodes in the first connected subgraph G' after deleting the second core node, and sort them in reverse order of degree, determining the node with the highest degree as another second core node; then delete the other second core node with the highest degree from the first connected subgraph G' after deleting the second core node, and set the other second core node as the second core node. Two core nodes are added to the core node list `cores`. Then, it is determined whether there are any nodes left in the first connected subgraph G' after deleting the other second core node with the highest degree. If there are no nodes left, the splitting of the first connected subgraph G' is completed. Otherwise, if there are nodes left, it is determined whether any two remaining nodes after deleting the other second core node are connected. If any two remaining nodes after deleting the other second core node are connected, then step 2) above is returned, and all remaining nodes in the first connected subgraph G' after deleting the other second core node are processed in a similar manner to steps 2) to 4) above, ... until after deleting a certain second core node, there are no nodes left in the first connected subgraph G', or at least two of the remaining nodes in the first connected subgraph G' after deleting a certain second core node are not connected.

[0087] 5) If, after deleting the second core node, there are at least two nodes among the remaining nodes that are not connected, then all the second connected subgraphs in the first connected subgraph G' after deleting the second core node are added to the children list.

[0088] Next, split each second connected subgraph in the child node list children, and the splitting method can refer to the splitting of the dimensional co-occurrence graph G and the first connected subgraph mentioned above.

[0089] Take the splitting of any second connected subgraph as an example:

[0090] (1) Set the core node list cores and the child node list children, and initialize the core node list cores to include the first core node in the above dimension co-occurrence graph and the second core node corresponding to the first connected subgraph, and initialize the child node list to include all dimension fields in the second connected subgraph.

[0091] (2) Calculate the degree of all nodes in the second connected subgraph and sort them in reverse order according to the degree. The node with the largest degree is determined as the third core node.

[0092] (3) Remove the third core node with the highest degree from the second connected subgraph and put the third core node into the core node list cores.

[0093] (4) Determine whether there are any nodes in the second connected subgraph after deleting the third core node with the highest degree. If there are no nodes, the split of the second connected subgraph is completed. Otherwise, if there are nodes, determine whether any two remaining nodes after deleting the third core node are connected. If any two remaining nodes after deleting the third core node are connected, return to step (2) above and process all remaining nodes in the second connected subgraph after deleting the third core node in a similar manner to steps (2) to (4) above. That is, continue to calculate the degree of all remaining nodes in the second connected subgraph after deleting the third core node, and sort them in reverse order according to the degree. The node with the highest degree is determined as another third core node. Then, delete the other third core node with the highest degree from the second connected subgraph after deleting the third core node, and set the other third core node as the third core node. The third core node is placed in the core node list cores. Then, it is determined whether there are any nodes in the second connected subgraph after deleting the other third core node with the highest degree. If there are no nodes, the split of the second connected subgraph is completed. Otherwise, if there are nodes, it is determined whether any two remaining nodes after deleting the other third core node are connected. If any two remaining nodes after deleting the other third core node are connected, the process is returned to step (2) above. All remaining nodes in the second connected subgraph after deleting the other third core node are processed in a similar manner to steps (2) to (4) above, until after deleting a certain third core node, there are no nodes in the second connected subgraph, or at least two nodes in the remaining nodes in the second connected subgraph after deleting a certain third core node are not connected.

[0094] (5) If at least two nodes among the remaining nodes after deleting the third core node are not connected, then all the third connected subgraphs in the second connected subgraph after deleting the third core node are added to the children list.

[0095] Then, split each third connected subgraph in the children list...

[0096] During the above splitting process, it can be determined whether the current splitting level of the splitting tree is greater than the splitting level threshold. If the current splitting level is greater than the splitting level threshold, the splitting stops. Conversely, if the current splitting level is less than or equal to the splitting level threshold, and the children of the currently split graph (e.g., the aforementioned dimension co-occurrence graph or the first connected subgraph, second connected subgraph, and third connected subgraph) are not empty, then the splitting continues for all connected subgraphs in the children of the currently split graph. For example, the first connected subgraphs of the dimension co-occurrence graph are split, the second connected subgraphs of the first connected subgraph are split, and so on, to obtain the splitting results of each connected subgraph.

[0097] In this embodiment of the disclosure, the current splitting level refers to the current depth of the split tree. The splitting level threshold can be determined based on specific business requirements, such as considering the number of dimension fields and the required number of cubes. Here, it is assumed that the classification level threshold is set to 6.

[0098] The following is combined Figures 3 to 6 The above splitting process will be illustrated with an example.

[0099] According to the above Figure 2 It can be seen that partition_time is Figure 2 The node with the highest degree in the co-occurrence graph shown, which is connected by 7 edges, is deleted. Figure 2 After the first core node partition_time in the process, you can obtain the following: Figure 3 The two first connected subgraphs shown are, i.e. Figure 2 The dimensional co-occurrence graph shown is split into: a first connected subgraph 301 consisting of two nodes, user_name and user_id, and one edge connecting them; and a first connected subgraph 302 consisting of four nodes, city_name, city_id, Y, M, and D, and six edges connecting these four nodes.

[0100] Up Figure 3 In the connected subgraph 302, we can further determine that the node with the highest degree is Y. After deleting the second core node Y in the first connected subgraph 302, we obtain the following... Figure 4The two second connected subgraphs shown are: second connected subgraph 401 consisting of two nodes, city_name and city_id, and one edge connecting them; and second connected subgraph 402 consisting of two nodes, M and D, and one edge connecting them.

[0101] like Figure 5 As shown above, Figures 2 to 4 For example, the splitting process may include the following steps.

[0102] In step S11, the core node list is initialized to be empty, i.e., cores = [], and the child node list children = [partition_time, city_id, city_name, user_id, user_name, Y, M, D].

[0103] In step S12, the first core node is added to the core node list, then cores = [partition_time].

[0104] At the same time, add the two first connected subgraphs to the child node list, then children = [user_id, user_name], [city_id, city_name, Y, M, D].

[0105] In step S13, the second core node of the first connected subgraph 301 is determined to be user_id. The second core node user_id is added to the core node list, then cores = [partition_time, user_id], and the child node list children = [user_name].

[0106] In step S14, another second core node of the first connected subgraph 301 is determined to be user_name. The other second core node user_name is added to the core node list, then cores = [partition_time, user_id, user_name], and the child node list children = [].

[0107] In step S15, the second core node of the first connected subgraph 302 is determined to be Y. The second core node Y is added to the core node list, then cores = [partition_time,Y]. Two second connected subgraphs 401 and 402 of the first connected subgraph 302 are obtained. These two second connected subgraphs 402 are added to the child node list, then children = ([city_id,tity_name],[M,D]).

[0108] In step S16, the third core node of the second connected subgraph 401 is determined to be city_id. The third core node city_id is added to the core node list, then cores = [partition_time, Y, city_id], and the child node list children = [city_name].

[0109] In step S17, another third core node of the second connected subgraph 401 is determined to be city_name. This other third core node city_name is added to the core node list, then cores = [partition_time, Y, city_id, city_name], and the child node list children = [].

[0110] In step S18, the third core node of the second connected subgraph 402 is determined to be M. The third core node M is added to the core node list, then cores = [partition_time, Y, M], and the child node list children = [D].

[0111] In step S19, another third core node of the second connected subgraph 402 is determined to be D. This other third core node D is added to the core node list, then cores = [partition_time, Y, M, D], and the child node list children = [].

[0112] pass Figure 5 The splitting process shown can yield the following results: Figure 6 The split tree shown.

[0113] The node list `nodes` of the root node 600 of the split tree includes `[partition_time, city_id, city_name, user_id, user_name, Y, M, D]`. The node lists `nodes` of the two branch nodes included in the first split level 601 are `[partition_time, user_id, user_name]` and `[partition_time, Y, city_id, city_name, M, D]`, respectively. That is, `[partition_time, user_id, user_name]` and `[partition_time, Y, city_id, city_name, M, D]` are both child nodes of the root node 600.

[0114] The branch node [partition_time,Y, city_id, city_name,M,D] in the first split layer 601 serves as the parent node of the second split layer 602. The node lists of the two branch nodes included in the second split layer 602 are [partition_time,user_id,user_name] and [partition_time,Y, city_id, city_name,M,D], respectively. That is, [partition_time,user_id,user_name] and [partition_time,Y, city_id, city_name,M,D] are child nodes of the parent node [partition_time,Y, city_id, city_name,M,D].

[0115] In related technologies, there is a problem of data bloat: these technologies pre-calculate dimensional combinations, and the formula for calculating dimensional combinations is 2^N (where N is the number of dimensions, and N is a positive integer greater than or equal to 1). For example, for a model containing (A, B, C, D), where D is the indicator field and A, B, and C are the dimension fields, then all its corresponding dimensional combinations are as follows: [(),(A),(B),(C),(A,B),(A,C),(B,C),(A,B,C)]. If the data model has a particularly large number of dimensions, such as 60 or more, then dimensional explosion is unacceptable. Although related technologies provide dimensional pruning methods to reduce the number of dimensional combinations, for complex businesses such as advertising, the number of combinations that cannot be pruned in a single data model construction task can still reach more than 1000. With a particularly large number of dimensional combinations, the resulting data size will inevitably expand rapidly. Therefore, how to prune more effectively is an important issue in Cube design.

[0116] In related technologies, to reduce the complexity of the Cube and the number of Cuboids, some Cube pruning algorithms have been provided. However, these algorithms all have a problem: they calculate the yield based on the assumption that all query requests are evenly distributed across every dimension combination, which is impossible in reality. In actual business scenarios, user queries are usually concentrated on certain dimension combinations, while other dimension combinations are rarely or never queried. If some dimension combinations are pre-computed but not found, or if some dimension combinations are frequently queried but not pre-computed, these discrepancies affect the Cube's resource utilization and query performance.

[0117] In addition, some pruning methods in related technologies are not suitable for Cube pruning with too many dimensions, or other pruning methods have stability issues, that is, the combination of dimensions calculated each time is completely different.

[0118] Obtain as Figure 6 After showing the split tree, its leaf nodes are taken as primary leaf nodes. Figure 6 Some branch nodes may not need to be subdivided, but they are split. Therefore, in this embodiment of the disclosure, the splitting benefit after each split is calculated and the tree is pruned to obtain the target leaf node of the split tree.

[0119] Figure 7 The illustration shows a schematic diagram of a pruning operation on a split tree according to an embodiment of the present disclosure.

[0120] like Figure 7 As shown, assume that the first parent node 701 in the split tree includes three dimension fields: a, b, and c. The first branch node includes two first child nodes 702 of the first parent node 701. That is, one first child node 702 includes two dimension fields: a and b, and the other first child node 702 includes two dimension fields: a and c. It can be understood that any parent node in the split tree can be chosen as the first parent node, and each parent node is a child node relative to the previous split level.

[0121] Assume that the two dimension fields a and c in the first child node 702 are the second parent node 703, which further includes two second child nodes 704, namely the second child node 704 including the dimension field a and the second child node 704 including the dimension field c.

[0122] In this embodiment of the disclosure, the split income Split_income1 of the first branch node can be calculated using the following formula (1):

[0123]

[0124] Alternatively, the split income Split_income1 of the first branch node can be calculated according to the following formula (2):

[0125]

[0126] In formulas (1) and (2) above, D(a,b,c) represents the size of the dimension combination formed by the three dimension fields (a,b,c) in the first parent node. This can be calculated by counting the number of records formed by these three dimension fields in the original data of the target table, for example, by using unique values. Q(a,b,c) represents the query co-occurrence frequency among the three dimension fields (a,b,c) in the first parent node. Q(all) represents the query co-occurrence frequency among all dimension fields in the root node. D(a,b,c)*(0.5*Q(a,b,c) / Q(all)+0.5) considers the query cost and construction cost of the first parent node and is called the query construction cost of the first parent node. b) represents the size of the dimension combination formed by the two dimension fields (a,b) in the first child node; Q(a,b) represents the query co-occurrence frequency between the two dimension fields (a,b) in the first child node; (D(a,b)*(0.5*Q(a,b) / Q(all)+0.5) is called the query construction cost of the first child node; D(a,c) represents the size of the dimension combination formed by the two dimension fields (a,c) in the first child node; Q(a,c) represents the query co-occurrence frequency between the two dimension fields (a,c) in the first child node; D(a,c)*(0.5*Q(a,c) / Q(all)+0.5) is called the query construction cost of the first child node.

[0127] In this embodiment of the disclosure, the split income Split_income2 of the second branch node can be calculated using the following formula (3):

[0128]

[0129] Alternatively, the split income Split_income2 of the second branch node can be calculated according to the following formula (4):

[0130]

[0131] In formulas (3) and (4) above, D(a,c) represents the size of the dimension combination formed by the combination of the two dimension fields (a,c) in the second parent node; Q(a,c) represents the query co-occurrence frequency between the two dimension fields (a,c) in the second parent node; D(a,c)*(0.5*Q(a,c) / Q(all)+0.5) is called the query construction cost of the second parent node; D(a) represents the size of the dimension combination of the dimension field (a) in the second child node; Q(a) represents the query co-occurrence frequency between the two dimension fields (a,c) in the second parent node; and Q(a) represents the query co-occurrence frequency between the two dimension fields (a,c) in the second parent node. The query co-occurrence frequency among the dimension fields (a) in the second child node is represented by D(a)*(0.5*Q(a) / Q(all)+0.5), which is called the query construction cost of the second child node; D(c) represents the size of the dimension combination of the dimension field (c) in the second child node; Q(c) represents the query co-occurrence frequency among the dimension fields (c) in the second child node; D(c)*(0.5*Q(c) / Q(all)+0.5), which is called the query construction cost of the second child node.

[0132] However, the calculation method for the splitting gains of each branch node in this embodiment is not limited to the example given by the above formula.

[0133] Using the above calculation method, the splitting reward of all branch nodes of the split tree can be calculated. Then, the splits with low rewards are pruned according to the size of the splitting reward. For example, assuming that the splitting reward of the first branch node is less than that of the second branch node, the first child nodes (a,b) and (a,c) are deleted until the number of primary leaf nodes of the split tree meets the leaf node threshold. Each primary leaf node is retained as a target leaf node, and each target leaf node is represented as an aggregation group of a Cube. All dimension fields on each target leaf node can be used as dimensions of a single Cube. Thus, through the above splitting process, multiple Cubes of the target table can be constructed, which achieves the purpose of splitting a large Cube into multiple smaller Cubes.

[0134] In this embodiment, the leaf node threshold can be set according to actual needs, and this disclosure does not limit it. Furthermore, the specific value of the low-yield parameter is not limited; it can be determined based on the set leaf node threshold. For example, assuming four aggregation groups need to be set, and the leaf node threshold is set to 4, then during the pruning process described above, the four initial leaf nodes with the highest splitting yield of the branch nodes can be retained as target leaf nodes.

[0135] It should be noted that in the above pruning process, it is not required that the final number of target leaf nodes be equal to the leaf node threshold. It is sufficient that the number of target leaf nodes is less than or equal to the leaf node threshold.

[0136] The data processing method provided in this disclosure constructs a dimensional co-occurrence graph by analyzing historical query statements of the target table, and obtains aggregation groups of the split tree by splitting the dimensional co-occurrence graph. Multiple smaller cubes are then constructed based on these aggregation groups, thereby improving computational resources and construction time during the cube construction process and reducing the number of Cuboids. Furthermore, the aforementioned pruning operation further reduces the number of aggregation groups, thereby further reducing computational resources consumed and construction time, while ensuring that the constructed cube meets actual business requirements. Simultaneously, it achieves automatic determination of aggregation groups, eliminating the need for manual user design, thus reducing the design complexity of the cube and lessening the design difficulty when there are a large number of dimensions.

[0137] Figure 8 A flowchart illustrating another embodiment of a data processing method according to this disclosure is shown schematically. Figure 8 As shown, compared with the other embodiments described above, Figure 8 The method provided in the embodiments further includes the following steps.

[0138] In step S810, the target edge weights of each edge in the dimensional co-occurrence graph are obtained based on the initial edge weights of each edge.

[0139] In an exemplary embodiment, the dimensional co-occurrence graph may include a first edge and a first node and a second node of the first edge. Obtaining the target edge weights of each edge in the dimensional co-occurrence graph based on the initial edge weights of each edge may include: obtaining the first query frequency of the dimension field corresponding to the first node in historical query statements; obtaining the second query frequency of the dimension field corresponding to the second node in historical query statements; determining the target query frequency based on the first and second query frequencies; and obtaining the target edge weight of the first edge based on the initial edge weights of the first edge and the target query frequency.

[0140] In this embodiment of the disclosure, by analyzing the query co-occurrence frequency of dimension fields in historical query statements, some dimension fields with high correlation can be combined together to form a joint dimension. Specifically, the correlation analysis between dimension fields can be performed by the following steps: recalculating the edge weights of the dimension co-occurrence graph G to obtain the target edge weights.

[0141] For example, for the first edge (a, b, w) in a dimensional co-occurrence graph, where a and b are the first and second nodes of the first edge, respectively, and w is the initial edge weight of the first edge, the target edge weight w′ of the first edge can be calculated using the following formula:

[0142] w′=w / max(count(a),count(b)) (5)

[0143] In the above formula, count(a) represents the first query frequency of the first node a in the historical query statement, that is, the total number of times it appears; count(b) represents the second query frequency of the second node b in the historical query statement, that is, the total number of times it appears.

[0144] In step S820, edges with target edge weights lower than the weight threshold are removed to obtain the subgraphs of the dimension co-occurrence graph.

[0145] Then, edges whose target edge weights are less than the weight threshold are removed from the dimension co-occurrence graph. After removing the target edge weights that are less than the weight threshold, the dimension co-occurrence graph may be split into multiple subgraphs.

[0146] In this embodiment of the disclosure, the weight threshold can be set according to experience, and the value range is (0~1), for example, it is assumed to be set to 0.6 here.

[0147] In step S830, the maximum clique in each subgraph is obtained, and each associated dimension group is formed based on the dimension fields in each maximum clique.

[0148] Then, the maximum clique algorithm is used to find the maximum clique in each subgraph after removing edges whose target edge weights are lower than the weight threshold. Each maximum clique is treated as a group of associated dimensions.

[0149] Given subgraphs, if a subgraph contains an edge between any two nodes, then the subgraph is called a complete subgraph of the subgraph, the complete subgraph is called a clique, and the maximum clique is the maximum complete subgraph. This disclosure does not limit the choice of maximum clique algorithm.

[0150] In step S840, the correlation and merging inflation rate between each associated dimension group are obtained.

[0151] In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Obtaining the relevance between the associated dimension groups may include: obtaining the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; obtaining the product of the dimension combination sizes of the first and second associated dimension groups; merging the first and second associated dimension groups to obtain a merged associated dimension group; obtaining the dimension combination size of the merged associated dimension group; and obtaining the relevance between the first and second associated dimension groups based on the dimension combination size of the merged associated dimension group and the product of the dimension combination sizes of the first and second associated dimension groups.

[0152] In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Obtaining the merge expansion rate between the associated dimension groups may include: obtaining the combined dimension size of the first associated dimension group and the combined dimension size of the second associated dimension group; obtaining the sum of the combined dimension sizes of the first and second associated dimension groups; merging the first and second associated dimension groups to obtain a merged associated dimension group; obtaining the combined dimension size of the merged associated dimension group; and obtaining the merge expansion rate between the first and second associated dimension groups based on the combined dimension size of the merged associated dimension group and the sum of the combined dimension sizes of the first and second associated dimension groups.

[0153] In step S850, the associated dimension groups are merged according to the correlation and merging expansion rate between them to obtain the joint dimension of the dimension co-occurrence graph. The number of joint dimensions is less than or equal to the joint dimension threshold.

[0154] In this embodiment of the disclosure, some dimension fields are combined together as a joint dimension, and the joint dimension is pre-calculated, thereby reducing the number of dimension combinations, reducing Cube complexity, and reducing the number of Cuboids.

[0155] For example, if dimension fields a, b, and c are defined as a joint dimension, only Cuboid ab C will be constructed, while Cuboids ab, bc, a, etc. will not be generated.

[0156] In an exemplary embodiment, merging the associated dimension groups based on their relevance and merging expansion rate to obtain the joint dimension of the dimension co-occurrence graph may include: determining the associated dimension group pairs that meet the merging conditions in each associated dimension group based on their relevance and merging expansion rate; obtaining the target associated dimension group pair with the highest relevance among the associated dimension group pairs; merging the associated dimension groups in the target associated dimension group pair to obtain the target merged associated dimension group, and deleting the associated dimension groups in the target associated dimension group pair; if the number of the target merged associated dimension group and the number of unmerged associated dimension groups are less than or equal to the joint dimension threshold, then the target merged associated dimension group and the unmerged associated dimension group are respectively used as the joint dimension of the dimension co-occurrence graph.

[0157] In an exemplary embodiment, determining the associated dimension group pairs that meet the merging conditions based on the relevance and merging expansion rate among the associated dimension groups may include: if the relevance between the associated dimension groups is greater than a first relevance threshold; or if the relevance between the associated dimension groups is greater than a second relevance threshold and the merging expansion rate is less than a first expansion rate threshold; or if the relevance between the associated dimension groups is greater than a third relevance threshold and the merging expansion rate is less than a second expansion rate threshold, then the corresponding associated dimension group is determined to be an associated dimension group pair that meets the merging conditions; wherein the first relevance threshold is greater than the second relevance threshold, the second relevance threshold is greater than the third relevance threshold, and the first expansion rate threshold is greater than the second expansion rate threshold.

[0158] Specifically, after obtaining each associated dimension group, the dimension fields are further aggregated. For example, suppose the following combinations of associated dimension groups are obtained through the above method, and suppose the goal is to compress the number of associated dimension groups in the combination to a given joint dimension threshold N (N is a positive integer greater than or equal to 1, which can be set according to actual needs; here, it is assumed to be 4):

[0159] Groups = [

[0160] [user_id,user_name],

[0161] [partition_time],

[0162] [city_id, city_name],

[0163] [year],

[0164] [month],

[0165] [day]

[0166] Assuming we obtain six related dimension groups: [user_id, user_name], [partition_time], [city_id, city_name], [year], [month], and [day], then we perform the following calculations:

[0167] 1.1: Traverse all related dimension groups in groups, and obtain the size of each related dimension group by counting the number of unique values ​​of each related dimension group in the source data of the target table.

[0168] 1.2: Calculate the correlation between all related dimension groups in the groups.

[0169] For example, assuming the first related dimension group is [user_id, user_name] and the second related dimension group is [partition_time], where [user_id, user_name, partition_time] represents the merged related dimension group obtained by combining the first and second related dimension groups, the relevance between the first and second related dimension groups can be calculated using the following formula: Rel

[0170]

[0171] In the above formula, D(user_id,user_name) represents the size of the dimension combination of the first associated dimension group; D(partition_time) represents the size of the dimension combination of the second associated dimension group; (D(user_id,user_name)*D(partition_time)) represents the product of the size of the dimension combination of the first associated dimension group and the size of the dimension combination of the second associated dimension group; D(user_id,user_name,partition_time) represents the size of the dimension combination of the merged associated dimension groups.

[0172] Calculate the merge inflation rate (Exp) relative to the combined dimension size of the merged related dimension groups and the combined dimension size of the unmerged first related dimension group and the second related dimension group. For example, the merge inflation rate Exp can be calculated using the following formula:

[0173]

[0174] In the above formula, (D(user_id,user_name)+D(partition_time)) represents the sum of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group.

[0175] 1.3: By calculating the relevance and merging inflation rate among the various related dimension groups, determine whether any two related dimension groups meet the merging criteria. For example, assume that if any one of the following conditions is met, the corresponding two related dimension groups are considered to meet the merging criteria:

[0176] 1) The relevance is greater than 0.85, that is, assuming the first relevance threshold is 0.85, but this disclosure is not limited to this. The first relevance threshold is a real number greater than 0 and less than 1.

[0177] 2) The correlation is greater than 0.75 and the combined inflation rate is less than 10. That is, assuming the second correlation threshold is 0.75 and the first inflation rate threshold is 10, but this disclosure is not limited to this. The second correlation threshold is a real number greater than 0 and less than 1, and the first inflation rate threshold is a real number greater than or equal to 1.

[0178] 3) The correlation is greater than 0.5 and the combined inflation rate is less than 5. That is, assuming the third correlation threshold is 0.5 and the second inflation rate threshold is 5, but this disclosure is not limited to this. The third correlation threshold is a real number greater than 0 and less than 1, and the second inflation rate threshold is a real number greater than or equal to 1.

[0179] 1.4: If two related dimension groups satisfy the merging criteria, these two groups are treated as a single related dimension pair. Then, the related dimension pairs are sorted in reverse order of relevance, and the pair with the highest relevance is identified as the target related dimension pair. The two related dimension groups in the target pair are merged, and the merged pair is added to the `groups` folder. The two original related dimension pairs from the original `groups` folder are then removed. If no two related dimension groups satisfy the merging criteria, the merging process is terminated.

[0180] 1.5: If the number of associated dimension groups in groups is less than or equal to the given joint dimension threshold, then terminate the merging; otherwise, repeat steps 1.1 to 1.5 above.

[0181] In an exemplary embodiment, constructing a data cube oriented towards the target table based on the aggregation group of dimension fields may include: obtaining the row key order of each field in the historical query statement, where the fields include dimension fields; and constructing a data cube oriented towards the target table based on the row key order of each field, the joint dimension of the dimension co-occurrence graph, and the aggregation group of the dimension fields.

[0182] In an exemplary embodiment, obtaining the row key order of each field in a historical query statement may include: obtaining weight parameters for different field types in the historical query statement; obtaining each field and its field type in the historical query statement; obtaining the field weight value of each field based on the weight parameters corresponding to the field type of each field and the query frequency of each field in the historical query statement; and determining the row key order of each field based on the field weight value of each field.

[0183] Specifically, the field weight values ​​of each field are calculated using the fields obtained from the analysis of the historical query statements mentioned above.

[0184] For example, suppose a weight parameter is set for the aggregation field (group) in the dimension field, for example, to 4; and a weight parameter is set for the filter field in the dimension field, for example, to 2. In other embodiments, a weight parameter can also be set for the indicator field, such as the sum mentioned above, for example, to 1. However, this disclosure is not limited to this, and the settings can be made according to actual needs. Considering the frequency of use of different field types in actual queries, the weight parameter of the more frequently used field type can be set to be larger. For example, the weight parameter of the aggregation field can be set to be greater than that of the filter field, and the weight parameter of the filter field can be set to be greater than that of the indicator field, etc., while the difference in weight parameters between different field types will not be large. Thus, the field weight values ​​of each field in the above historical query statement can be obtained as follows:

[0185] (user_data, user_id, 4),

[0186] (user_data, user_name, 4),

[0187] (user_data, partition_time, 2),

[0188] (user_data, value, 1)

[0189] For each field obtained from all historical query statements, the field weight value is obtained by multiplying the weight parameter of the field type to which the field belongs by the query frequency of the field in all historical query statements. For example, for the field user_id, which is an aggregate field, if it appears 50 times in all historical query statements, the field weight value of the field user_id is 200.

[0190] After obtaining the field weight values ​​of each field in all historical query statements, sort the fields in reverse order according to the size of the field weight values. The order of the fields obtained by the reverse sort can be directly used for the design of RowKey order.

[0191] The data processing method provided in this disclosure analyzes historical query statements of a target table, extracts the relevant fields, and labels the field types of each field. It then obtains the dimension fields and the query co-occurrence frequency between different dimension fields. Dimension fields are used as nodes, and two nodes corresponding to two dimension fields with query co-occurrence frequencies are connected to form edges. The query co-occurrence frequency between the two dimension fields corresponding to the two nodes connected by each edge is used as the initial edge weight of the corresponding edge. This automatically constructs a dimension co-occurrence graph of the target table. On one hand, by splitting this dimension co-occurrence graph and performing pruning operations by calculating the splitting benefit of each branch node, target leaf nodes less than or equal to a set leaf node threshold can be obtained in the dimension co-occurrence graph. Then, the dimension fields contained in each target leaf node are combined into an aggregation group. A single aggregation group can be used to construct a relatively small Cube, thereby splitting a large Cube construction task into multiple smaller ones. The small cube construction task saves computing resources, reduces construction time, and makes the constructed cube more consistent with actual query needs, satisfying the distribution of query requests in actual queries. On the other hand, the target edge weight of each edge can be recalculated based on the query frequency of the two dimension fields corresponding to the two nodes connected by each edge. After removing edges with target edge weights below the weight threshold, multiple subgraphs of the co-occurrence graph of that dimension can be obtained. Then, the maximum clique of each subgraph is obtained, and all dimension fields in each maximum clique are combined to form a related dimension group. Then, based on the relevance and merging expansion rate between different related dimension groups, the two related dimension groups with the highest relevance that meet the merging conditions are merged to form a joint dimension less than or equal to the joint dimension threshold. This achieves the further combination of the most relevant related dimension groups into a joint dimension, reducing the size of the constructed cube, reducing the storage resources occupied, and ensuring that the final constructed cube meets the relevance expansion rate requirements. In addition, weight parameters of different field types can be obtained. The field weight value of each field is determined based on the query frequency of each field and the weight parameter of its field type. The RowKey order is designed based on the magnitude of the field weight values, ensuring that the designed RowKey order meets the actual query requirements. After the above process, we have the core parameters for designing a Cube, including the division of aggregation groups, the design of joint dimensions, and the design of RowKey order, which allows us to automatically construct a Cube.

[0192] Figure 9 A schematic flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown. Figure 9 As shown, the method provided in this disclosure embodiment may include the following steps.

[0193] In step S910, historical query statements of the target table are obtained, and the historical query statements are analyzed to obtain the query co-occurrence frequency of dimension fields and different dimension fields.

[0194] like Figure 9 As shown, assuming the system input is a record of historical SQL statements from users on the same data source (the same table or the same combination of fact table and dimension table) on Hive, by analyzing the historical query statements, the query co-occurrence frequency of dimension fields and different dimension fields can be obtained. The weight parameters of fields and different field types can also be obtained, as well as the query frequency of each field in the historical query statements. Dimension fields are included in these fields.

[0195] In step S920, a dimension co-occurrence graph is constructed based on the dimension fields and the query co-occurrence frequency between different dimension fields.

[0196] Then, each dimension field is used as a node, and edges are constructed based on the query co-occurrence frequency, and the initial edge weights of the edges are determined to build a weighted undirected dimension co-occurrence graph.

[0197] In step S930, the dimensional co-occurrence graph is decomposed to obtain a split tree.

[0198] The dimensional co-occurrence graph is decomposed using the method described in the above embodiments to obtain a split tree including the initial leaf nodes.

[0199] In step S940, the split tree is pruned to determine the aggregation group.

[0200] The obtained split tree is pruned using the method described in the above embodiments. Some primary leaf nodes are reduced based on the splitting gains to obtain target leaf nodes. All dimension fields in each target leaf node are combined into a single aggregation group.

[0201] In step S950, the joint dimension in the dimension co-occurrence graph is found.

[0202] By recalculating the initial edge weights of the edges in the dimension co-occurrence graph in the above embodiments, determining the target edge weights, splitting the dimension co-occurrence graph into multiple subgraphs based on the target edge weights, and determining one or more maximal cliques in each subgraph, and discovering joint dimensions with high correlation and meeting the expansion rate requirements based on these maximal cliques.

[0203] In step S960, the field weight values ​​of the fields are calculated to determine the row key order.

[0204] You can also calculate the field weight value of each field based on the weight parameter of the field type to which each field belongs and the query frequency of each field, and determine the row key order of each field based on the size of the field weight value.

[0205] In step S970, the data cube is designed and generated based on the aggregation group, joint dimension, and row key order.

[0206] Based on the aggregation group, joint dimension, and row key order obtained in the above steps, a complete Cube design scheme can be obtained.

[0207] The data processing method provided in this disclosure analyzes historical query statements of the target table, extracts all fields involved in the historical query statements, constructs a dimension co-occurrence graph using the query co-occurrence frequency between dimension fields, and then uses the maximum clique algorithm to find all maximum cliques in all subgraphs of the dimension co-occurrence graph. All nodes in each maximum clique are then combined to form a joint dimension, further improving the correlation between dimension fields. Alternatively, the query co-occurrence graph can be partitioned based on a graph partitioning algorithm for weighted undirected query co-occurrence graphs (the graph splitting process described above), resulting in a split tree. Pruning yields the final partitioning result, i.e., the target leaf node, which can be directly used for Cube design. This automates Cube design and reduces the design difficulty of building Cubes on large-scale dimensions. Furthermore, the graph partitioning method allows large Cubes to be split into aggregate groups or multiple smaller Cubes, with each smaller Cube's corresponding subtasks meeting requirements without increasing redundant computation. This partitioning reduces the computational resources consumed during Cube construction and shortens the construction time, achieving a balance between construction resources and construction time. In some complex business scenarios, there are usually very complex data models with many data dimensions. Using the solutions provided in this disclosure, a good Cube design can be given based on business needs, so that the designed Cube can not only meet the query requirements, but also ensure query efficiency and construction efficiency, while ensuring that the data expansion rate is within an appropriate range.

[0208] In the test, for a dataset with 61 dimensions and a daily data volume of approximately 300 million, without using the solution provided in this embodiment, manually designing the Cube and then using optimization tools provided by related technologies resulted in an average Cube construction time of approximately 120 minutes, with a daily pre-calculated data volume of approximately 600 GB. However, after optimization and splitting using the solution provided in this embodiment, the original Cube was split into 3 Cubes, reducing the average construction time to approximately 50 minutes. Furthermore, the overall storage space occupied by the resulting data was approximately 200 GB. Good results were also demonstrated in several other medium-sized Cube optimization tasks, as shown in Table 1 below.

[0209] Table 1

[0210]

[0211] The test results above demonstrate that this embodiment analyzes business logic using historical query statements, automatically generates Cube designs, simplifies user design, and optimizes Cube design. By using graph splitting, large Cubes can be divided, reducing Cube construction time and compressing data size.

[0212] The method provided in this disclosure can be applied to the optimization of OLAP data query and data platform products. It can reduce the difficulty of OLAP Cube design for users. Through analysis and optimization, the Cube can be automatically designed, which can reduce the time and resource consumption of pre-computation tasks. It can also be used in some cache design applications, and can also be used in scenarios where business data is split by reverse analysis through query. Figure 10 The Apache Kylin solution, which generates Cubes based on MapReduce pre-computation and provides low-latency queries, is used as an example for illustration.

[0213] Figure 10 A schematic diagram of a system architecture applying the data processing method provided in embodiments of this disclosure is shown. Let's first look at the offline construction section. From... Figure 10 As shown, the left side represents data source 1010. The default data source can be Apache Hive (or Kafka, RDBMS (Relational Database Management System), etc.), which stores the user data to be analyzed. According to the definition in metadata 1024, the Cube Build Engine 1025 extracts data from data source 1010 and builds the Cube. The building technology can be MapReduce. The built OLAP data cube 1030 is stored in the storage engine on the right, with Apache HBase as the default storage.

[0214] After offline construction is complete, users can send SQL queries for analysis from the query system above (e.g., third-party APP1040, SQL-Based tool1050). RESTful API (Representational State Transfer Application Programming Interface), JDBC (Java Database Connectivity) / ODBC (Open Database Connectivity) interfaces are provided for user calls. Regardless of the interface used, the SQL ultimately reaches the REST Server1021, and is then handed over to the Query Engine1022 for processing. The Query Engine1022 parses the SQL, generates a logical execution plan based on relational tables, and then translates it into a physical execution plan based on a Cube. It then queries the pre-computed Cube based on routing1023 and produces the results. The entire process does not access the original data source. If the user-submitted query is not predefined, an error will be returned.

[0215] Figure 10 In this embodiment, an abstraction layer is extracted from the three modules: data source, data cube building engine, and cube storage. These three modules can be arbitrarily extended and replaced. For example, Spark can be used instead of MapReduce as the data cube building engine, and Cassandra can be used instead of HBase for storing the data after cube computation, thus enabling good scalability and allowing users to deeply customize it according to their own business characteristics.

[0216] Apache Kylin uses pre-computation techniques to address the limitation of dataset size, enabling better support for queries on massive datasets. Also benefiting from pre-computation, Kylin's query speed is extremely fast because complex joins, aggregations, and other operations are completed during the Cube's construction process. Apache Kylin can also be horizontally scaled using cluster deployment.

[0217] The method provided in this disclosure can employ database technology from cloud technology.

[0218] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0219] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0220] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, with minimal redundancy, and independent of application programs.

[0221] A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile devices; or according to the query language used, such as SQL or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0222] Figure 11 A block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically. Figure 11 As shown, the data processing apparatus 1100 provided in this embodiment may include a co-occurrence frequency acquisition unit 1110, a dimension co-occurrence graph construction unit 1120, a core node determination unit 1130, a connected subgraph acquisition unit 1140, a graph split tree acquisition unit 1150, a field aggregation group determination unit 1160, and a data cube construction unit 1170.

[0223] In this embodiment, the co-occurrence frequency acquisition unit 1110 can be used to obtain the co-occurrence frequency of dimension fields and different dimension fields in the historical query statements of the target table. The dimension co-occurrence graph construction unit 1120 can be used to use dimension fields as nodes, determine the initial edge weights between different nodes based on the query co-occurrence frequency between different dimension fields, and form a dimension co-occurrence graph. The core node determination unit 1130 can be used to determine the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph. The connected subgraph acquisition unit 1140 can be used to obtain the first connected subgraph after deleting the first core node in the dimension co-occurrence graph. The graph split tree acquisition unit 1150 can be used to obtain the split tree of the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and the first connected subgraph. The field aggregation group determination unit 1160 can be used to determine the aggregation group of dimension fields based on the target leaf node of the split tree. The data cube construction unit 1170 can be used to construct a data cube oriented towards the target table based on the aggregation group of dimension fields.

[0224] The data processing apparatus provided in this embodiment analyzes historical query statements of a target table, constructs a dimension co-occurrence graph based on the dimension fields in the historical query statements and the frequency of co-occurrence between different dimension fields, determines the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph, obtains a first connected subgraph after deleting the first core node in the dimension co-occurrence graph, and splits the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and the first connected subgraph to obtain a split tree of the dimension co-occurrence graph. The aggregation group of dimension fields in the historical query statements can then be determined based on the target leaf node of the split tree. When the aggregation group of dimension fields is used to construct a data cube (Cube) for the target table, a large Cube is split into aggregation groups, and each aggregation group can be used to construct smaller Cubes. In other words, by splitting the dimension co-occurrence graph, the data scale is compressed, the computational resources consumed during Cube construction are reduced, and the Cube construction time is shortened, achieving a balance between construction resources and construction time. Meanwhile, since the dimension co-occurrence graph in this embodiment is constructed based on the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, splitting this dimension co-occurrence graph allows the construction task of a large Cube to be divided into multiple smaller Cube construction tasks. This ensures that each smaller Cube can meet the query requirements of actual business operations without increasing the amount of redundant calculations. On the other hand, by analyzing historical query statements, the Cube construction is automated, simplifying the difficulty for users to design Cubes and optimizing Cube design.

[0225] In an exemplary embodiment, the graph split tree obtaining unit 1150 may include: a root node forming unit, which can be used to combine nodes in the dimension co-occurrence graph to form the root node of the split tree; a second core node determining unit, which can be used to determine the second core node in the first connected subgraph based on the degree of each node in the first connected subgraph; and a first primary leaf node obtaining unit, which can be used to combine the first core node and the second core node as the primary leaf node of the split tree if no node exists in the first connected subgraph after deleting the second core node.

[0226] In an exemplary embodiment, the graph split tree obtaining unit 1150 may further include: a second connected subgraph obtaining unit, which can be used to obtain a second connected subgraph after deleting the second core node in the first connected subgraph if there are nodes in the first connected subgraph after deleting the second core node, and the nodes after deleting the second core node are connected; a third core node determining unit, which can be used to determine the third core node in the second connected subgraph based on the degree of each node in the second connected subgraph if the current split level of the split tree is less than or equal to the split level threshold; and a second primary leaf node obtaining unit, which can be used to combine the first core node, the second core node, and the third core node as primary leaf nodes of the split tree if there are no nodes in the second connected subgraph after deleting the third core node.

[0227] In an exemplary embodiment, the field aggregation group determination unit 1160 may include: a split gain acquisition unit, which can be used to obtain the split gain of each branch node in the split tree; a pruning operation unit, which can be used to perform a pruning operation on the split tree according to the size of the split gain of each branch node in the split tree and the leaf node threshold; a target leaf node determination unit, which can be used to take the initial leaf node retained after the split tree pruning operation as the target leaf node, and the number of target leaf nodes is less than or equal to the leaf node threshold; and an aggregation group determination unit, which can be used to take the target leaf node as the aggregation group of the dimension field.

[0228] In an exemplary embodiment, each branch node in the split tree may include a first branch node, and the first branch node may include each first child node of the first parent node. The splitting benefit obtaining unit may include: a first parent node dimension combination size obtaining unit, which can be used to obtain the dimension combination size of each dimension field in the first parent node; a first parent node query co-occurrence frequency obtaining unit, which can be used to obtain the query co-occurrence frequency among each dimension field in the first parent node; a root node query co-occurrence frequency obtaining unit, which can be used to obtain the query co-occurrence frequency among each dimension field in the root node; a first parent node query construction cost obtaining unit, which can be used to obtain the query construction cost of the first parent node based on the dimension combination size of each dimension field in the first parent node, the query co-occurrence frequency among each dimension field in the first parent node, and the query co-occurrence frequency among each dimension field in the root node; and a first child node dimension group. The unit for obtaining the combined size can be used to obtain the combined size of the dimensions of each dimension field in each first child node; the unit for obtaining the query co-occurrence frequency of each first child node can be used to obtain the query co-occurrence frequency among the dimensions of each first child node; the unit for obtaining the query construction cost of each first child node can be used to obtain the query construction cost of each first child node based on the combined size of the dimensions of each dimension field in each first child node, the query co-occurrence frequency among the dimensions of each first child node, and the query co-occurrence frequency among the dimensions of each dimension field in the root node; the unit for obtaining the splitting benefit of each first branch node can be used to obtain the splitting benefit of the first branch node based on the query construction cost of the first parent node and the query construction cost of each first child node.

[0229] In an exemplary embodiment, the data processing apparatus 1100 may further include: a target edge weight acquisition unit, configured to acquire the target edge weight of each edge in the dimension co-occurrence graph based on the initial edge weight of each edge in the dimension co-occurrence graph; a co-occurrence graph subgraph acquisition unit, configured to remove edges whose target edge weights are lower than a weight threshold to acquire each subgraph of the dimension co-occurrence graph; an associated dimension group acquisition unit, configured to acquire the maximum clique in each subgraph and form each associated dimension group based on the dimension fields in each maximum clique; a correlation inflation rate acquisition unit, configured to acquire the correlation and merging inflation rate between each associated dimension group; and a joint dimension acquisition unit, configured to merge each associated dimension group based on the correlation and merging inflation rate between each associated dimension group to acquire the joint dimension of the dimension co-occurrence graph, wherein the number of joint dimensions is less than or equal to a joint dimension threshold.

[0230] In an exemplary embodiment, the dimensional co-occurrence graph may include a first edge and a first node and a second node of the first edge. The target edge weight acquisition unit may include: a first query frequency acquisition unit, which can be used to obtain the first query frequency of the dimension field corresponding to the first node in historical query statements; a second query frequency acquisition unit, which can be used to obtain the second query frequency of the dimension field corresponding to the second node in historical query statements; a target query frequency determination unit, which can be used to determine the target query frequency based on the first query frequency and the second query frequency; and a first target edge weight acquisition unit, which can be used to obtain the target edge weight of the first edge based on the initial edge weight of the first edge and the target query frequency.

[0231] In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. The relevance inflation rate obtaining unit may include: an associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; a dimension combination size product unit, which can be used to obtain the product of the dimension combination sizes of the first associated dimension group and the dimension combination sizes of the second associated dimension group; a merged associated dimension group obtaining unit, which can be used to merge the first associated dimension group and the second associated dimension group to obtain a merged associated dimension group; a merged associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the merged associated dimension group; and an associated dimension group relevance obtaining unit, which can be used to obtain the relevance between the first associated dimension group and the second associated dimension group based on the dimension combination size of the merged associated dimension group and the product of the dimension combination sizes of the first associated dimension group and the dimension combination sizes of the second associated dimension group.

[0232] In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. The relevance inflation rate obtaining unit may include: an associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; a dimension combination size summing unit, which can be used to obtain the sum of the dimension combination sizes of the first associated dimension group and the second associated dimension group; a merged associated dimension group obtaining unit, which can be used to merge the first associated dimension group and the second associated dimension group to obtain a merged associated dimension group; a merged associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the merged associated dimension group; and an associated dimension combination inflation rate obtaining unit, which can be used to obtain the merging inflation rate between the first associated dimension group and the second associated dimension group based on the dimension combination size of the merged associated dimension group and the sum of the dimension combination sizes of the first associated dimension group and the second associated dimension group.

[0233] In an exemplary embodiment, the joint dimension obtaining unit may include: an associated dimension pair determining unit, which can be used to determine the associated dimension pair that meets the merging conditions in each associated dimension pair based on the relevance and merging expansion rate between each associated dimension pair; a target associated dimension pair obtaining unit, which can be used to obtain the target associated dimension pair with the highest relevance in the associated dimension pair; a target merged associated dimension pair obtaining unit, which can be used to merge the associated dimension pairs in the target associated dimension pair to obtain the target merged associated dimension pair, and delete the associated dimension pairs in the target associated dimension pair; and a joint dimension determining unit, which can be used to determine the target merged associated dimension pair and the unmerged associated dimension pair as the joint dimension of the dimension co-occurrence graph if the number of the target merged associated dimension pair and the number of the unmerged associated dimension pairs are less than or equal to the joint dimension threshold.

[0234] In an exemplary embodiment, the associated dimension pair determination unit may include: a merging condition determination unit, which can be used to determine that the corresponding associated dimension pair is an associated dimension pair that satisfies the merging condition if the correlation between the associated dimension pairs is greater than a first correlation threshold; or if the correlation between the associated dimension pairs is greater than a second correlation threshold and the merging expansion rate is less than a first expansion rate threshold; or if the correlation between the associated dimension pairs is greater than a third correlation threshold and the merging expansion rate is less than a second expansion rate threshold; wherein the first correlation threshold is greater than the second correlation threshold, the second correlation threshold is greater than the third correlation threshold, and the first expansion rate threshold is greater than the second expansion rate threshold.

[0235] In an exemplary embodiment, the data cube construction unit 1170 may include: a row key order determination unit, which can be used to obtain the row key order of each field in a historical query statement, the fields including dimension fields; and a data cube generation unit, which can be used to construct a data cube oriented towards the target table based on the row key order of each field, the joint dimension of the dimension co-occurrence graph, and the aggregation group of the dimension fields.

[0236] In an exemplary embodiment, the row key order determination unit may include: a field type weight parameter obtaining unit, which can be used to obtain weight parameters of different field types in historical query statements; a field type determining unit, which can be used to obtain each field and its field type in historical query statements; a field weight value obtaining unit, which can be used to obtain the field weight value of each field based on the weight parameters corresponding to the field type of each field and the query frequency of each field in historical query statements; and a row key order design unit, which can be used to determine the row key order of each field based on the field weight value of each field.

[0237] Other aspects of the data processing apparatus of this disclosure can be found in the above embodiments.

[0238] It should be noted that although several units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0239] The following is for reference. Figure 12 It shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application. Figure 12 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0240] Reference Figure 12 The electronic device provided in this embodiment may include: a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204.

[0241] The processor 1201, communication interface 1202 and memory 1203 communicate with each other through communication bus 1204.

[0242] Optionally, the communication interface 1202 can be an interface for a communication module, such as the interface for a GSM (Global System for Mobile communications) module. The processor 1201 is used to execute the program. The memory 1203 is used to store the program. The program may include a computer program containing computer operation instructions. The program may include a game client program.

[0243] Processor 1201 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0244] The memory 1203 may include high-speed RAM (random access memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0245] Specifically, the program can be used to: obtain the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields; use the dimension fields as nodes, determine the initial edge weights between different nodes based on the query co-occurrence frequency between different dimension fields, and form a dimension co-occurrence graph; determine the first core node in the dimension co-occurrence graph based on the degree of each node in the dimension co-occurrence graph; obtain the first connected subgraph after deleting the first core node in the dimension co-occurrence graph; obtain the split tree of the dimension co-occurrence graph based on the dimension co-occurrence graph, its first core node, and its first connected subgraph; determine the aggregation group of dimension fields based on the target leaf node of the split tree; and construct a data cube oriented towards the target table based on the aggregation group of dimension fields.

[0246] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0247] It should be understood that any number of elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0248] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0249] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Obtain the query co-occurrence frequency of dimension fields and different dimension fields in the historical query statements of the target table; Using the dimension fields as nodes, the initial edge weights between different nodes are determined based on the query co-occurrence frequency between different dimension fields, thus forming a dimension co-occurrence graph. Based on the degree of each node in the dimensional co-occurrence graph, determine the first core node in the dimensional co-occurrence graph; Obtain the first connected subgraph after deleting the first core node in the co-occurrence graph of the aforementioned dimension; Based on the dimensional co-occurrence graph, its first core node, and the first connected subgraph, a split tree of the dimensional co-occurrence graph is obtained, which includes: combining the nodes in the dimensional co-occurrence graph to form the root node of the split tree; determining the second core node in the first connected subgraph based on the degree of each node in the first connected subgraph; if after deleting the second core node in the first connected subgraph, there are no nodes in the first connected subgraph, then the first core node and the second core node are combined as the initial leaf node of the split tree. Determining the aggregation group of the dimension field based on the target leaf node of the split tree includes: obtaining the splitting reward of each branch node in the split tree; performing a pruning operation on the split tree based on the size of the splitting reward of each branch node and a leaf node threshold; using the initial leaf nodes retained after the split tree pruning operation as the target leaf nodes, and the number of the target leaf nodes is less than or equal to the leaf node threshold; and using the target leaf nodes as the aggregation group of the dimension field. Construct a data cube oriented toward the target table based on the aggregation groups of the dimension fields.

2. The method according to claim 1, characterized in that, Based on the dimensional co-occurrence graph, its first core node, and the first connected subgraph, a split tree of the dimensional co-occurrence graph is obtained, which further includes: If, after deleting the second core node in the first connected subgraph, there are still nodes in the first connected subgraph, and the nodes after deleting the second core node are connected, then a second connected subgraph after deleting the second core node in the first connected subgraph is obtained. If the current number of split layers in the split tree is less than or equal to the number of split layers threshold, then the third core node in the second connected subgraph is determined based on the degree of each node in the second connected subgraph. If, after deleting the third core node in the second connected subgraph, there are no nodes in the second connected subgraph, then the first core node, the second core node, and the third core node are combined as the initial leaf nodes of the split tree.

3. The method according to claim 1 or 2, characterized in that, Each branch node in the split tree includes a first branch node, and the first branch node includes each first child node of the first parent node; wherein, obtaining the splitting benefit of each branch node in the split tree includes: Obtain the combined size of the dimensions of each dimension field in the first parent node; Obtain the query co-occurrence frequency among the various dimension fields in the first parent node; Obtain the query co-occurrence frequency among the various dimension fields in the root node; The query construction cost of the first parent node is obtained based on the size of the dimension combination of each dimension field in the first parent node, the query co-occurrence frequency among each dimension field in the first parent node, and the query co-occurrence frequency among each dimension field in the root node. Obtain the combined size of the dimensions of each dimension field in each first child node; Obtain the query co-occurrence frequency among the various dimension fields in each first child node; The query construction cost of each first child node is obtained based on the size of the dimension combination of each dimension field in each first child node, the query co-occurrence frequency among each dimension field in each first child node, and the query co-occurrence frequency among each dimension field in the root node. The splitting benefit of the first branch node is obtained based on the query construction cost of the first parent node and the query construction cost of each first child node.

4. The method according to claim 1, characterized in that, Also includes: Based on the initial edge weights of each edge in the dimensional co-occurrence graph, the target edge weights of each edge in the dimensional co-occurrence graph are obtained. Remove edges whose target edge weights are lower than the weight threshold to obtain the subgraphs of the dimensional co-occurrence graph; Obtain the maximum clique in each subgraph, and form related dimension groups based on the dimension fields in each maximum clique; Obtain the correlation and merging expansion rate among the various related dimension groups; Based on the correlation and merging expansion rate between each associated dimension group, the associated dimension groups are merged to obtain the joint dimension of the dimension co-occurrence graph. The number of joint dimensions is less than or equal to the joint dimension threshold.

5. The method according to claim 4, characterized in that, The dimensional co-occurrence graph includes a first edge, a first node of the first edge, and a second node of the first edge; wherein, the target edge weights of each edge of the dimensional co-occurrence graph are obtained according to the initial edge weights of each edge, including: Obtain the first query frequency of the dimension field corresponding to the first node in the historical query statement; Obtain the second query frequency of the dimension field corresponding to the second node in the historical query statement; The target query frequency is determined based on the first query frequency and the second query frequency. The target edge weight of the first edge is obtained based on the initial edge weight of the first edge and the target query frequency.

6. The method according to claim 4, characterized in that, Each association dimension group includes a first association dimension group and a second association dimension group; wherein, obtaining the correlation between each association dimension group includes: Obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Obtain the product of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Merge the first associated dimension group and the second associated dimension group to obtain the merged associated dimension group; Obtain the size of the dimension combination of the merged related dimension group; The correlation between the first and second related dimension groups is obtained based on the product of the combined dimension size of the merged related dimension groups and the combined dimension size of the first and second related dimension groups.

7. The method according to claim 4, characterized in that, Each associated dimension group includes a first associated dimension group and a second associated dimension group; wherein, obtaining the merge expansion rate among each associated dimension group includes: Obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Obtain the sum of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Merge the first associated dimension group and the second associated dimension group to obtain the merged associated dimension group; Obtain the size of the dimension combination of the merged related dimension group; The merging expansion rate between the first and second associated dimension groups is obtained based on the sum of the dimension combination size of the merged associated dimension group and the dimension combination size of the first associated dimension group and the second associated dimension group.

8. The method according to claim 4, characterized in that, Based on the relevance and merging expansion rate among the various related dimension groups, the related dimension groups are merged to obtain the joint dimension of the dimension co-occurrence graph, including: Based on the correlation and merging expansion rate among the various related dimension groups, determine the related dimension group pairs that meet the merging conditions in each related dimension group; Obtain the target association dimension pair with the highest relevance among the association dimension pairs; Merge the associated dimension groups in the target associated dimension group pair to obtain the target merged associated dimension group, and delete the associated dimension groups in the target associated dimension group pair; If the number of target merged related dimension groups and unmerged related dimension groups is less than or equal to the joint dimension threshold, then the target merged related dimension groups and unmerged related dimension groups are respectively used as the joint dimensions of the dimension co-occurrence graph.

9. The method according to claim 4, characterized in that, Constructing a data cube oriented towards the target table based on the aggregation groups of the dimension fields, including: Obtain the row key order of each field in the historical query statement, where the fields include the dimension fields; A data cube oriented towards the target table is constructed based on the row key order of each field, the joint dimension of the dimension co-occurrence graph, and the aggregation group of the dimension fields.

10. The method according to claim 9, characterized in that, Obtain the row key order of each field in the historical query statement, including: Obtain the weight parameters of different field types in the historical query statements; Obtain each field and its field type from the historical query statement; Based on the weight parameters corresponding to the field types of each field and the query frequency of each field in the historical query statements, the field weight value of each field is obtained. The row key order of each field is determined based on its field weight value.

11. A data processing apparatus, characterized in that, include: The co-occurrence frequency acquisition unit is used to obtain the query co-occurrence frequency of dimension fields and different dimension fields in the historical query statements of the target table. The dimension co-occurrence graph construction unit is used to take the dimension fields as nodes, determine the initial edge weights between different nodes based on the query co-occurrence frequency between different dimension fields, and form a dimension co-occurrence graph. The core node determination unit is used to determine the first core node in the dimensional co-occurrence graph based on the degree of each node in the dimensional co-occurrence graph. A connected subgraph obtaining unit is used to obtain the first connected subgraph after deleting the first core node in the dimensional co-occurrence graph; The graph split tree acquisition unit is used to obtain the split tree of the dimensional co-occurrence graph based on the dimensional co-occurrence graph, its first core node, and the first connected subgraph. A field aggregation group determination unit is used to determine the aggregation group of the dimension field based on the target leaf node of the split tree; A data cube construction unit is used to construct a data cube oriented towards the target table based on the aggregation group of the dimension fields; The field aggregation group determination unit includes: The splitting reward acquisition unit is used to obtain the splitting reward of each branch node in the split tree; The pruning operation unit is used to perform pruning operations on the split tree based on the size of the splitting reward of each branch node and the threshold of the leaf node. The target leaf node determination unit is used to select the primary leaf nodes retained after the split tree pruning operation as target leaf nodes, and the number of target leaf nodes is less than or equal to the leaf node threshold. Aggregation group determination unit, used for aggregation groups that use the target leaf node as a dimension field; The graph split tree acquisition unit includes: a root node formation unit, used to combine nodes in the dimension co-occurrence graph to form the root node of the split tree; a second core node determination unit, used to determine the second core node in the first connected subgraph based on the degree of each node in the first connected subgraph; and a first primary leaf node acquisition unit, used to combine the first core node and the second core node as the primary leaf node of the split tree if no node exists in the first connected subgraph after deleting the second core node.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, include: At least one processor; A storage device configured to store at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the data processing method as described in any one of claims 1 to 10.

14. A computer program product comprising computer instructions, wherein executing the computer instructions causes the data processing method as described in any one of claims 1 to 10 to be performed.

Images