Method and device for optimizing data query information, electronic equipment and storage medium

Abstract

This disclosure relates to a method, apparatus, electronic device, and storage medium for optimizing data query information. The method for optimizing data query information includes: acquiring query information to be optimized and determining the graph structure features corresponding to the query information to be optimized, wherein the graph structure features are used to generate a graph structure of the query information, and the graph structure of the query information is used to represent execution information for performing a data query using the query information, the execution information including node information and node relationships; acquiring a target graph structure matching the graph structure features in a graph structure database, the graph structure database including graph structures generated based on historical query information; and optimizing the query information to be optimized based on the target graph structure. Through this disclosure, it is possible to acquire a matching graph structure generated from historical query information based on the graph structure features of the query information to be optimized, and then use the graph structure for optimization, thereby achieving accurate optimization of the query information to be optimized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for optimizing data query information. Background Technology

[0002] In the era of big data, with the rapid growth of data scale and the diversification of data applications, the efficiency of data processing using query information is becoming increasingly important.

[0003] In related technologies, data query statements are mainly optimized through preset rules or templates. However, using preset rules or templates to optimize query statements has problems of low accuracy and poor stability, resulting in poor optimization effect on the query information. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an optimization method, apparatus, electronic device and storage medium for data query information.

[0005] According to a first aspect of the present disclosure, a method for optimizing data query information is provided, comprising: acquiring query information to be optimized and determining a graph structure feature corresponding to the query information to be optimized, wherein the graph structure feature is used to generate a graph structure of the query information, and the graph structure of the query information is used to characterize execution information for performing a data query using the query information, the execution information including node information and node relationships; acquiring a target graph structure matching the graph structure feature in a graph structure database, the graph structure database including graph structures generated based on historical query information; and optimizing the query information to be optimized based on the target graph structure.

[0006] In one embodiment, obtaining a target graph structure matching the graph structure features in a graph structure database includes: generating a first graph structure based on the graph structure features corresponding to the query information to be optimized; obtaining a second graph structure in the graph structure database based on the first graph structure, wherein the second graph structure is the graph structure in the graph structure database with the highest similarity to the first graph structure; and using the second graph structure as the target graph structure matching the graph structure features.

[0007] In one embodiment, the graph structure database is determined as follows: parsing historical query information in a historical query information set to obtain node information and node relationships during the execution of data queries; generating a graph structure of the historical query information based on the node information and node relationships during the execution of data queries, wherein the graph structure is a directed acyclic graph, the node information during the execution process is the node of the directed acyclic graph, and the node relationships are the edges of the directed acyclic graph; and storing the graph structure of the historical query information to obtain the graph structure database.

[0008] In one embodiment, obtaining a second graph structure from a graph structure database based on the first graph structure includes: determining a graph query statement corresponding to the first graph structure, wherein the graph query statement is used to query the graph structure in the graph structure database that has the highest similarity to the first graph structure; and, based on the graph query statement, selecting the graph structure in the graph structure database that has the highest feature similarity to the first graph structure as the target graph structure.

[0009] In one embodiment, the method further includes: saving the first graph structure to the graph structure database and updating the graph structure database.

[0010] According to a second aspect of the present disclosure, an optimization apparatus for data query information is provided, comprising: a determining unit, configured to acquire query information to be optimized and determine a graph structure feature corresponding to the query information to be optimized, the graph structure feature being used to generate a graph structure of the query information, the graph structure of the query information being used to characterize execution information for performing a data query using the query information, the execution information including node information and node relationships; and a processing unit, configured to acquire a target graph structure matching the graph structure feature in a graph structure database, the graph structure database including graph structures generated based on historical query information, and optimize the query information to be optimized based on the target graph structure.

[0011] In one embodiment, the determining unit obtains a target graph structure matching the graph structure features from the graph structure database in the following manner: generating a first graph structure based on the graph structure features corresponding to the query information to be optimized; obtaining a second graph structure from the graph structure database based on the first graph structure, wherein the second graph structure is the graph structure in the graph structure database with the highest similarity to the first graph structure; and using the second graph structure as the target graph structure matching the graph structure features.

[0012] In one embodiment, the graph structure database is determined as follows: parsing historical query information in a historical query information set to obtain node information and node relationships during the execution of data queries; generating a graph structure of the historical query information based on the node information and node relationships during the execution of data queries, wherein the graph structure is a directed acyclic graph, the node information during the execution process is the node of the directed acyclic graph, and the node relationships are the edges of the directed acyclic graph; and storing the graph structure of the historical query information to obtain the graph structure database.

[0013] In one embodiment, obtaining a second graph structure from a graph structure database based on the first graph structure includes: determining a graph query statement corresponding to the first graph structure, wherein the graph query statement is used to query the graph structure in the graph structure database that has the highest similarity to the first graph structure; and, based on the graph query statement, selecting the graph structure in the graph structure database that has the highest feature similarity to the first graph structure as the target graph structure.

[0014] In one embodiment, the processing unit is further configured to: save the first graph structure to the graph structure database, and update the graph structure database.

[0015] According to a third aspect of the present disclosure, an optimization apparatus for data query information is provided, comprising:

[0016] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the optimized method for querying data information as described in the first aspect or any embodiment of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor of a terminal, can perform the method described in the first aspect or any embodiment of the first aspect.

[0018] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by determining the graph structure features corresponding to the query information to be optimized, and obtaining, from the graph structure database, a graph structure matching the node information and node relationships of the graph structure generated by the graph structure features, which is then used as the target graph structure to optimize the query information. Through the graph structure features of the query information to be optimized and the graph structures in the graph structure database, accurate matching of the graph structure matching the query information to be optimized is achieved, and optimization is performed based on the matched graph structure, thereby improving the accuracy of optimizing the query information.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 This is a flowchart illustrating an optimized method for querying data information according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating a method for determining a target graph structure according to an exemplary embodiment.

[0023] Figure 3 This is a flowchart illustrating a graph-structured database determination method according to an exemplary embodiment.

[0024] Figure 4 This is a flowchart illustrating a method for determining a second graph structure according to an exemplary embodiment.

[0025] Figure 5 This is a flowchart illustrating a graph-structured database update method according to an exemplary embodiment.

[0026] Figure 6 This is a schematic diagram illustrating how to save a graph structure to a graph structure database according to an exemplary embodiment.

[0027] Figure 7 This is a schematic diagram of a graph structure according to an exemplary embodiment.

[0028] Figure 8 This is a schematic diagram illustrating an example of obtaining and optimizing a target graph structure according to an exemplary embodiment.

[0029] Figure 9 This is a block diagram illustrating an optimization device for data query information according to an exemplary embodiment.

[0030] Figure 10 This is a block diagram illustrating an apparatus for optimizing data query information according to an exemplary embodiment.

[0031] Figure 11 This is a block diagram illustrating an apparatus for optimizing data query information according to an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0033] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, of the embodiments of this disclosure. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0034] The data query information optimization method provided in this disclosure is mainly used in data analysis scenarios. For example, scenarios involving the parsing and structured storage of Structured Query Language (SQL), or scenarios involving the design of database products for SQL analysis.

[0035] In the era of big data, with the growth of data scale and the diversification of data applications, data query efficiency has become extremely important. In the process of data processing, using poorly optimized data query information for data retrieval results in excessively long query times, which not only affects the normal operation of business but also risks delaying decision-making. Therefore, optimizing the query performance of data query information is a crucial aspect of the big data development process.

[0036] In related technologies, optimization techniques for data query information are primarily performed internally within the query engine. Examples include Rule-Based Optimizer (RBO) and Cost-Based Optimizer (CBO). Rule-based optimization determines the execution plan for the query information to be optimized based on predefined rules. Cost-based optimization determines the execution cost of the data query information through a preset evaluation strategy and optimizes the query information based on that cost. While RBO and CBO optimizations are versatile and can be applied to any data, they suffer from limitations in flexibility and the ability to optimize only predefined rules. They also have limitations in optimizing complex queries and may fail to generate execution plans that meet optimization requirements.

[0037] In related technologies, the optimization of data query information has expanded to History-Based Optimizer (HBO) to address the shortcomings of fixed-rule-based optimization methods such as Restricted Optimization (RBO) and Cost-Based Optimization (CBO). HBO optimization is strongly correlated with historical information, meaning it is highly relevant to business logic, adaptable, and offers performance improvements, generating execution plans that meet optimization requirements. In the HBO optimization process, feature extraction and analysis of historical query information, as well as correlation analysis of data query information, are important optimization methods. By identifying commonalities and similarities in historical query information, optimization is performed on the query information to be optimized based on these patterns and characteristics.

[0038] Optimizing data query information can involve adjusting the data query information itself, or preprocessing the database or tables within the database to be queried. For example, applications of determining historical query information based on data query similarity include, but are not limited to, at least one of the following:

[0039] 1. Query result preloading: By determining a sub-execution plan similar to the execution plan of the query to be optimized, and executing the similar sub-execution plan before executing the query to be optimized, it is possible to obtain some of the already executed results when executing the query to be optimized. Based on the already executed results, the query to be optimized is then executed, which speeds up the execution efficiency of the query to be optimized and improves the speed of obtaining the corresponding query results.

[0040] 2. Slow Query Analysis: Analyze slow query information from historical queries. Based on preset rules and features, retrieve historical query information that has similar characteristics to slow query information, obtain the historical query information, and optimize it accordingly.

[0041] 3. Data Index Construction: If the data tables have similar structures among the query information executed on them, the data tables are indexed to improve query efficiency. By comparing similar rows of the query information to be optimized, the data tables that need index optimization are identified, and indexes are built. For example, Z-Order indexes are used to optimize the data tables.

[0042] 4. Query blocking: For historical queries with poor results, extract the common features included in the historical queries with poor results, and quickly identify and block the execution of historical queries with poor results based on similarity comparison, so as to prevent waste of resources.

[0043] In related technologies, methods for similarity matching of query data include rule-based similarity matching methods, template-based similarity matching methods, semantic-based similarity matching methods, and machine learning-based similarity matching methods.

[0044] Rule-based similarity matching methods primarily determine the similarity of data query information through pre-defined rules and patterns. These pre-defined rules can be defined based on syntactic structure, keywords, table structure, etc. Template-based similarity matching methods convert data query information into template form and then determine the similarity of query information by comparing the similarity of templates. This method mainly focuses on the overall structure of the data query information for similarity matching and reduces the influence of semantic details on the similarity matching. Semantic similarity matching methods primarily determine similarity by analyzing the semantic information of the data query information. This requires the device used for similarity matching to have the ability to recognize patterns and data in the database, achieving accurate judgment of the similarity of data query information. Machine learning-based similarity matching methods mainly use machine learning algorithms to train models to learn the patterns and features of data query information, resulting in a model capable of judging the similarity of data query information. The training process of this model requires a large amount of training data and computational resources to improve the accuracy of the model's judgment.

[0045] However, the aforementioned technologies for data query information similarity matching have shortcomings. For example, rule-based similarity matching methods suffer from poor flexibility due to the complexity and low coverage of the rules, making it impossible to accurately and comprehensively define all rules. Furthermore, they cannot achieve similarity matching when new, special statements appear. Similarly, template-based similarity matching methods do not process the detailed information of the data query, reducing the accuracy of the similarity matching results and failing to handle complex data queries, thus exhibiting limitations. Semantic-based similarity matching methods, which primarily determine the database schema and the semantic content of the data, face significant implementation challenges. Therefore, the shortcomings of the aforementioned technologies for data query information similarity matching can be summarized as follows:

[0046] In view of this, embodiments of this disclosure provide a method for optimizing data query information. In a graph structure database including graph structures corresponding to historical query information, a target graph structure feature matching the graph structure feature corresponding to the query information to be optimized is obtained. The query information to be optimized is then optimized based on the target graph structure feature, thereby achieving accurate acquisition of the graph structure matching the query information to be optimized, and accurate optimization of the query information to be optimized based on the acquired graph structure.

[0047] Figure 1 This is a flowchart illustrating an optimized method for querying data information according to an exemplary embodiment, such as... Figure 1 As shown, it includes the following steps.

[0048] In step S11, the query information to be optimized is obtained, and the graph structure features corresponding to the query information to be optimized are determined.

[0049] In this embodiment, graph structure features are used to generate the graph structure of the query information. The graph structure features corresponding to the optimized query information can be extracted from the query information to be optimized using a preset graph structure feature extraction scheme. The graph structure features can be in formats such as text, vectors, or images. It should be understood that the above-described formats of graph structure features are for illustrative purposes only, and this embodiment does not specifically limit the form of the graph structure features.

[0050] In this embodiment of the disclosure, the graph structure of the query information is used to represent the execution information of performing data queries using the query information. The execution information includes node information and node relationships.

[0051] In step S12, a target graph structure matching the graph structure features is obtained from the graph structure database.

[0052] In this embodiment of the disclosure, the graph structure database includes graph structures generated based on historical query information. The target graph structure may be one or more graph structures in the graph structure database that match the graph structure features of the query information to be optimized. The node information and node relationships in the execution process of the historical query information represented by the target graph structure are matched with the node information and node relationships in the execution process of data query represented by the graph structure of the query information to be optimized.

[0053] In step S13, the query information to be optimized is optimized based on the target graph structure.

[0054] In this embodiment of the disclosure, by analyzing the node information and node relationships in the execution process used to represent historical query information in the target graph structure, the optimization operations that need to be performed for the query information to be optimized are determined, and the optimization is performed on the query information to be optimized, thereby improving the query efficiency and accuracy of querying using the query information to be optimized.

[0055] In this embodiment, by using the graph structure features of the query information to be optimized, a graph structure matching the graph structure features of the query information to be optimized is obtained from the graph structure database and determined as the target graph structure. The graph structures in the graph structure database are generated based on historical query information. Determining the graph structure corresponding to the historical query information that matches the query information to be optimized achieves accurate matching based on the execution logic and semantics of the query information, improving the accuracy of similarity judgments that are often caused by comparing text rules based on query information. Furthermore, by optimizing the query information to be optimized using the accurately determined target graph structure, the optimization effect can be improved.

[0056] The following embodiments of this disclosure provide a detailed description of the process of obtaining a target graph structure that matches the graph structure features in a graph structure database.

[0057] Figure 2 This is a flowchart illustrating a method for determining a target graph structure according to an exemplary embodiment, such as... Figure 2 As shown, it includes the following steps.

[0058] In step S21, a first graph structure is generated based on the graph structure features corresponding to the query information to be optimized.

[0059] In this embodiment, the first graph structure includes node information and node relationships during the execution process of querying data using the query information to be optimized. This information is used to characterize the execution plan of the query information to be optimized and the performance parameters involved in the execution plan. In other words, the first graph structure can be understood as the graph structure corresponding to the query information to be optimized.

[0060] In step S22, based on the first graph structure, the second graph structure is obtained from the graph structure database.

[0061] In this embodiment, the second graph structure is the graph structure in the graph structure database that has the highest similarity to the first graph structure. The similarity between the node information in the second graph structure and the node information in the first graph structure is greater than a similarity threshold, and the similarity between the node relationships in the second graph structure and the node relationships in the first graph structure is also greater than a similarity threshold. That is, the second graph structure can be understood as a graph structure stored in the graph database that matches the first graph structure.

[0062] In this embodiment of the disclosure, the matching requirements for node information and / or node relationships can also be determined based on the first graph structure, and the graph structure in the graph structure database that meets the matching requirements for node information and / or node relationships can be determined as the second graph structure.

[0063] In step S23, the second graph structure is selected as the target graph structure that matches the graph structure features.

[0064] In this embodiment, by using the second graph structure as the target graph structure for matching the graph structure features of the query information to be optimized, and determining the graph structure in the graph structure database with the highest similarity to the first graph structure as the target graph structure, the acquisition of graph structures generated from historical query information is achieved. Through prior similarity matching of graph structures, the accuracy of determining the graph structure corresponding to historical query information similar to the query information to be optimized is improved.

[0065] In this embodiment of the disclosure, the graph structure can be a tree graph or a directed acyclic graph (DAG). If the graph structure is a tree graph, the nodes in the tree graph can be used to represent node information, the connecting lines between nodes in the tree graph can be used to represent node relationships, and the parent-child relationships between nodes can be used to represent the order of node operations or the direction of data flow. If the graph structure is a directed acyclic graph, the nodes in the directed acyclic graph can be used to represent node information, the edges in the directed acyclic graph can be used to represent node relationships, and the arrows on the edges can be used to represent the order of node operations or the direction of data flow.

[0066] The following embodiments of this disclosure use a directed acyclic graph (DAG) as an example, where the graph structure is a DAG, the node information is the nodes of the DAG, and the node relationships are the edges of the DAG, to illustrate the determination of the graph structure database.

[0067] Figure 3 This is a flowchart illustrating a method for determining a graph-structured database according to an exemplary embodiment, such as... Figure 3 As shown, it includes the following steps.

[0068] In step S31, the historical query information in the historical query information set is parsed to obtain the node information and node relationships in the execution process of the historical query information during the data query.

[0069] In this embodiment of the disclosure, the historical query information set is a preset set used to store historical query information of executed data queries. The node information and node relationships in the execution process of the historical query information can characterize the execution plan and performance data of the historical query information during the execution process.

[0070] In step S32, a graph structure of historical query information is generated based on the node information and node relationships during the execution of data queries in the historical query information.

[0071] In this embodiment, the graph structure is a directed acyclic graph (DAG), i.e., a directed graph without loops. The node information during execution is represented by the nodes of the DAG, and the node relationships are represented by the edges of the DAG. Specifically, the nodes of the DAG represent the node information during the execution of the query statement. The node information includes the attributes corresponding to the nodes. The nodes of the DAG can be used to represent an operation in the execution plan of the query statement, and the attributes corresponding to the nodes in the DAG can represent information such as the function, state, and execution status of the operation. Edges of the DAG can be set to represent the node relationships during the execution of the query statement, indicating the direction of data flow between operations.

[0072] In step S33, the graph structure for storing historical query information is obtained, thus creating a graph structure database.

[0073] In this embodiment, a directed acyclic graph (DAG) representing the node information and relationships during the execution process of data querying based on historical query information is stored as the graph structure of the historical query information. This enables the construction of a graph structure representing the operations and corresponding attribute information in the execution plan of the historical query information, resulting in a graph structure database including the graph structure of the historical query information. Specifically, the nodes and edges in the DAG are used to provide detailed execution information of key operations during the data querying process of the historical query information, as well as the execution order of each operation, thus achieving a precise representation of the execution plan of the historical query information.

[0074] In this embodiment of the disclosure, the structured graph structure can be saved in the form of a graph structure database, and the similarity matching of query information can be achieved by using the efficient search capability of the graph structure database through graph matching.

[0075] In this embodiment, the similarity can be determined by comparing the graph structure features of the first graph structure and the graph structures in the graph structure database. Alternatively, the second graph structure can be obtained by querying the graph structure database after determining the graph query statement corresponding to the first graph structure. The graph query statement can be a graph database language such as Gremlin, Cypher, or nGQL.

[0076] The following embodiments of this disclosure describe how to obtain the second graph structure by determining the graph query statement corresponding to the first graph structure.

[0077] Figure 4 This is a flowchart illustrating a second graph structure determination method according to an exemplary embodiment, such as... Figure 4 As shown, it includes the following steps.

[0078] In step S41, the graph query statement corresponding to the first graph structure is determined.

[0079] In this embodiment of the disclosure, the graph query statement is used to query the graph structure in the graph structure database that has the highest similarity to the graph structure features. The graph query statement corresponding to the first graph structure or graph structure feature can be determined through preset rules.

[0080] In step S42, based on the graph query statement, the graph structure with the highest similarity to the graph structure features in the graph structure database is selected as the target graph structure.

[0081] In this embodiment of the disclosure, a graph query statement is used to query graph structures composed of complex nodes and edges in the graph structure database, obtain graph structures that match the graph structure features as target graph structures, realize the efficient search capability of the graph structure database, achieve second-level search of the graph structure database, and shorten the matching time.

[0082] In this embodiment of the disclosure, after optimizing the query information to be optimized, the graph structure corresponding to the information to be optimized can also be saved to the graph structure database to realize real-time updates of the graph structure database.

[0083] Figure 5 This is a flowchart illustrating a graph-structured database update method according to an exemplary embodiment, such as... Figure 5 As shown, it includes the following steps.

[0084] In step S51, a first graph structure is generated based on the graph structure features corresponding to the query information to be optimized.

[0085] In step S52, the first graph structure is saved to the graph structure database, and the graph structure database is updated.

[0086] In this embodiment, after obtaining target graph data in the graph structure database that matches the graph structure features of the query information to be optimized, a first graph structure generated based on the graph structure features of the query information to be optimized can be saved in the graph structure database as a graph structure generated based on the query information to be optimized. During the optimization of other query information to be optimized, the graph structures corresponding to historical query information similar to the query information to be optimized are obtained by searching the graph structure database for graph structures with the graph structure features of the query information to be optimized. By analyzing the graph structures corresponding to similar historical query information, the common execution features and performance indicators of the query information to be optimized and historical query information are determined, improving the accuracy of the analysis of the query information to be optimized. Furthermore, the optimization strategy is determined and optimized based on the analysis results of the accurately obtained query information to be optimized, achieving accurate optimization of the query information to be optimized.

[0087] In this embodiment, by saving each graph structure generated based on the query information to be optimized to a graph structure database, the data content in the graph structure data is increased, and the graph data included in the graph structure database is expanded. This allows the graph structure database to accumulate query information and has good scalability. By using the updated graph structure database as the database for obtaining the second graph structure, support for real-time storage of historical query information is achieved. This makes the graph structure database highly flexible, automatically storing the graph structure corresponding to the newly executed query information to be optimized without manual operation, facilitating subsequent analysis and optimization of other query information to be optimized.

[0088] It should be understood that, Figure 5 Step S51 and Figure 2 The steps in step S21 are the same and will not be repeated here. Furthermore, step S51 can also be the same as... Figure 2 The steps S22 or S23 are the same, and in this embodiment, step S52 is... Figure 2 Execute after step S21, without proceeding to step S52. Figure 2 The execution order of steps S21 or S22 in the process is limited.

[0089] In an exemplary embodiment, the process of generating a graph structure for query information and saving the graph structure corresponding to the query information to a graph structure database is as follows: Figure 6 As shown. Figure 6 This is a schematic diagram illustrating how to save a graph structure to a graph structure database according to an exemplary embodiment.

[0090] exist Figure 6 In this process, the query information is the query statement (SQL), which is used to obtain the query statement for which the corresponding graph structure needs to be saved to the graph structure database. The SQL execution engine is controlled to collect execution information representing the SQL statement execution plan. This is achieved by adding an information collection module for the SQL execution engine's execution logic and nodes, and collecting execution information from the SQL execution plan in the form of node information. For example, if the SQL execution engine is the Spark SQL execution engine, it covers the valid node information in the Spark execution plan. Based on the extracted SQL node information, a graph relationship structure is constructed, the query statement execution logic is structured, and a graph structure for the query statement is generated. The generated graph structure is then stored in the graph structure database, with nodes and edges added to the database. Furthermore, all currently executing query statements can be saved to the graph structure database through the above operations, completing the process of collecting and storing the structured features (graph structure) of all historical query information.

[0091] For example, valid node information in a Spark execution plan includes nodes and corresponding node attributes, including but not limited to one or more of the following nodes and node attributes: The `BatchScanExec` node, used for batch scanning of data from a data source, includes attributes such as `cluster` (cluster name), `catalogName` (catalog name), `databaseName` (database name), `tableName` (table name), `format` (data format), `partitionNum` (number of partitions scanned), `targetSize` (partition size), `pushedDownFilters` (pushed-down filters), `readSchema` (columns read), `output` (output columns), `dataFileNum` (number of data files), and `deleteFileNum` (number of deleted files). The `ProjectExec` node, used to represent the projection operation of the executed fields, i.e., selecting the columns or column expressions needed in the query, includes attributes such as `projectList` (output columns). The `FilterExec` node, used to represent the application of filtering conditions to allow only rows that meet specific conditions to pass, includes the `condition` (filter condition) attribute. The `SortExec` node represents the sorting of data, ranking rows according to a specified column and sorting direction (ascending or descending), including attributes such as `sortOrder` (sorting rule) and `global` (whether it's a global sort). The `BaseAggregateExec` node represents the execution of aggregation operations, including nodes such as `requiredChildDistributionExpressions` (required child node data distribution method), `groupingExpressions` (aggregate key), `aggregateExpressions` (aggregate function), `aggregateAttributes` (aggregate function attr), `resultExpressions` (output columns), and `aggType` (aggregate type). The `BaseJoinExec` node is the base class for the physical plan nodes of all join operations, including attributes such as `joinType` (join type), `condition` (join condition), `leftKeys` (left table join column), `rightKeys` (right table join column), `isNullAwareAntiJoin` (whether NULL values ​​need special handling), `isSkewJoin` (whether there is data skew), and `buildSide` (broadcast table construction). The CoalesceExec node represents a reduction in the number of partitions without requiring data shuffling, and includes attributes such as numPartitions (number of partitions).The `SubqueryExec` node represents a subquery. The `ExpandExec` node represents cross-join operations in SQL, including attributes such as `projections` (aggregated columns) and `output` (output columns). The `GenerateExec` node implements lateral views and the `POSEXPR` function in SQL, including attributes such as `generator` (expression), `requiredChildOutput` (required child node output columns), `outer` (whether the generator expression is executed in the context of the outer query), and `generatorOutput` (output columns). The `LocalLimitExec` node represents limiting the number of query results (LIMIT) within a single partition, including attributes such as `limit` (limited number of records). The `TakeOrderedAndProjectExec` node represents executing a TOP-K type query, retrieving the first preset number of records from the sorted results, including attributes such as `limit` (preset number of records), `sortOrder` (sorting expression), and `projectList` (output columns).

[0092] It should be understood that the above nodes and corresponding attributes are for illustrative purposes only, and no limitation is made on the nodes and corresponding node attributes in this disclosure.

[0093] In an exemplary embodiment, if the query is "select a,b,count(c)from tmp.test groupby 1,2", it represents grouping the data in the test table of the tmp database according to the values ​​of columns a and b, and returning the values ​​of columns a and b for each group, as well as the number of non-NULL values ​​in column c within that group. By analyzing the execution of this query in the SQL engine, the execution plan corresponding to the query is determined to include: Scan node, Aggregate1 node, Exchange node, and Aggregate2 node. Specifically, in the Scan node, the database name attribute is tmp, the table name attribute is test, the number of files read is 1, the file scan time attribute is 114 milliseconds (ms), the data size attribute is 953.0B, and the number of data rows read is 3. In the Aggregate1 node, the number of output data rows attribute is 3. In the Exchange node, the attribute value for the number of rows written in the shuffle operation is 3, the attribute value for the shuffle write size is 6KB, the attribute value for the number of rows read in the shuffle operation is 3, and the attribute value for the shuffle read size is 6KB. In the Aggregate2 node, the attribute value for the number of rows output is 3. The execution plan determines the operations for creating the graph structure corresponding to this query information: create nodes (Scan, Aggregate1, Exchange, Aggregate2) and attach corresponding attributes to the nodes, such as table name and file read volume. Create relationships (Scan)->(Aggregate1), (Aggregate1)->(Exchange), and (Exchange)->(Aggregate2). The graph structure of the query information is generated through these creation operations. A schematic diagram of the generated graph structure is shown below. Figure 7 As shown, Figure 7 This is a schematic diagram illustrating a graph structure according to an exemplary embodiment. Figure 7 In the graph structure, the nodes in the execution plan and their corresponding attributes are included, and the execution order of the nodes is represented by directed edges.

[0094] In one exemplary embodiment, the process of obtaining the target graph structure from the graph structure database based on the graph structure features of the query information to be optimized, and then optimizing based on the target graph structure, is as follows: Figure 8 As shown. Figure 8 This is a schematic diagram illustrating an example of obtaining and optimizing a target graph structure according to an exemplary embodiment.

[0095] exist Figure 8In this context, node attributes such as the number of files read, scan time, and data volume (also known as key node metrics) help evaluate performance. Constructing a graph relationship structure, such as the flow from Scan to Aggregate and Exchange, helps identify the execution pattern of the query statement. A graph structure representation of the query statement can be generated based on human experience and preset optimization strategies, including: the parent node is Filter, the child nodes are DataScan, there are multiple ScanTasks, the data volume read by DataScan, and the data volume output by Filter. Based on this graph structure representation, a graph structure of the query information to be optimized is generated, i.e., the first graph structure.

[0096] Based on the first graph structure, a graph query statement (also known as a graph language query expression) corresponding to the first graph structure is generated. Taking nGQL as the graph query statement as an example, the generated graph query statement can be:

[0097] MATCH(s:BatchScanExec)-->(f:FilterExec)

[0098] / / Used to find specific nodes and relationships in a graph. Here, it searches for a node of type BatchScanExec (denoted as s), which points to a node of type FilterExec (denoted as f) through a directed relationship.

[0099] where

[0100] s.BatchScanExec.sqlStartTime >= '${start time}'

[0101] The `sqlStartTime` property of the `BatchScanExec` node must be greater than or equal to the specified start time.

[0102] and s.BatchScanExec.sqlStartTime<'${endtime}'

[0103] The sqlStartTime property must be less than the specified end time.

[0104] and s.BatchScanExec.nonEmptyInputDataCount>=${scanTask quantity threshold}

[0105] The `nonEmptyInputDataCount` property must be greater than or equal to the specified threshold number.

[0106] and s.BatchScanExec.numOutputRows>=${scanTask output row threshold}

[0107] The numOutputRows property must be greater than or equal to the specified threshold for the number of output rows.

[0108] and s.BatchScanExec.numOutputRows / f.FilterExec.numOutputRows>=${the number of items reduced after filtering}

[0109] / / The ratio of the number of output rows of the BatchScanExec node to the number of output rows of the FilterExec node must be greater than or equal to the specified reduction factor.

[0110] and s.BatchScanExec.pushedDownFilters! ="

[0111] / / The `pushedDownFilters` property cannot be an empty string.

[0112] RETURN

[0113] s.BatchScanExec.catalogName as catalogName,

[0114] / / Return the catalogName property of the BatchScanExec node and name it catalogName.

[0115] s.BatchScanExec.databaseName as databaseName,

[0116] / / Returns the databaseName property and names it databaseName

[0117] s.BatchScanExec.tableName as tableName,

[0118] / / Returns the tableName property and names it tableName

[0119] s.BatchScanExec.pushedDownFilters as pushedDownFilters,

[0120] / / Returns the pushedDownFilters property and names it pushedDownFilters

[0121] f.FilterExec.condition as condition,

[0122] / / Returns the condition property of the FilterExec node and names it condition

[0123] count(1) as count

[0124] / / Returns the number of matching patterns and names it as count

[0125] orderby

[0126] / / The results are sorted by catalogName, databaseName, tableName, pushedDownFilters, condition, and count, where count is catalogName sorted in descending order.

[0127] databaseName,

[0128] tableName,

[0129] pushedDownFilters

[0130] condition

[0131] count desc

[0132] limit 30;

[0133] / / The maximum number of results returned is 30.

[0134] Submit a graph query to the graph database service to find graph structures and table names that match the query to be optimized. Optimize or manage the found graph structures. For example, if historical query information corresponding to the found graph structures matches the query to be optimized and is suitable for Z-Order index optimization, then Z-Order indexes can be created on the tables within the found graph structures. This allows all queries with similar characteristics to the query to be optimized to retrieve data through the index, improving query efficiency.

[0135] In this embodiment, by acquiring the graph structure features corresponding to the query information to be optimized, and based on the same structure features, obtaining the target graph structure in the graph structure database that matches the graph structure features, accurate acquisition of graph structures matching the query information to be optimized is achieved. By using in-depth semantic analysis and structured graph structures to acquire the target graph structure, false positives and false negatives are avoided, providing more reliable historical query information for optimization such as query optimization and slow query analysis. Furthermore, by leveraging the efficient search capabilities of the graph structure database, the complex similarity comparison of query information is transformed into a graph structure matching problem, shortening the query time.

[0136] Based on the same concept, embodiments of this disclosure also provide an optimization device for data query information.

[0137] It is understood that the data query information optimization apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0138] Figure 9 This is a block diagram illustrating a data query information optimization apparatus 100 according to an exemplary embodiment. (Refer to...) Figure 9 The device includes a determining unit 101 and a processing unit 102.

[0139] The determining unit 101 is used to obtain the query information to be optimized and determine the graph structure features corresponding to the query information to be optimized. The graph structure features are used to generate the graph structure of the query information. The graph structure of the query information is used to represent the execution information of data query using the query information. The execution information includes node information and node relationships.

[0140] The processing unit 102 is used to obtain a target graph structure that matches the graph structure features from the graph structure database. The graph structure database includes graph structures generated based on historical query information, and optimizes the query information to be optimized based on the target graph structure.

[0141] In one embodiment, the determining unit 101 obtains a target graph structure matching the graph structure features in the graph structure database in the following manner: a first graph structure is generated based on the graph structure features corresponding to the query information to be optimized; a second graph structure is obtained from the graph structure database based on the first graph structure, wherein the second graph structure is the graph structure in the graph structure database with the highest similarity to the first graph structure; and the second graph structure is used as the target graph structure matching the graph structure features.

[0142] In one implementation, the graph structure database is determined as follows: parsing historical query information from the historical query information set to obtain node information and node relationships during the execution of data queries; generating a graph structure of historical query information based on the node information and node relationships during the execution of data queries, wherein the graph structure is a directed acyclic graph, the node information during the execution process is the node of the directed acyclic graph, and the node relationships are the edges of the directed acyclic graph; and storing the graph structure of historical query information to obtain the graph structure database.

[0143] In one implementation, obtaining a second graph structure from a graph structure database based on a first graph structure includes: determining a graph query statement corresponding to the first graph structure, wherein the graph query statement is used to query the graph structure in the graph structure database that has the highest similarity to the first graph structure; and, based on the graph query statement, selecting the graph structure in the graph structure database that has the highest feature similarity to the graph structure as the target graph structure.

[0144] In one embodiment, the processing unit 102 is further configured to: save the first graph structure to the graph structure database and update the graph structure database.

[0145] According to a third aspect of the present disclosure, an optimization apparatus for data query information is provided, comprising:

[0146] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute an optimized method for querying data information in the first aspect or any embodiment of the first aspect.

[0147] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0148] Figure 10 This is a block diagram illustrating an optimized device 200 for data retrieval information according to an exemplary embodiment. For example, device 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0149] Reference Figure 10The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0150] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0151] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0152] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0153] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0154] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0155] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0156] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0157] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0158] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0160] Figure 11 This is a block diagram illustrating an optimized apparatus 300 for data querying information, according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 3 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned optimized method for querying data information.

[0161] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0162] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0163] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0164] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0165] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0166] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0167] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An optimization method for data query information, characterized in that, include: Obtain the query information to be optimized and determine the graph structure features corresponding to the query information to be optimized. The graph structure features are used to generate the graph structure of the query information. The graph structure of the query information is used to represent the execution information of data querying using the query information. The execution information includes node information and node relationships. In the graph structure database, a target graph structure that matches the graph structure features is obtained. The graph structure database includes graph structures generated based on historical query information. Based on the target graph structure, the query information to be optimized is optimized.

2. The method according to claim 1, characterized in that, The step of obtaining a target graph structure that matches the graph structure features in the graph structure database includes: Based on the graph structure features corresponding to the query information to be optimized, a first graph structure is generated; Based on the first graph structure, a second graph structure is obtained from the graph structure database, wherein the second graph structure is the graph structure in the graph structure database that has the greatest similarity to the first graph structure. The second graph structure is used as the target graph structure that matches the features of the graph structure.

3. The method according to claim 1 or 2, characterized in that, The graph structure database is determined in the following way: Parse the historical query information in the historical query information set to obtain the node information and node relationships of the historical query information during the data query execution process; Based on the node information and node relationships during the execution of the data query based on the historical query information, a graph structure of the historical query information is generated, wherein the graph structure is a directed acyclic graph, the node information during the execution process is the node of the directed acyclic graph, and the node relationships are the edges of the directed acyclic graph; The graph structure that stores historical query information is used to obtain the graph structure database.

4. The method according to claim 2, characterized in that, The step of obtaining the second graph structure from the graph structure database based on the first graph structure includes: Determine the graph query statement corresponding to the first graph structure, wherein the graph query statement is used to query the graph structure in the graph structure database that has the greatest similarity to the first graph structure; Based on the graph query statement, the graph structure with the highest feature similarity to the graph structure in the graph structure database is selected as the target graph structure.

5. The method according to claim 2, characterized in that, The method further includes: The first graph structure is saved to the graph structure database, and the graph structure database is updated.

6. An optimization device for data query information, characterized in that, include: A determining unit is used to obtain query information to be optimized and determine the graph structure features corresponding to the query information to be optimized. The graph structure features are used to generate the graph structure of the query information. The graph structure of the query information is used to characterize the execution information of data query using the query information. The execution information includes node information and node relationships. The processing unit is configured to obtain a target graph structure that matches the graph structure features from a graph structure database, the graph structure database including graph structures generated based on historical query information, and optimize the query information to be optimized based on the target graph structure.

7. The apparatus according to claim 6, characterized in that, The determining unit obtains the target graph structure that matches the graph structure features from the graph structure database in the following manner: Based on the graph structure features corresponding to the query information to be optimized, a first graph structure is generated; Based on the first graph structure, a second graph structure is obtained from the graph structure database, wherein the second graph structure is the graph structure in the graph structure database that has the greatest similarity to the first graph structure. The second graph structure is used as the target graph structure that matches the features of the graph structure.

8. The apparatus according to claim 6 or 7, characterized in that, The graph structure database is determined in the following way: Parse the historical query information in the historical query information set to obtain the node information and node relationships of the historical query information during the data query execution process; Based on the node information and node relationships during the execution of the data query based on the historical query information, a graph structure of the historical query information is generated, wherein the graph structure is a directed acyclic graph, the node information during the execution process is the node of the directed acyclic graph, and the node relationships are the edges of the directed acyclic graph; The graph structure that stores historical query information is used to obtain the graph structure database.

9. The apparatus according to claim 7, characterized in that, The step of obtaining the second graph structure from the graph structure database based on the first graph structure includes: Determine the graph query statement corresponding to the first graph structure, wherein the graph query statement is used to query the graph structure in the graph structure database that has the greatest similarity to the first graph structure; Based on the graph query statement, the graph structure with the highest feature similarity to the graph structure in the graph structure database is selected as the target graph structure.

10. The apparatus according to claim 7, characterized in that, The processing unit is also used for: The first graph structure is saved to the graph structure database, and the graph structure database is updated.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method described in any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by the processor of the electronic device, enable the terminal to perform the method described in any one of claims 1 to 5.

Images