HQL statement detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of data processing technologies and financial science and technology, and provides an HQL statement detection method which comprises the steps that an HQL statement to be detected is analyzed to generate an abstract syntax tree and optimized, and an execution plan is generated according to the optimized syntax tree; performing operation step decomposition on the execution plan according to task process stages, and generating a performance analysis syntax tree corresponding to the HQL statement by taking operation steps obtained after decomposition as syntax tree nodes; querying a database according to a query object contained in the operation steps in the performance analysis syntax tree to obtain a related database table of the HQL statement, and calculating data of the related database table and data of the operation steps of the performance analysis syntax tree to obtain numerical values of a plurality of preset data indexes of the HQL statement; and calculating the score of the HQL statement according to the numerical value of the preset data index and a preset data index scoring algorithm, and generating a detection report of the HQL statement. According to the HQL statement detection method, the quality of the HQL statement can be automatically evaluated according to a unified rule.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to an HQL statement detection method, device, electronic device and storage medium. Background Art

[0002] As the core part of various financial technology information systems, non-standard database operations can easily lead to problems such as transaction response timeouts and database failures, which directly affect the availability of the entire information system. At present, in the process of database construction, manually written and uploaded HQL statements are often used for database operations. The development skills and experience of the writers are different, and the quality of the uploaded HQL statements varies. Since there is no unified set of HQL statement evaluation standards, it is impossible to conduct standardized quality inspections on HQL statements. Summary of the invention

[0003] The present invention provides an HQL statement detection method, device, electronic device and storage medium, the main purpose of which is to provide a method for automatically evaluating the quality of HQL statements using a unified rule algorithm to solve the technical problem that it is impossible to perform standardized quality detection on HQL statements.

[0004] In a first aspect, the present invention provides a HQL statement detection method, the method comprising:

[0005] Obtaining an HQL statement to be detected in a database, parsing the HQL statement to generate an abstract syntax tree, optimizing the abstract syntax tree, and generating an execution plan according to the optimized syntax tree;

[0006] Decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decomposing the map operation tree and the reduce operation tree into operation steps, and using the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement;

[0007] querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain a related database table of the HQL statement, calculating the data in the related database table and the data in the operation step of the performance analysis syntax tree, and obtaining the values ​​of multiple preset data indicators of the HQL statement according to the calculation results;

[0008] Obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0009] In a second aspect, the present invention further provides an HQL statement detection device, the HQL statement detection device comprising:

[0010] A statement parsing module is used to obtain the HQL statement to be detected in the database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree;

[0011] A step decomposition module is used to decompose the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decompose the map operation tree and the reduce operation tree into operation steps, and use the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement;

[0012] A query calculation module, used to query the database according to the query object included in the operation step in the performance analysis syntax tree to obtain the relevant database table of the HQL statement, calculate the data of the relevant database table and the data of the operation step of the performance analysis syntax tree, and obtain the values ​​of multiple preset data indicators of the HQL statement according to the calculation results;

[0013] A statement scoring module is used to obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0014] In a third aspect, the present invention further provides an electronic device, the electronic device comprising:

[0015] at least one processor; and,

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the HQL statement detection method as described above.

[0018] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned HQL statement detection method is implemented.

[0019] The present invention generates a logical execution plan and a physical execution plan by parsing an HQL statement, decomposes the execution steps of the logical execution plan and the physical execution plan to generate a performance analysis syntax tree, queries the relevant database table of the HQL statement based on the performance analysis syntax tree and counts the data indicators of the HQL statement, calculates the quality score of the HQL statement according to the relevant database table, the data indicators and the pre-set scoring rules, and realizes automatic evaluation of the quality of the HQL statement with unified rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 It is a schematic diagram of an application environment of an HQL statement detection method in one embodiment of the present invention;

[0022] Figure 2 It is a flow chart of a HQL statement detection method in one embodiment of the present invention;

[0023] Figure 3 It is a structural schematic diagram of an HQL statement detection device in one embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clear, the principle and spirit of the present invention will be described below with reference to several specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical solutions between each embodiment can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0028] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0029] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" that appear in the differences in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0030] The HQL statement detection method provided by the embodiment of the present invention can be applied in the following situations: Figure 1In an application environment, the client communicates with the server through a network. The server can obtain the HQL statement to be detected in the database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree; decompose the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decompose the map operation tree and the reduce operation tree into operation steps, and use the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement; query the database according to the query object contained in the operation step in the performance analysis syntax tree to obtain the relevant database table of the HQL statement, calculate the data of the relevant database table and the data of the operation step of the performance analysis syntax tree, and obtain the values ​​of multiple preset data indicators of the HQL statement according to the calculation results; obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement. In the present invention, a logical execution plan and a physical execution plan are generated by parsing HQL statements, the execution steps of the logical execution plan and the physical execution plan are decomposed to generate a performance analysis syntax tree, the relevant database table of the HQL statement is queried based on the performance analysis syntax tree and the data indicators of the HQL statement are counted, and the quality score of the HQL statement is calculated according to the relevant database table, data indicators and pre-set scoring rules, thereby realizing automatic evaluation of the quality of HQL statements with unified rules. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2 1 is a flow chart of the HQL statement detection method of the present invention, wherein the HQL statement detection method comprises steps S1-S4.

[0032] S1. Obtain an HQL statement to be detected in a database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree.

[0033] In one embodiment, the parsing of the HQL statement to generate an abstract syntax tree, optimizing the abstract syntax tree, and generating an execution plan according to the optimized syntax tree includes:

[0034] Performing lexical analysis and grammatical analysis on the HQL statement according to predefined lexical rules and grammatical rules to obtain an abstract syntax tree (AST Tree) corresponding to the HQL statement;

[0035] Traversing the abstract syntax tree to abstract a query block (Query Block);

[0036] Traversing the query block, converting the query block into a logical execution plan (execution operation tree OperatorTree), and optimizing the logical execution plan through a logical optimizer;

[0037] Traverse the optimized logical execution plan, convert the optimized logical execution plan into a physical execution plan (MapReduce task flow), and optimize the physical execution plan through a physical optimizer;

[0038] An execution plan corresponding to the HQL statement is generated based on the optimized logical execution plan and the optimized physical execution plan.

[0039] Specifically, HQL (Hibernate Query Language) is an object-oriented query language, similar to SQL, but instead of operating on tables and columns, it operates on objects and their properties. HQL queries can be translated by Hibernate into traditional SQL queries to operate on the database.

[0040] Specifically, the abstract syntax tree is a data model for representing various parts of an HQL query statement, and its nodes are specific data structures for representing part of the source code of a query statement. The data structure of each node contains four attributes: position, type, value, and children, wherein position is used to represent the position information of the node, describing the path of the node in the syntax tree and the order of the node as a child node; type is used to represent the type of the node; value is used to represent the value of the node; and children is used to represent the list of child nodes of the node. The abstract syntax tree sets the node type according to the use of the node value in the query statement.

[0041] In one embodiment, the lexical analysis and grammatical analysis of the HQL statement according to the predefined lexical rules and grammatical rules to obtain the abstract syntax tree corresponding to the HQL statement can be replaced by the following steps:

[0042] The HQL statement is used as a program source code, and the program is compiled using predefined lexical rules, grammatical rules and semantic analysis rules to obtain an abstract syntax tree corresponding to the HQL statement.

[0043] Specifically, lexical analysis is the first stage of the program compilation process. The program source code is input as a character stream in sequence, and the read character sequence is scanned and analyzed, and finally converted into a word sequence output, where a "word" is the smallest unit that makes up the program. This process does not consider the relationship between words. Lexical analysis provides symbolic nodes for the generation of an abstract syntax tree.

[0044] Specifically, syntax analysis is the second stage of the program compilation process, which mainly performs logical analysis on the program. Its main task is to analyze the basic syntax structure of the program, such as methods, variables, expressions and other components of the program, based on the word sequence generated in the first stage and according to the grammatical rules of the analyzed language.

[0045] Specifically, semantic analysis is the third stage of the compilation process, which mainly conducts logical analysis of the program. The main analysis target is the source program without structural errors. The main analysis method is to check the context-related properties and types of such source programs, mainly to determine whether the source program is semantically accurate. For example, check whether the operation object of each algorithm in the source program conforms to the specification of the program language, and report an error if it does not conform.

[0046] In one embodiment, optimizing the logical execution plan by a logical optimizer includes:

[0047] The logic execution plan is logically optimized by a logic optimization algorithm of a logic optimizer, wherein the logic optimization algorithm includes predicate pushdown, column pruning, merging adjacent operations, equi-join optimization and sub-query optimization.

[0048] In one embodiment, optimizing the physical execution plan by a physical optimizer includes:

[0049] The physical execution plan is logically optimized by a physical optimization algorithm of a physical optimizer, wherein the physical optimization algorithm includes data localization, data block merging, data compression, query rewriting and parallel execution.

[0050] Specifically, logical optimization is to make logically equivalent changes to a logical operator tree (LogicalPlan Tree), and the final result is a better logical operator tree, while physical optimization is to convert a logical operator tree into a physical operator tree (PhysicalPlan Tree), which is the physical execution plan.

[0051] In one embodiment, the lexical analysis and grammatical analysis are performed on the HQL statement according to the predefined lexical rules and grammatical rules to obtain an abstract syntax tree (AST Tree) corresponding to the HQL statement, including:

[0052] Verify the legitimacy of the characters in the HQL statement by using the lexical rules of a lexer, decompose the verified character stream into multiple tokens, and pass the multiple tokens to a syntax analyzer, wherein each token includes information such as type, lexeme, literal, and line number;

[0053] The multiple tokens are parsed by a parser according to grammar rules to construct an abstract syntax tree.

[0054] Specifically, lexical analysis is the process of converting a sequence of characters into a sequence of words (Tokens) in computer science. The program or function that performs lexical analysis is called a lexical analyzer (Lexer for short), also called a scanner. Lexical analyzers generally exist in the form of functions for the syntax analyzer to call.

[0055] The scanner recognizes all possible character sequences that can be included in the tokens it can process. For example, the "integer" token can contain all numeric characters. In many cases, the first non-whitespace character can be used to infer the type of the token, and the subsequent characters can be processed one by one until a character that does not belong to the character set of the token type is encountered.

[0056] The job of a lexical analyzer is low-level analysis: converting characters or character sequences into tokens. When talking about lexical analysis, the terms "lexical token" (token for short), "pattern", and "token" are used to imply specific meanings.

[0057] During analysis, one can treat the lexical analyzer as part of the syntax analysis, or as an independent part of the compiler. In the former case, the lexical analyzer is constantly called by the syntax analyzer. Each time the lexical analyzer is called, it spells out a word from the character sequence of the source program and returns its token value to the syntax analyzer. The latter case is different. The lexical analyzer is not constantly called by the syntax analyzer, but scans all words at once to complete the compiler's independent task.

[0058] Specifically, a parser usually appears as a component of a compiler or interpreter. Its function is to perform syntax checking and construct a data structure composed of input words, generally a hierarchical data structure such as a syntax analysis tree or an abstract syntax tree. A parser usually uses an independent lexical analyzer to separate individual "words" from the input character stream and uses the word stream as its input. In actual development, a parser can be written manually or automatically generated using a tool.

[0059] The task of the parser is to determine whether and how the input symbol string (input text) can be derived from the starting symbol of the grammar. This can be done in two main ways:

[0060] Top-down parsing: According to the formal grammar rules, the top-down expansion of the grammar tree searches for the possible leftmost derivation of the input symbol string. Words are used in order from left to right.

[0061] Bottom-up parsing: The parser starts with an existing input string of symbols and tries to rewrite it according to the given formal grammar rules, eventually rewriting it into the starting symbol of the grammar.

[0062] S2. Decompose the execution plan into a map operation tree and a reduce operation tree according to the task flow stages, decompose the map operation tree and the reduce operation tree into operation steps, and use the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement.

[0063] In one embodiment, decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stages includes:

[0064] Identifying a task flow phase of the execution plan, wherein the task flow phase includes a mapping phase and a reduction phase;

[0065] A map operation tree is constructed based on the operation steps of the execution plan in the mapping phase, and a reduce operation tree is constructed based on the operation steps of the execution plan in the reduction phase.

[0066] Specifically, the optimized execution plan MapReduce task process includes two stages: the mapping stage (Map) and the reduction stage (Reduce).

[0067] In the Map stage, the input data is split into multiple independent data blocks, each of which is processed by a Map task. The Map function converts the input data blocks into key-value pairs. Suppose there is a text file that needs to count the number of occurrences of each word. The Map function will split each line of text into words and generate a key-value pair for each word, where the key is the word itself and the value is 1 (indicating that the word appears once).

[0068] The Map operation is highly parallel and can process multiple data blocks at the same time, which greatly improves the efficiency of data processing. The output of this stage is an intermediate result set, usually a set of key-value pairs, which will serve as the input of the Reduce stage.

[0069] The task of the Reduce stage is to summarize and process the intermediate results output by the Map stage. All intermediate results will be sorted and grouped according to the key, and then passed to the Reduce function. The Reduce function will process all values ​​with the same key and generate the final output result.

[0070] As in the word counting example above, the Reduce function receives a set of key-value pairs, where the key is a word and the value is the number of times the word appears in each data block. The Reduce function adds these values ​​to get the total number of times the word appears in the entire data set.

[0071] In one embodiment, the map operation tree includes the following step types: Table Scan full table scan, SelectOperator selects operators, Group By Operator groups according to operators, and Reduce Output Operator passes operators to the Reduce operation tree. A map operation tree can contain multiple parallel operation steps of the same step type.

[0072] In one embodiment, the reduce operation tree includes the following step types: Group By Operator operator grouping, Select Operator selects the grouping operator, and File Output Operator writes the grouping operator into a file output stream. A reduce operation tree may contain multiple parallel operation steps of the same step type.

[0073] In one embodiment, the step of using the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement includes:

[0074] Each operation step obtained after decomposition is used as a syntax tree node, and the corresponding syntax tree nodes are connected according to the order of the operation steps in the task flow of the execution plan, and a syntax tree with the starting operation step of the task flow as the root node is generated as the performance analysis syntax tree corresponding to the HQL statement

[0075] S3. According to the query object contained in the operation steps in the performance analysis syntax tree, the database is queried to obtain the relevant database table of the HQL statement, the data of the relevant database table and the data of the operation steps of the performance analysis syntax tree are calculated, and the values ​​of multiple preset data indicators of the HQL statement are obtained according to the calculation results.

[0076] In one embodiment, querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain a database table related to the HQL statement includes:

[0077] Obtaining query operation steps related to database table query in the performance analysis syntax tree, and extracting corresponding query objects from the query operation steps;

[0078] A database table related to the HQL statement is queried from the database based on the query object.

[0079] In one embodiment, the preset data indicators include the size of the relevant database table and the amount of table data, the number of maps, the number of reduces, the number of jobs, the associated primary key distribution data and the execution time.

[0080] The map number refers to the number of tasks in the Map phase in big data processing, especially in the MapReduce programming model. In the MapReduce computing model, data processing is divided into two phases: Map and Reduce. The main task of the Map phase is to split the large data set into small blocks, and multiple Map tasks process these small blocks in parallel. The number of these Map tasks is the so-called map number. The number of Map tasks is usually determined by the size of the input data. Specifically, it is equal to the total number of blocks of all input files. Each Map task processes a part of the input data and generates a series of key-value pairs, which are then passed to the Reduce phase for processing.

[0081] The reduce number is associated with the "reduce" operation in the reduce phase of a Hive query and is the number of tasks in the Reduce phase of the MapReduce programming model. In programming, "reduce" refers to an operation that reduces a collection of data (such as an array) to a single value by applying a function that accepts two parameters: an accumulated value (the return value of the previous call) and the current value (the element taken from the collection). In Hive, the reduce phase is part of the MapReduce programming model, where the "reduce" operation involves merging multiple blocks of input data into one or more output records.

[0082] The number of jobs is the number of tasks generated by the decomposition and transformation of the HQL statement in the performance analysis syntax tree. In Hive, an HQL (Hive Query Language) statement will be converted into multiple MapReduce tasks, each of which is called a job. Each job is responsible for processing a part of the data and generating intermediate results for the next job to continue processing. The HQL statement is decomposed into multiple stages, each of which performs specific operations, such as filtering, aggregation, and connection. Each stage can be regarded as an independent job.

[0083] In one embodiment, the data of the relevant database table and the data of the operation steps of the performance analysis syntax tree are calculated, and the values ​​of multiple preset data indicators of the HQL statement are obtained according to the calculation results, including:

[0084] Calculate the size and amount of table data of each table in the database table related to the HQL statement;

[0085] Calculate the number of maps, the number of reducers, and the number of jobs according to the operation steps of the performance analysis syntax tree, wherein the number of maps is the number of tasks in the mapping phase of the operation step, the number of reducers is the number of tasks in the reduction phase of the operation step, and the number of jobs is the number of task flows generated by decomposing and transforming the HQL statement in the performance analysis syntax tree;

[0086] Query the primary key index of each table in the database table related to the HQL statement, obtain the query field of the HQL statement, calculate the proportion of the number of query fields matching the primary key index in the total number of query fields to obtain the associated primary key distribution data;

[0087] The execution time of each operation step in the performance analysis syntax tree is counted.

[0088] In one embodiment, after counting the execution time of each operation step in the performance analysis syntax tree, the method further includes:

[0089] Obtain a preset time consumption threshold, compare the execution time of each operation step with the time consumption threshold, and generate an early warning notification based on the information of the operation step whose execution time is greater than the time consumption threshold and send it to the server that executes the HQL statement.

[0090] S4. Obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0091] In one embodiment, the step of calculating the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm includes:

[0092] Obtain a numerical score table and an indicator weight corresponding to each preset data indicator, and obtain a score corresponding to each preset data indicator according to the numerical value of the preset data indicator and the corresponding numerical score table;

[0093] The data indicator scoring algorithm is used to calculate the score of the HQL statement according to the scores corresponding to the respective preset data indicators and the indicator weights.

[0094] In one embodiment, generating a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement includes:

[0095] According to the value of the preset data indicator and the score of the HQL statement, the object naming, index proportion, modifiability, standardization and accuracy of the HQL statement are analyzed, and a corresponding test report is generated.

[0096] It can be seen from the above embodiments that the embodiments of the present invention parse HQL statements to generate a logical execution plan and a physical execution plan, decompose the execution steps of the logical execution plan and the physical execution plan to generate a performance analysis syntax tree, query the relevant database tables of the HQL statements based on the performance analysis syntax tree and count the data indicators of the HQL statements, calculate the quality score of the HQL statements according to the relevant database tables, data indicators and pre-set scoring rules, and realize automatic evaluation of the quality of HQL statements with unified rules.

[0097] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0098] In one embodiment, the present invention provides an HQL statement detection device, which corresponds one-to-one to the HQL statement detection method in the above embodiment. Figure 3 As shown, the HQL statement detection device 100 includes a statement parsing module 110, a step decomposition module 120, a query calculation module 130 and a statement scoring module 140. The detailed description of each functional module is as follows:

[0099] The statement parsing module 110 is used to obtain the HQL statement to be detected in the database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree;

[0100] The step decomposition module 120 is used to decompose the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decompose the map operation tree and the reduce operation tree into operation steps, and use the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement;

[0101] The query calculation module 130 is used to query the database according to the query object included in the operation step in the performance analysis syntax tree to obtain the relevant database table of the HQL statement, calculate the data of the relevant database table and the data of the operation step of the performance analysis syntax tree, and obtain the values ​​of multiple preset data indicators of the HQL statement according to the calculation results;

[0102] The statement scoring module 140 is used to obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0103] In one embodiment, when the statement parsing module 110 parses the HQL statement to generate an abstract syntax tree, optimizes the abstract syntax tree, and generates an execution plan according to the optimized syntax tree, it is used to:

[0104] Performing lexical analysis and grammatical analysis on the HQL statement according to predefined lexical rules and grammatical rules to obtain an abstract syntax tree corresponding to the HQL statement;

[0105] Traversing the abstract syntax tree to abstract query blocks;

[0106] Traversing the query block, converting the query block into a logical execution plan, and optimizing the logical execution plan through a logical optimizer;

[0107] Traversing the optimized logical execution plan, converting the optimized logical execution plan into a physical execution plan, and optimizing the physical execution plan through a physical optimizer;

[0108] An execution plan corresponding to the HQL statement is generated based on the optimized logical execution plan and the optimized physical execution plan.

[0109] In one embodiment, when the statement parsing module 110 performs lexical analysis and grammatical analysis on the HQL statement according to predefined lexical rules and grammatical rules to obtain an abstract syntax tree corresponding to the HQL statement, it is used to:

[0110] Verify the legitimacy of characters in the HQL statement by using the lexical rules of the lexical analyzer, decompose the verified character stream into multiple tokens, and pass the multiple tokens to the syntax analyzer, wherein each token includes a type, a lexical rule, a literal, and a line number;

[0111] The multiple tokens are parsed by a syntax analyzer according to syntax rules to construct an abstract syntax tree.

[0112] In one embodiment, when the step decomposition module 120 decomposes the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, it is used to:

[0113] Identifying a task flow phase of the execution plan, wherein the task flow phase includes a mapping phase and a reduction phase;

[0114] A map operation tree is constructed based on the operation steps of the execution plan in the mapping phase, and a reduce operation tree is constructed based on the operation steps of the execution plan in the reduction phase.

[0115] In one embodiment, when querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain the relevant database table of the HQL statement, the query calculation module 130 is used to:

[0116] Obtaining query operation steps related to database table query in the performance analysis syntax tree, and extracting corresponding query objects from the query operation steps;

[0117] A database table related to the HQL statement is queried from the database based on the query object.

[0118] In one embodiment, when the query calculation module 130 calculates the data of the relevant database table and the data of the operation steps of the performance analysis syntax tree and obtains the values ​​of multiple preset data indicators of the HQL statement according to the calculation results, it is used to:

[0119] Calculate the size and amount of table data of each table in the database table related to the HQL statement;

[0120] Calculate the number of maps, the number of reducers, and the number of jobs according to the operation steps of the performance analysis syntax tree, wherein the number of maps is the number of tasks in the mapping phase of the operation step, the number of reducers is the number of tasks in the reduction phase of the operation step, and the number of jobs is the number of task flows generated by decomposing and transforming the HQL statement in the performance analysis syntax tree;

[0121] Query the primary key index of each table in the database table related to the HQL statement, obtain the query field of the HQL statement, calculate the proportion of the number of query fields matching the primary key index in the total number of query fields to obtain the associated primary key distribution data;

[0122] The execution time of each operation step in the performance analysis syntax tree is counted.

[0123] In one embodiment, when calculating the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, the statement scoring module 140 is used to:

[0124] Obtain a numerical score table and an indicator weight corresponding to each preset data indicator, and obtain a score corresponding to each preset data indicator according to the numerical value of the preset data indicator and the corresponding numerical score table;

[0125] The data indicator scoring algorithm is used to calculate the score of the HQL statement according to the scores corresponding to the respective preset data indicators and the indicator weights.

[0126] The present invention provides an HQL statement detection device, which generates a logical execution plan and a physical execution plan by parsing HQL statements, decomposes the execution steps of the logical execution plan and the physical execution plan to generate a performance analysis syntax tree, queries the relevant database table of the HQL statement based on the performance analysis syntax tree and counts the data indicators of the HQL statement, calculates the quality score of the HQL statement according to the relevant database table, the data indicators and the pre-set scoring rules, and realizes automatic evaluation of the quality of the HQL statement with unified rules.

[0127] For the specific definition of the HQL statement detection device, please refer to the definition of the HQL statement detection method above, which will not be repeated here. Each module in the above-mentioned HQL statement detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0128] like Figure 4 , which is a schematic diagram of an embodiment of an electronic device of the present invention. The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The electronic device 1 can be a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets.

[0129] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which can be interconnected for communication via a system bus. The memory 11 stores an HQL statement detection program 10, and the HQL statement detection program 10 can be executed by the processor 12. Figure 3 Only the electronic device 1 having the components 11-13 and the HQL statement detection program 10 is shown. It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] Wherein, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium can be volatile or non-volatile. Specifically, the readable storage medium can be a storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium can be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the storage medium can also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 mainly includes a program storage area and a data storage area, wherein the program storage area is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the HQL statement detection program 10 in an embodiment of the present invention; the data storage area can store data created according to the use of the blockchain node, such as various types of data that have been output or are to be output.

[0131] In some embodiments, the processor 12 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 12 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect each component of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing resource dynamic scheduling programs, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data. The processor 12 is generally used to control the overall operation of the electronic device 1, such as executing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the HQL statement detection program 10, etc.

[0132] The network interface 13 may include a wireless network interface or a wired network interface.

[0133] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (Organic Light-Emitting Diode, OLED) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0134] In one embodiment of the present invention, when the HQL statement detection program 10 is executed by the processor 12, the following HQL statement detection method is implemented, including steps S1-S4:

[0135] S1. Obtain an HQL statement to be detected in a database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree;

[0136] S2, decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decomposing the map operation tree and the reduce operation tree into operation steps, and using the operation steps obtained after the decomposition as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement;

[0137] S3, querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain a relevant database table of the HQL statement, calculating the data in the relevant database table and the data in the operation step of the performance analysis syntax tree, and obtaining the values ​​of multiple preset data indicators of the HQL statement according to the calculation results;

[0138] S4. Obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0139] The specific operation steps implemented by the above steps S1-S4 are substantially the same as steps S1-S4 of the first embodiment of the HQL statement detection method of the present invention, and will not be repeated here.

[0140] In other embodiments, the HQL statement detection program 10 can also be divided into one or more modules, one or more modules are stored in the memory 11, and are executed by one or more processors (processor 12 in this embodiment) to complete the present invention. The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, and is used to describe the execution process of the HQL statement detection program 10 in the electronic device 1.

[0141] In addition, the embodiment of the present invention further proposes a computer-readable storage medium, which can be volatile or non-volatile. Specifically, the computer-readable storage medium can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium stores an HQL statement detection program, and when the computer program is executed by a processor, the following steps are implemented:

[0142] Obtaining an HQL statement to be detected in a database, parsing the HQL statement to generate an abstract syntax tree, optimizing the abstract syntax tree, and generating an execution plan according to the optimized syntax tree;

[0143] Decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decomposing the map operation tree and the reduce operation tree into operation steps, and using the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement;

[0144] querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain a related database table of the HQL statement, calculating the data in the related database table and the data in the operation step of the performance analysis syntax tree, and obtaining the values ​​of multiple preset data indicators of the HQL statement according to the calculation results;

[0145] Obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

[0146] The specific software function implementation of the computer-readable storage medium of the present invention is substantially the same as the specific software implementation in the above-mentioned electronic device 1, and will not be described in detail here.

[0147] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0148] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the method described in each embodiment of the present invention.

[0150] The above are only preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention and the drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention. It should be noted that if software tools or components other than our company appear in the embodiments of this application, they are only used for example introduction and do not represent actual use.

Claims

1. A HQL statement detection method, characterized in that: The method comprises: Obtaining an HQL statement to be detected in a database, parsing the HQL statement to generate an abstract syntax tree, optimizing the abstract syntax tree, and generating an execution plan according to the optimized syntax tree; Decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decomposing the map operation tree and the reduce operation tree into operation steps, and using the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement; querying the database according to the query object included in the operation step in the performance analysis syntax tree to obtain a related database table of the HQL statement, calculating the data in the related database table and the data in the operation step of the performance analysis syntax tree, and obtaining the values ​​of multiple preset data indicators of the HQL statement according to the calculation results; Obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

2. The HQL statement detection method according to claim 1, characterized in that: The step of parsing the HQL statement to generate an abstract syntax tree, optimizing the abstract syntax tree, and generating an execution plan according to the optimized syntax tree includes: Performing lexical analysis and grammatical analysis on the HQL statement according to predefined lexical rules and grammatical rules to obtain an abstract syntax tree corresponding to the HQL statement; Traversing the abstract syntax tree to abstract query blocks; Traversing the query block, converting the query block into a logical execution plan, and optimizing the logical execution plan through a logical optimizer; Traversing the optimized logical execution plan, converting the optimized logical execution plan into a physical execution plan, and optimizing the physical execution plan through a physical optimizer; An execution plan corresponding to the HQL statement is generated based on the optimized logical execution plan and the optimized physical execution plan.

3. The HQL statement detection method according to claim 2, characterized in that: The lexical analysis and grammatical analysis of the HQL statement according to the predefined lexical rules and grammatical rules to obtain an abstract syntax tree corresponding to the HQL statement includes: Verify the legitimacy of characters in the HQL statement by using the lexical rules of the lexical analyzer, decompose the verified character stream into multiple tokens, and pass the multiple tokens to the syntax analyzer, wherein each token includes a type, a lexical rule, a literal, and a line number; The multiple tokens are parsed by a syntax analyzer according to syntax rules to construct an abstract syntax tree.

4. The HQL statement detection method according to claim 1, characterized in that: Decomposing the execution plan into a map operation tree and a reduce operation tree according to the task flow stages includes: Identifying a task flow phase of the execution plan, wherein the task flow phase includes a mapping phase and a reduction phase; A map operation tree is constructed based on the operation steps of the execution plan in the mapping phase, and a reduce operation tree is constructed based on the operation steps of the execution plan in the reduction phase.

5. The HQL statement detection method according to claim 1, characterized in that: The querying the database according to the query object included in the operation steps in the performance analysis syntax tree to obtain the relevant database table of the HQL statement includes: Obtaining query operation steps related to database table query in the performance analysis syntax tree, and extracting corresponding query objects from the query operation steps; A database table related to the HQL statement is queried from the database based on the query object.

6. The HQL statement detection method according to claim 1, characterized in that: The preset data indicators include the size of the relevant database table and the amount of table data, the number of maps, the number of reduces, the number of jobs, the associated primary key distribution data and the execution time; The data of the relevant database table and the data of the operation steps of the performance analysis syntax tree are calculated, and the values ​​of multiple preset data indicators of the HQL statement are obtained according to the calculation results, including: Calculate the size and amount of table data of each table in the database table related to the HQL statement; Calculate the number of maps, the number of reducers, and the number of jobs according to the operation steps of the performance analysis syntax tree, wherein the number of maps is the number of tasks in the mapping phase of the operation step, the number of reducers is the number of tasks in the reduction phase of the operation step, and the number of jobs is the number of task flows generated by decomposing and transforming the HQL statement in the performance analysis syntax tree; Query the primary key index of each table in the database table related to the HQL statement, obtain the query field of the HQL statement, calculate the proportion of the number of query fields matching the primary key index in the total number of query fields to obtain the associated primary key distribution data; The execution time of each operation step in the performance analysis syntax tree is counted.

7. The HQL statement detection method according to claim 1, characterized in that: The calculating the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm includes: Obtain a numerical score table and an indicator weight corresponding to each preset data indicator, and obtain a score corresponding to each preset data indicator according to the numerical value of the preset data indicator and the corresponding numerical score table; The data indicator scoring algorithm is used to calculate the score of the HQL statement according to the scores corresponding to the respective preset data indicators and the indicator weights.

8. A HQL statement detection device, characterized in that: The HQL statement detection device comprises: A statement parsing module is used to obtain the HQL statement to be detected in the database, parse the HQL statement to generate an abstract syntax tree, optimize the abstract syntax tree, and generate an execution plan according to the optimized syntax tree; A step decomposition module is used to decompose the execution plan into a map operation tree and a reduce operation tree according to the task flow stage, decompose the map operation tree and the reduce operation tree into operation steps, and use the decomposed operation steps as syntax tree nodes to generate a performance analysis syntax tree corresponding to the HQL statement; A query calculation module, used to query the database according to the query object included in the operation step in the performance analysis syntax tree to obtain the relevant database table of the HQL statement, calculate the data of the relevant database table and the data of the operation step of the performance analysis syntax tree, and obtain the values ​​of multiple preset data indicators of the HQL statement according to the calculation results; A statement scoring module is used to obtain a preset data indicator scoring algorithm, calculate the score of the HQL statement according to the value of the preset data indicator and the data indicator scoring algorithm, and generate a detection report of the HQL statement according to the value of the preset data indicator and the score of the HQL statement.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the HQL statement detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the HQL statement detection method according to any one of claims 1 to 7 is implemented.

Images