A method for parsing Flink SQL to obtain field attributes and the Calcite parser

Abstract

This invention discloses a method for obtaining field attributes from Flink SQL based on Apache Calcite and a Calcite parser. The invention involves creating a Calcite parser and using it to parse Flink SQL to obtain all SqlNode array objects. Then, by sequentially filtering each SqlNode array object by field column objects and field type objects, the field attributes of all SqlNode objects are finally obtained. Practice has shown that the method described in this invention can significantly improve the accuracy and execution efficiency of obtaining field attributes, reduce latency, and enhance system robustness.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for parsing Flink SQL to obtain field attributes and a Calcite parser. Background Technology

[0002] Currently, regular expressions are commonly used to distinguish between DML and DDL statements in Flink SQL, and regular expressions and string splitting are used to extract field names and field types from DDL statements. However, regular expressions have catastrophic backtracking. Improper use can not only lead to low matching efficiency, but may even block threads. In addition, field names may contain special characters, which may cause errors in regular expression matching and string splitting. Summary of the Invention

[0003] This invention provides a method for parsing Flink SQL to obtain field attributes based on Apache Calcite and a Calcite parser, to solve the problems of non-retrospective and inefficient matching of obtaining field names and field types in Flink SQL statements through regular expressions in the prior art.

[0004] In a first aspect, the present invention provides a method for obtaining field attributes by parsing Flink SQL based on Apache Calcite. The method includes: parsing a preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement; and sequentially judging the field column object and field type object of each SqlNode array object to determine the field attributes of all SqlNode objects.

[0005] Optionally, parsing a preset Flink SQL statement includes: creating a Calcite parser and parsing the Flink SQL statement using the Calcite parser.

[0006] Optionally, the method further includes setting the comment characters and case strategy of the Calcite parser.

[0007] Optionally, for each of the SqlNode array objects, the field column object and field type object are judged sequentially to determine the field attributes of all SqlNode objects, including: using the Calcite parser to judge the field column object and field type object for each of the SqlNode array objects sequentially to determine the field attributes of all SqlNode objects.

[0008] Optionally, for each of the SqlNode array objects, the field column object and field type object are determined sequentially to determine the field attributes of all SqlNode objects, including:

[0009] For each of the SqlNode array objects, the following processing is performed:

[0010] The filter type is a SqlRegularColumn field column object;

[0011] Further filter the SqlNode objects that are of type SqlRegularColumn to those of type SqlBasicTypeNameSpec to determine the field properties of all SqlNode objects.

[0012] Optionally, the field column objects of the filtering type are SqlRegularColumn, including: obtaining the table name defined by the DDL statement of the current SqlNode object through the getTableName() method, traversing all SqlNode objects, obtaining the field column objects of each SqlNode object through getColumnList(), traversing all field column objects, and obtaining the field name and field type of the field column objects;

[0013] Further filtering from the SqlNode objects of type SqlRegularColumn includes field type objects of type SqlBasicTypeNameSpec, including:

[0014] From the SqlNode objects that are filtered to be of type SqlRegularColumn, find all objects whose field type description objects are of type SqlBasicTypeNameSpec.

[0015] Optionally, the method further includes: determining the actual precision or number of bits of the SqlBasicTypeNameSpec type object based on the field attributes of the filtered SqlBasicTypeNameSpec type object, so as to ensure the correctness and completeness of the field type.

[0016] Optionally, determining the actual precision or number of digits of an object of type SqlBasicTypeNameSpec includes: for each object of type SqlBasicTypeNameSpec, obtaining the precision of its field type using the getPrecision() method; if the precision of the field is greater than -1 and not equal to Integer.MAX_VALUE, then the precision of the object of type SqlBasicTypeNameSpec is valid precision; for each object of type SqlBasicTypeNameSpec, obtaining the number of decimal places of its field type using the getScale() method; if the number of decimal places of the field is greater than -1, then the number of decimal places of the object of type SqlBasicTypeNameSpec is valid decimal places.

[0017] Secondly, the present invention provides a Calcite parser, characterized in that the Calcite parser comprises: a parsing unit, used to parse a preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement; and a processing unit, used to sequentially determine the field column object and field type object of each SqlNode array object to determine the field attributes of all SqlNode objects.

[0018] Thirdly, the present invention provides a computer-readable storage medium storing a computer program of signal mapping, which, when executed by at least one processor, implements any of the methods described above for parsing Flink SQL to obtain field attributes based on Apache Calcite.

[0019] The beneficial effects of the present invention are as follows:

[0020] This invention creates a Calcite parser and uses it to parse Flink SQL to obtain all SqlNode array objects. Then, by filtering each SqlNode array object sequentially through field column objects and field type objects, the field attributes of all SqlNode objects are finally obtained. Practice has proven that the method described in this invention can significantly improve the accuracy and execution efficiency of obtaining field attributes, reduce latency, and enhance the robustness of the system.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0023] Figure 1 This is a flowchart illustrating a method for obtaining field attributes by parsing Flink SQL based on Apache Calcite, provided in the first embodiment of the present invention.

[0024] Figure 2 This is a flowchart illustrating another method for obtaining field attributes by parsing Flink SQL based on Apache Calcite, provided in the first embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the Calcite parser provided in the first embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention.

[0027] Existing methods that use regular expressions to distinguish between DML and DDL statements in Flink SQL, and to extract field names and types from DDL statements using regular expressions and string splitting, suffer from catastrophic backtracking. Improper use can lead to inefficient matching and even thread blocking. Furthermore, special characters in field names can cause errors in regular expression matching and string splitting, and the diverse range of field types and their precision means that regular expression matching and string splitting may not cover all potential issues. The first embodiment of this invention provides a method for parsing Flink SQL and obtaining field attributes based on Apache Calcite. See [link to Apache Calcite method]. Figure 1 The method includes:

[0028] S101. Parse the preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement;

[0029] Specifically, in this embodiment of the invention, a Calcite parser is first created, and then the Flink SQL statement is parsed using the Calcite parser.

[0030] In other words, in this embodiment of the invention, the Flink SQL is parsed using the Calcite parser to obtain field attributes.

[0031] It should be noted that after creating the Calcite parser in this embodiment of the invention, it is necessary to set its comment characters and case sensitivity to ensure the normal use of the Calcite parser. Of course, in specific implementations, those skilled in the art can also make other settings for the Calcite parser according to actual needs, and this invention does not specifically limit this.

[0032] S102. For each of the SqlNode array objects, the field column object and field type object are judged in turn to determine the field attributes of all SqlNode objects.

[0033] Specifically, in this embodiment of the invention, after creating a Calcite parser, the Calcite parser is used to sequentially determine the field column object and field type object of each SqlNode array object to determine the field attributes of all SqlNode objects, thereby achieving accurate acquisition of field attributes. Furthermore, since this embodiment of the invention uses the Calcite parser to specifically perform all parsing operations, it can achieve higher efficiency, reduce latency, and ultimately enhance the robustness of the system.

[0034] In specific implementation, embodiments of the present invention use the Calcite parser to perform the following processing on each SqlNode array object:

[0035] The filter type is a SqlRegularColumn field column object;

[0036] Further filter the SqlNode objects that are of type SqlRegularColumn to those of type SqlBasicTypeNameSpec to determine the field properties of all SqlNode objects.

[0037] In simple terms, the embodiment of this invention iterates through all SqlNode array objects, that is, iterates through all SqlCreateTable type objects to determine SqlRegularColumn type objects, then iterates through all SqlRegularColumn type objects to determine SqlBasicTypeNameSpec type objects, and finally determines the precision or number of bits of the SqlBasicTypeNameSpec type object to ensure the correctness and integrity of the field type.

[0038] Specifically, in this embodiment of the invention, the table name defined by the DDL statement of the current SqlNode object is obtained by the getTableName() method, all SqlNode objects are traversed, the field column objects of each SqlNode object are obtained by the getColumnList() method, all field column objects are traversed, and the field name and field type of the field column objects are obtained.

[0039] Further filtering from the SqlNode objects of type SqlRegularColumn includes field type objects of type SqlBasicTypeNameSpec, including:

[0040] From the SqlNode objects that are filtered to be of type SqlRegularColumn, find all objects whose field type description objects are of type SqlBasicTypeNameSpec.

[0041] Furthermore, after filtering and obtaining the field attributes of the array of type SqlBasicTypeNameSpec, the method described in this embodiment of the invention further includes determining the actual precision or number of bits of the object of type SqlBasicTypeNameSpec based on the field attributes of the object of type SqlBasicTypeNameSpec obtained through filtering, so as to ensure the correctness and completeness of the field type.

[0042] Specifically, in this embodiment of the invention, for each object of type SqlBasicTypeNameSpec, the precision of its field type is obtained through the getPrecision() method. If the precision of the field is greater than -1 and not equal to Integer.MAX_VALUE, then the precision of the SqlBasicTypeNameSpec object is valid. For each object of type SqlBasicTypeNameSpec, the number of decimal places of its field type is obtained through the getScale() method. If the number of decimal places of the field is greater than -1, then the number of decimal places of the SqlBasicTypeNameSpec object is valid.

[0043] In other words, the embodiments of the present invention parse and obtain the table name and its field attributes in the DDL statement. The field attributes include the field name, field type, field precision, number of decimal places, etc., and then perform corresponding processing on the field attributes to ultimately achieve the purpose of system optimization.

[0044] In short, this invention addresses the problem of inaccurate field attribute retrieval in existing Flink SQL parsing methods. It uses Calcite to parse Flink SQL, converting the SQL into objects such as SqlCreate and SqlInsert. It iterates through all SqlCreate objects, first obtaining the corresponding table name using the getTableName() method. Then, it obtains an array of field columns using the getColumnList() method, and iterates through this array to retrieve the corresponding field column objects. Next, it determines the type of the field column object: if it's of type SqlRegularColumn, the field is a regular field; if it's of type SqlComputedColumn, the field type is a computed column. For SqlRegularColumn fields, the field name is obtained using the getName() method, the field type description object SqlTypeNameSpec is obtained using the getTypeNameSpec() method, and finally, the precision and decimal places are obtained using the getPrecision() and getScale() methods to determine if the field's precision and decimal places are valid, etc.

[0045] For example, in specific implementation, the method described in the embodiments of the present invention may specifically include the following steps:

[0046] S1. Parse the Flink SQL and obtain a SqlNode array object;

[0047] S2. Iterate through the SqlNode array object and get each SqlNode in it;

[0048] S3. Determine if the SqlNode object is of type SqlCreateTable;

[0049] S4. Get the table name corresponding to the SqlNode object;

[0050] S5. Obtain the array of field columns from the SqlNode object;

[0051] S6. Iterate through the array of field columns and get each field column object;

[0052] S7. Determine if the type of the column object is SqlRegularColumn;

[0053] S8. Retrieve the field name and field type from the field column object;

[0054] S9. Determine if the field type in the field column object is SqlBasicTypeNameSpec;

[0055] S10. Retrieve field precision from field type object;

[0056] S11. Determine if the field precision is valid;

[0057] S12. Get the number of decimal places from a field type object;

[0058] S13. Determine if the number of decimal places is valid.

[0059] The following will combine Figure 2 The method described in the embodiments of the present invention will be explained and illustrated in detail through a specific example:

[0060] Create a Calcite SQL parser and set comment characters and case sensitivity.

[0061] The parser's parseStmtList() method is called to parse the Flink SQL, resulting in a parsed SqlNode array object;

[0062] Iterate through all SqlNode arrays. If a next one exists, continue parsing; otherwise, the parsing is complete, and proceed to the next step.

[0063] Determine if the type of SqlNode is SqlCreateTable. If it is, record it; otherwise, continue traversing the other SqlNode arrays until all SqlNode arrays have been traversed.

[0064] Get the current SqlCreateTable object, and use the getTableName() method to get the table name defined in the current DDL statement;

[0065] Get the column list object using getColumnList(), iterate through the column list until the data value has been traversed, and then continue iterating through the SqlNode array;

[0066] Iterate through all the array of field columns and determine whether the object type of the field column is SqlRegularColumn;

[0067] Get the current SqlRegularColumn object and retrieve the field name using the getName() method;

[0068] The field type description object can be obtained through the getType() method;

[0069] Determine if the field type description object type is SqlBasicTypeNameSpec. If so, record it and continue until the array is traversed.

[0070] Get the current SqlBasicTypeNameSpec object, and use the getPrecision() method to get the precision of this field type. If the precision is greater than -1 and not equal to Integer.MAX_VALUE, it means the precision is valid. Then use the getScale() method to get the number of decimal places for this field type. If the number of decimal places is greater than -1, it means the number of decimal places is valid.

[0071] In summary, the embodiments of the present invention use program settings to traverse all SqlNode array objects of type SqlCreateTable, SqlRegularColumn, and SqlBasicTypeNameSpec, thereby quickly and easily filtering out the field attributes of the array of type SqlBasicTypeNameSpec.

[0072] The second embodiment of the present invention provides a Calcite parser, see [link to documentation]. Figure 3 The Calcite parser includes:

[0073] The parsing unit is used to parse a preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement;

[0074] The processing unit is used to sequentially determine the field column object and field type object for each of the SqlNode array objects in order to determine the field attributes of all SqlNode objects.

[0075] In other words, in this embodiment of the invention, the Flink SQL statement is parsed by the parsing unit, and the field column object and field type object are determined sequentially for each SqlNode array object by the processing unit to determine the field attributes of all SqlNode objects.

[0076] In specific implementation, the processing unit of this embodiment of the invention performs the following processing on each of the SqlNode array objects: First, it filters field column objects of type SqlRegularColumn, and then further filters field type objects of type SqlBasicTypeNameSpec from the filtered SqlNode objects of type SqlRegularColumn, so as to determine the field attributes of all SqlNode objects.

[0077] Specifically, the processing unit of this embodiment of the invention obtains the table name defined by the DDL statement of the current SqlNode object through the getTableName() method, traverses all SqlNode objects, obtains the field column objects of each SqlNode object through getColumnList(), traverses all field column objects, obtains the field name and field type of the field column objects, and then finds all objects with the field type description object type of SqlBasicTypeNameSpec from the SqlNode objects that are filtered out as having the type SqlRegularColumn.

[0078] Furthermore, in specific implementation, the processing unit in this embodiment of the invention is also used to determine the actual precision or number of digits of the SqlBasicTypeNameSpec type object based on the field attributes of the filtered SqlBasicTypeNameSpec type objects, in order to ensure the correctness and completeness of the field type. That is, for each SqlBasicTypeNameSpec type object, the precision of its field type is obtained through the getPrecision() method. If the precision of the field is greater than -1 and not equal to Integer.MAX_VALUE, then the precision of the SqlBasicTypeNameSpec type object is valid precision; for each SqlBasicTypeNameSpec type object, the number of decimal places of its field type is obtained through the getScale() method. If the number of decimal places of the field is greater than -1, then the number of decimal places of the SqlBasicTypeNameSpec type object is valid decimal places.

[0079] In general, the embodiments of the present invention parse and obtain the table name and its field attributes in the DDL statement. The field attributes include the field name, field type, field precision, number of decimal places, etc., and then perform corresponding processing on the field attributes to ultimately achieve the goal of system optimization.

[0080] The Calcite parser described in this embodiment of the invention can be understood by referring to the first embodiment of the invention, and will not be described in detail here.

[0081] A third embodiment of the present invention provides a computer-readable storage medium storing a computer program for signal mapping. When executed by at least one processor, the computer program implements any of the methods described in the first embodiment of the present invention for parsing Flink SQL and obtaining field attributes based on Apache Calcite.

[0082] The relevant content of the embodiments of the present invention can be understood by referring to the first embodiment of the present invention, and will not be discussed in detail here.

[0083] Although preferred embodiments of the invention have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible, and therefore the scope of the invention should not be limited to the embodiments described above.

Claims

1. A method for parsing Flink SQL to obtain field attributes, characterized in that, include: Parse the preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement; For each SqlNode array object, the field column object and field type object are judged sequentially to determine the field attributes of all SqlNode objects; The process involves sequentially determining the field column object and field type object for each SqlNode array object to ascertain the field attributes of all SqlNode objects, including: For each of the SqlNode array objects, the following processing is performed: The filter type is a SqlRegularColumn field column object; Further filter the SqlNode objects that are of type SqlRegularColumn to those of type SqlBasicTypeNameSpec to determine the field properties of all SqlNode objects.

2. The method according to claim 1, characterized in that, Parse the default Flink SQL statement, including: Create a Calcite parser to parse Flink SQL statements.

3. The method according to claim 2, characterized in that, The method further includes: Configure the comment symbols and case strategy for the Calcite parser.

4. The method according to claim 2, characterized in that, For each of the SqlNode array objects, the field column object and field type object are checked sequentially to determine the field attributes of all SqlNode objects, including: The Calcite parser sequentially checks the field column object and field type object of each SqlNode array object to determine the field attributes of all SqlNode objects.

5. The method according to claim 1, characterized in that, The filter type is a SqlRegularColumn field column object, including: The getTableName() method is used to obtain the table name defined by the DDL statement of the current SqlNode object. All SqlNode objects are traversed. The getColumnList() method is used to obtain the field column objects of each SqlNode object. All field column objects are traversed to obtain the field name and field type of the field column objects. Further filtering from the SqlNode objects of type SqlRegularColumn includes field type objects of type SqlBasicTypeNameSpec, including: From the SqlNode objects that are filtered to be of type SqlRegularColumn, find all objects whose field type description objects are of type SqlBasicTypeNameSpec.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: determining the actual precision or number of bits of the SqlBasicTypeNameSpec type object based on the field attributes of the filtered SqlBasicTypeNameSpec type object, so as to ensure the correctness and completeness of the field type.

7. The method according to claim 6, characterized in that, Determine the actual precision or number of bits for an object of type SqlBasicTypeNameSpec, including: For each object of type SqlBasicTypeNameSpec, the precision of its field type is obtained by using the getPrecision() method. If the precision of the field type is greater than -1 and not equal to Integer.MAX_VALUE, then the precision of the object of type SqlBasicTypeNameSpec is the valid precision. For each object of type SqlBasicTypeNameSpec, the getScale() method is used to obtain the number of decimal places of its field type. If the number of decimal places of the field type is greater than -1, then the number of decimal places of the SqlBasicTypeNameSpec object is valid.

8. A Calcite parser, characterized in that, The Calcite parser includes: The parsing unit is used to parse a preset Flink SQL statement to obtain all SqlNode array objects in the preset Flink SQL statement; The processing unit is used to sequentially determine the field column object and field type object for each of the SqlNode array objects in order to determine the field attributes of all SqlNode objects; The processing unit is specifically configured to perform the following processing on each of the SqlNode array objects: The filter type is a SqlRegularColumn field column object; Further filter the SqlNode objects that are of type SqlRegularColumn to those of type SqlBasicTypeNameSpec to determine the field properties of all SqlNode objects.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that maps signals, which, when executed by at least one processor, implements the method for parsing Flink SQL to obtain field attributes as described in any one of claims 1-7.

Images