Data analysis method and device and electronic equipment

Abstract

The invention discloses a data analysis method and device and electronic equipment, and the method comprises the steps: obtaining a table alias of a table data source from a table mapping intermediate layer, obtaining table data corresponding to a table real name from a database according to the table real name corresponding to the table alias in the table mapping intermediate layer, and analyzing the table data according to an analysis grammar supported by a database type of the database. And compiling a model statement for analyzing and processing the table data to obtain a data analysis model, and outputting the data analysis model to a database to execute to obtain a data analysis result. The table mapping middle layer serving as an isolation layer is added between the user side and the database, the table mapping middle layer stores the corresponding relation between the table alias and the table real name, and when the data analysis model is written, the table mapping middle layer provides the table alias to the user side; the user side cannot directly obtain the database underlying design from the database according to the table alias, so that the situation that the underlying data in the database is leaked or even tampered is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and more specifically, to a data analysis method, device and electronic equipment. Background Art

[0002] In the process of big data analysis, it is usually necessary to write a data analysis model and submit the data analysis model to the analysis platform. The analysis platform performs query, data processing and other operations based on the content of the data analysis model and outputs the data analysis results.

[0003] In the existing solution, the analysis platform can not only display the data analysis results to the upper-level users, but also display the underlying design of the database to the user side, such as the logic of database sub-library and table sub-library, the table name and library name of the real data, etc. Therefore, there are certain security risks, and the underlying data in the database is prone to leakage or even tampering. Summary of the invention

[0004] In view of this, the present invention discloses a data analysis method, device and electronic device to solve the problem that the underlying data in the database is easily leaked or even tampered with due to the display of the underlying design of the database to the user side.

[0005] A data analysis method, comprising:

[0006] Obtaining a table alias of a table data source from a table mapping middle layer, wherein the table mapping middle layer serves as an isolation layer between the user side and the database and stores a correspondence between the table alias and the table real name;

[0007] According to the table real name corresponding to the table alias in the table mapping middle layer, obtaining the table data corresponding to the table real name from the database;

[0008] According to the analysis syntax supported by the database type of the database, a model statement for analyzing and processing the table data is written to obtain a data analysis model;

[0009] The data analysis model is output to the database for execution to obtain data analysis results.

[0010] Optionally, the outputting the data analysis model to the database to obtain the data analysis result includes:

[0011] Performing accuracy testing on the data analysis model;

[0012] When the data analysis model passes the accuracy test, the data analysis model is output to the database for execution to obtain the data analysis result.

[0013] Optionally, the performing accuracy test on the data analysis model includes:

[0014] Inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value, wherein the compilation result test value includes the actual execution content of the database;

[0015] Output the compilation result test value to the test library for execution;

[0016] If the test library executes the compilation result test value to obtain a data test result, it is determined that the data analysis model passes the accuracy test;

[0017] If the test library executes the compilation result test value and fails to obtain the data test result, it is determined that the data analysis model fails the accuracy test.

[0018] Optionally, when the data analysis model passes the accuracy test, outputting the data analysis model to the database for execution to obtain the data analysis result includes:

[0019] When the data analysis model passes the accuracy test, the data analysis model is input into the unified model compiler in the execution mode for processing to obtain a true value of the compilation result;

[0020] The true value of the compilation result is input into the database for execution to obtain the data analysis result.

[0021] Optionally, the unified model compiler supports multiple syntax type models, and different syntax type models correspond to different types of databases, wherein the multiple syntax type models include at least: Spark-SQL model, Spark operator model and Oracle-SQL model.

[0022] Optionally, when the data analysis model is a structured query language SQL model, the step of inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value includes:

[0023] Parsing the data analysis model into an abstract syntax tree;

[0024] Acquire a first relevant table alias when constructing the SQL model from the abstract syntax tree, wherein the first relevant table alias includes an original table alias, an external table alias, and a model table alias;

[0025] Verifying the abstract syntax tree;

[0026] When the abstract syntax tree passes the verification, mapping and matching the first related table alias with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias;

[0027] When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the first target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement;

[0028] When there is no variable parameter in the abstract syntax tree, determining the first target table real name as the compilation result test value;

[0029] When the variable parameter exists in the abstract syntax tree, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the first target table to obtain the compilation result test value.

[0030] Optionally, the verifying the abstract syntax tree includes:

[0031] Performing a validity check on the statement operation type in the abstract syntax tree;

[0032] When the statement operation type is legal, the data range in the abstract syntax tree is checked, and when the data range passes the check, a permission control condition is added after the table alias that needs to be controlled permission through the abstract syntax tree.

[0033] Optionally, when the data analysis model is a graphical model, the step of inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value includes:

[0034] Performing node parsing on the graphical model, converting the graphical model from a fixed format text into a database operator execution code;

[0035] Verifying the statement type and data range in the database operator execution code;

[0036] When the statement type and the data range pass the verification, the second related table alias when constructing the graphical model in the database operator execution code is mapped and matched with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias;

[0037] When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the second target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement;

[0038] When there is no variable parameter in the database operator execution code, determining the second target table real name as the compilation result test value;

[0039] When the variable parameter exists in the database operator execution code, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the second target table to obtain the compilation result test value.

[0040] A data analysis device, comprising:

[0041] A table alias acquisition unit, used to acquire a table alias of a table data source from a table mapping middle layer, wherein the table mapping middle layer serves as an isolation layer between a user side and a database and stores a correspondence between a table alias and a table real name;

[0042] A table real name determining unit, configured to obtain table data corresponding to the table real name from the database according to the table real name corresponding to the table alias in the table mapping intermediate layer;

[0043] A model writing unit, used to write a model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database, so as to obtain a data analysis model;

[0044] The data analysis unit is used to output the data analysis model to the database to execute and obtain the data analysis results.

[0045] An electronic device, comprising: a memory and a processor;

[0046] The memory is used to store at least one instruction;

[0047] The processor is used to execute the at least one instruction to implement the data analysis method described above.

[0048] It can be seen from the above technical solutions that the present invention discloses a data analysis method, device and electronic device, which obtains the table alias of the table data source from the table mapping middle layer, obtains the table data corresponding to the table real name from the database according to the table alias corresponding to the table real name in the table mapping middle layer, and writes the model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database to obtain the data analysis model, and outputs the data analysis model to the database for execution to obtain the data analysis result. The present application adds a table mapping middle layer as an isolation layer between the user side and the database, and the table mapping middle layer stores the correspondence between the table alias and the table real name. When writing the data analysis model, the table mapping middle layer provides the table alias to the user side, and the user side cannot directly obtain the underlying design of the database from the database according to the table alias, thereby improving the security of data analysis and effectively avoiding the leakage or even tampering of the underlying data in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0050] Figure 1 A flow chart of a data analysis method disclosed in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of a table mapping intermediate layer disclosed in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a data analysis model testing process and execution process disclosed in an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the working principle of a unified model compiler disclosed in an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of a data analysis device disclosed in an embodiment of the present invention;

[0055] Figure 6 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiments of the present invention disclose a data analysis method, device and electronic device, in which a table mapping middle layer as an isolation layer is added between the user side and the database, and the table mapping middle layer stores the correspondence between the table alias and the table real name. When writing a data analysis model, the table mapping middle layer provides the table alias to the user side, and the user side cannot directly obtain the underlying design of the database from the database based on the table alias, thereby improving the security of data analysis and effectively avoiding the leakage or even tampering of the underlying data in the database.

[0058] See also Figure 1, a flow chart of a data analysis method disclosed in an embodiment of the present invention, the method comprising:

[0059] Step S101: Obtain the table alias of the table data source from the table mapping middle layer.

[0060] Table types are divided into three categories according to the source of table data: original table, external table and model table.

[0061] The original table is used to record data generated by the production system based on business transactions.

[0062] External tables are used to record external data uploaded by users.

[0063] The model table is used to record the result data obtained by users through model analysis in the platform.

[0064] In enterprise-level data analysis scenarios, analysts in different positions and regions can usually only analyze data within their jurisdiction. In order to control the data scope that analysts can view and avoid horizontal unauthorized access between users and vertical unauthorized access to data, this application provides a table mapping middle layer. The table mapping middle layer serves as an isolation layer between the user side and the database, and stores the correspondence between table aliases and table real names to prevent users from directly contacting the real table names.

[0065] See also Figure 2 , a schematic diagram of the table mapping middle layer provided by the present application, in which the mapping relationship between table aliases and table real names is stored. The table name used by the user when writing a data analysis model is the table alias provided by the table mapping middle layer. For example, the table alias of the original table displayed in the user view is: ods.tableA, the table alias of the external table is: ext.tableB, and the table alias of the model table is: mod. tableC. The database stores the table real name, for example, the table real name of the original table is: ods.user_able, the table real name of the external table is: ods.user_relation, and the table real name of the model table is: mod.m000001.

[0066] The table name used by users when writing data analysis models is the table alias provided by the table mapping middle layer. Users cannot obtain the real name of the table in the database. At the same time, the table generated by the user through the model can also be set to a custom name and recorded in the table mapping middle layer. The custom name serves as the unique identifier of the table.

[0067] In practical applications, the table types involved in this application may be account tables, etc., which are not limited here.

[0068] Step S102: According to the table real name corresponding to the table alias in the table mapping middle layer, obtain the table data corresponding to the table real name from the database.

[0069] The table mapping middle layer stores the mapping relationship between table aliases and table real names. The table aliases of the obtained table data sources are matched with the table aliases in the table mapping middle layer. The table real name corresponding to the matched table alias is the table real name corresponding to the table alias of the table data source.

[0070] The corresponding table data can be obtained from the database using the real name of the table.

[0071] Step S103: compile a model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database to obtain a data analysis model.

[0072] The data analysis method disclosed in this application supports analysis syntax of multiple models, such as SparkSQL, Spark operators, and Oracle-SQL.

[0073] SparkSQL is a Spark module used by Spark for structured data processing. SparkSQL provides a unified programming interface that can process multiple data sources, including Hive tables, Parquet files, JSON files, relational databases, etc., allowing users to process data in a unified environment without switching between different tools and technologies.

[0074] Spark operators are mainly divided into two categories: Transformation operators and Action operators. Transformation operators are used to perform various transformation operations on RDD (Resilient Distributed Datasets), but the results are not calculated immediately. Instead, the actual calculation is triggered when an Action operator is encountered. Action operators are used to trigger the execution of Spark jobs and return the results to the driver.

[0075] Oracle-SQL is a proprietary query language used by Oracle database. It is a powerful and widely used relational database query language for managing and operating Oracle database.

[0076] Different database types support different analysis syntaxes. Based on this, this application writes model statements for analyzing and processing table data according to the analysis syntax supported by the database type to obtain a data analysis model.

[0077] In addition, since the present application supports the analysis syntax of multiple models, the present application can support the data analysis process of multiple databases.

[0078] Step S104: output the data analysis model to the database for execution to obtain data analysis results.

[0079] In summary, the present invention discloses a data analysis method, which obtains the table alias of the table data source from the table mapping middle layer, obtains the table data corresponding to the table real name from the database according to the table alias corresponding to the table real name in the table mapping middle layer, writes the model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database to obtain the data analysis model, and outputs the data analysis model to the database for execution to obtain the data analysis result. The present application adds a table mapping middle layer as an isolation layer between the user side and the database, and the table mapping middle layer stores the correspondence between the table alias and the table real name. When writing the data analysis model, the table mapping middle layer provides the table alias to the user side, and the user side cannot directly obtain the underlying design of the database from the database according to the table alias, thereby improving the security of data analysis and effectively avoiding the leakage or even tampering of the underlying data in the database.

[0080] In actual applications, the written data analysis model may contain syntax errors or use non-existent tables. If the written data analysis model is directly output to the database for execution, errors may occur when the data analysis model is executed at the intermediate steps, resulting in a waste of time and resources. Therefore, in order to improve the accuracy of the data analysis model, the data analysis model needs to be tested for accuracy before executing it.

[0081] In one embodiment, step S104 may specifically include:

[0082] Test the accuracy of data analysis models;

[0083] When the data analysis model passes the accuracy test, the data analysis model is output to the database for execution to obtain the data analysis result.

[0084] In summary, the present application performs an accuracy test on the data analysis model before executing the data analysis model, and when the data analysis model passes the accuracy test, the data analysis model is output to the database to execute and obtain the data analysis results, thereby improving the accuracy of the data analysis results.

[0085] When users use table aliases in data analysis models, they usually do not actively bring filtering conditions to narrow the query scope. In addition, in order to control the user's viewing permissions, the analysis platform needs to automatically add permission control conditions to the data analysis model. Based on this, the present application has adopted a unified model compiler suitable for a variety of databases. The unified model compiler parses, verifies, maps and compiles the data analysis model, and compiles the data analysis model into content that is actually executed by the database, taking into account permission control and underlying isolation, and has multiple characteristics such as scalability and easy maintenance.

[0086] It should be noted that the unified model compiler in this application supports multiple syntax type models, and different syntax type models correspond to different types of databases. Among them, the multiple syntax type models at least include: Spark-SQL model, Spark operator model and Oracle-SQL model. In practical applications, the multiple syntax type models can also include models of other syntax types to support multiple databases.

[0087] See also Figure 3 , a schematic diagram of a data analysis model testing process and execution process provided in an embodiment of the present application, the data analysis model includes: Spark-SQL model, Spark operator model and Oracle-SQL model.

[0088] The unified model compiler in the test mode parses, verifies, maps and compiles the data analysis model, compiles the data analysis model into content that is actually executed in the database, and obtains the compilation result. The compilation result is input into the test library which serves as the model execution library for testing. After the data analysis model is executed, no real data analysis result is generated.

[0089] When the data analysis model passes the test, the data analysis model is input into the unified model compiler of the execution mode. The unified model compiler of the execution mode parses, verifies, maps and compiles the data analysis model, compiles the data analysis model into the content that is actually executed in the database, and obtains the compilation result. The compilation result is input into the real library which serves as the model execution library. After the data analysis model is executed, the final data analysis result is generated.

[0090] The following is an explanation of the process of processing the data analysis model when the unified model compiler is in test mode, as follows:

[0091] In one embodiment, the process of testing the accuracy of the data analysis model may include:

[0092] (1) Inputting the data analysis model into the unified model compiler in the test mode for processing to obtain a compilation result test value, wherein the compilation result test value includes the actual execution content of the database;

[0093] (2) Outputting the compiled test result to the test library for execution;

[0094] (3) If the test library executes the compilation result test value to obtain a data test result, it is determined that the data analysis model passes the accuracy test;

[0095] (4) If the test library executes the compilation result test value and fails to obtain the data test result, it is determined that the data analysis model fails the accuracy test.

[0096] In this application, after the unified model compiler in the test mode is used to process the data analysis model and obtain the compilation result test value, the accuracy of the data analysis model can be verified by inputting the compilation result test value into the test library for execution. If the test library executes the compilation result test value to obtain the data test result, it is determined that the data analysis model has passed the accuracy test; conversely, if the test library executes the compilation result test value and does not obtain the data test result, it is determined that the data analysis model has not passed the accuracy test.

[0097] The following is an explanation of the process of processing the data analysis model when the unified model compiler is in execution mode, as follows:

[0098] In one embodiment, when the data analysis model passes the accuracy test, the data analysis model is output to the database for execution to obtain the data analysis results, including:

[0099] When the data analysis model passes the accuracy test, the data analysis model is input into the unified model compiler in the execution mode for processing to obtain the true value of the compilation result;

[0100] The actual value of the compilation result is input into the database for execution to obtain the data analysis result.

[0101] In the present application, when the data analysis model passes the accuracy test, the data analysis model is first processed using the unified model compiler in execution mode to obtain the true value of the compilation result, and then the true value of the compilation result is input into the database (i.e., the real library) for execution to obtain the final data analysis result.

[0102] It should be noted that the unified model compiler has the same process for data analysis models, whether in test mode or execution mode, and both include parsing, verification, mapping, and compilation. The following takes the unified model compiler of the test model as an example to illustrate the process of the unified model compiler compiling the data analysis model, as follows:

[0103] See also Figure 4, a schematic diagram of the working principle of a unified model compiler provided in an embodiment of the present application, wherein the data analysis model can be a SQL (Structured Query Language) model, and the corresponding unified model compiler is a SQL model compiler; the data analysis model can also be a graphical model, and the corresponding unified model compiler is a graphical model compiler.

[0104] The SQL model may include: SQLSpark-SQL model and Oracle-SQL model.

[0105] The graphical model can be a Spark operator model.

[0106] (I) See Figure 4 When the data analysis model is an SQL model, the process of inputting the data analysis model into the unified model compiler in the test mode for processing to obtain the compilation result test value may specifically include:

[0107] (1) Parse the data analysis model into an Abstract Syntax Tree (AST).

[0108] An abstract syntax tree is a tree representation of the abstract syntax structure of the source code, which is used in the source code analysis process in a compiler or interpreter. An abstract syntax tree does not represent all the details of the source code, such as specific syntax tags (such as brackets, semicolons, etc.), but organizes the structured information of the source code in the form of a tree to make it easier for the program to understand and process.

[0109] (2) Obtaining a first relevant table alias when constructing the SQL model from the abstract syntax tree, wherein the first relevant table alias includes an original table alias, an external table alias, and a model table alias.

[0110] The process of parsing the data analysis model to obtain the abstract syntax tree and acquiring the alias of the first related table is an SQL parsing process.

[0111] (3) Verifying the abstract syntax tree.

[0112] Specifically: Check the legality of the statement operation type in the abstract syntax tree;

[0113] When the statement operation type is legal, the data range in the abstract syntax tree is checked, and when the data range passes the check, the permission control condition is added after the table alias that needs to be controlled permission through the abstract syntax tree.

[0114] In actual applications, statement operation types generally allow CREATE AS SELECT type and INSERT type, and block DROP, TRUNCATE, UPDATE and other operations. When the statement operation type is illegal, it indicates that the abstract syntax tree contains illegal operations, and compilation is terminated directly.

[0115] When the statement type and data range in the abstract syntax tree pass the verification, the abstract syntax tree adds permission control conditions after the table alias that needs to be controlled, thereby controlling the user's viewing permissions and preventing the user from exceeding the permission.

[0116] (4) When the abstract syntax tree passes the verification, the first related table alias is mapped and matched with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias.

[0117] The first relevant table alias obtained from the abstract syntax tree is mapped and matched with each table alias in the table mapping middle layer of the platform, and the table permission check can be performed on the first relevant table alias to determine whether the user has the permission to use it.

[0118] (5) When the first related table alias is found from the table mapping middle layer, it is determined that the table permission passes the verification, and the first target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement.

[0119] When the first related table alias is not found in the table mapping middle layer, it is determined that the table permission has not passed the verification, and the compilation is directly terminated and the user is prompted. For the first related table alias that passes the verification, the first target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement.

[0120] (6) When there is no variable parameter in the abstract syntax tree, the first target table real name is determined as the compilation result test value.

[0121] (7) When the variable parameter exists in the abstract syntax tree, the variable parameter is replaced with the real value of the variable, and the compiled result test value is obtained by combining the real name of the first target table.

[0122] When compiling and outputting, if the data analysis model does not use variable parameters, the real name of the first target table is directly determined as the compilation result test value. If the data analysis model uses variable parameters, the variable parameters are replaced with the real values ​​of the variables.

[0123] Compared with SQL-based models, the difference of graphical models lies in the inconsistency of the model content format of the syntax. For example, "select * from tableA limit 20" in the SQL statement is written as "Spark.table('tableA').limit(20).show()" in the operator statement. The inconsistency of the statement structure leads to differences in the compilation method of the model content. In the unified model compiler of the present invention, the graphical model input by the user is serialized and objectified through parsing, and the member variables of the parsed object are verified and mapped, and finally the output is in the format of Spark program code type and submitted to the database for execution.

[0124] (ii) See Figure 4 When the data analysis model is a graphical model, the process of inputting the data analysis model into the unified model compiler in the test mode for processing to obtain the compilation result test value may specifically include:

[0125] (1) Perform node parsing on the graphical model, converting the graphical model from a fixed format text into a database operator execution code, and completing the information extraction process.

[0126] If the graphical model contains multiple model nodes, the model nodes are compiled and converted sequentially according to their order. Then the object type of each operator execution code is parsed, and key information such as statement type and data range are extracted from it according to different operator types, so as to perform verification, table mapping and compilation output later.

[0127] (2) Verify the statement type and data range in the database operator execution code.

[0128] (3) When the statement type and the data range pass verification, the second related table alias when constructing the graphical model in the database operator execution code is mapped and matched with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias.

[0129] (4) When the first related table alias is found from the table mapping middle layer, it is determined that the table permission passes the verification, and the second target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement.

[0130] (5) When there is no variable parameter in the database operator execution code, the real name of the second target table is determined as the compilation result test value.

[0131] (6) When the variable parameter exists in the database operator execution code, the variable parameter is replaced with the real value of the variable, and the compiled result test value is obtained by combining the real name of the second target table.

[0132] Corresponding to the above method embodiment, the present invention also discloses a data analysis device.

[0133] See also Figure 5 , a schematic diagram of the structure of a data analysis device disclosed in an embodiment of the present invention, the device may include:

[0134] The table alias acquisition unit 201 is used to acquire the table alias of the table data source from the table mapping middle layer.

[0135] Table types are divided into three categories according to the source of table data: original table, external table and model table.

[0136] The original table is used to record data generated by the production system based on business transactions.

[0137] External tables are used to record external data uploaded by users.

[0138] The model table is used to record the result data obtained by users through model analysis in the platform.

[0139] In enterprise-level data analysis scenarios, analysts in different positions and regions can usually only analyze data within their jurisdiction. In order to control the data scope that analysts can view and avoid horizontal unauthorized access between users and vertical unauthorized access to data, this application provides a table mapping middle layer. The table mapping middle layer serves as an isolation layer between the user side and the database, and stores the correspondence between table aliases and table real names to prevent users from directly contacting the real table names.

[0140] The table real name determining unit 202 is configured to obtain table data corresponding to the table real name from the database according to the table real name corresponding to the table alias in the table mapping intermediate layer.

[0141] The table mapping middle layer stores the mapping relationship between table aliases and table real names. The table aliases of the obtained table data sources are matched with the table aliases in the table mapping middle layer. The table real name corresponding to the matched table alias is the table real name corresponding to the table alias of the table data source.

[0142] The corresponding table data can be obtained from the database using the real name of the table.

[0143] The model writing unit 203 is used to write a model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database to obtain a data analysis model.

[0144] The data analysis method disclosed in this application supports analysis syntax of multiple models, such as SparkSQL, Spark operators, and Oracle-SQL.

[0145] SparkSQL is a Spark module used by Spark for structured data processing. SparkSQL provides a unified programming interface that can process multiple data sources, including Hive tables, Parquet files, JSON files, relational databases, etc., allowing users to process data in a unified environment without switching between different tools and technologies.

[0146] Spark operators are mainly divided into two categories: Transformation operators and Action operators. Transformation operators are used to perform various transformation operations on RDD (Resilient Distributed Datasets), but the results are not calculated immediately. Instead, the actual calculation is triggered when an Action operator is encountered. Action operators are used to trigger the execution of Spark jobs and return the results to the driver.

[0147] Oracle-SQL is a proprietary query language used by Oracle database. It is a powerful and widely used relational database query language for managing and operating Oracle database.

[0148] Different database types support different analysis syntaxes. Based on this, this application writes model statements for analyzing and processing table data according to the analysis syntax supported by the database type to obtain a data analysis model.

[0149] In addition, since the present application supports the analysis syntax of multiple models, the present application can support the data analysis process of multiple databases.

[0150] The data analysis unit 204 is used to output the data analysis model to the database for execution to obtain data analysis results.

[0151] In summary, the present invention discloses a data analysis device, which obtains the table alias of the table data source from the table mapping middle layer, obtains the table data corresponding to the table real name from the database according to the table alias corresponding to the table real name in the table mapping middle layer, writes the model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database to obtain the data analysis model, and outputs the data analysis model to the database for execution to obtain the data analysis result. The present application adds a table mapping middle layer as an isolation layer between the user side and the database, and the table mapping middle layer stores the correspondence between the table alias and the table real name. When writing the data analysis model, the table mapping middle layer provides the table alias to the user side, and the user side cannot directly obtain the underlying design of the database from the database according to the table alias, thereby improving the security of data analysis and effectively avoiding the leakage or even tampering of the underlying data in the database.

[0152] In actual applications, the written data analysis model may contain syntax errors or use non-existent tables. If the written data analysis model is directly output to the database for execution, errors may occur when the data analysis model is executed at the intermediate steps, resulting in a waste of time and resources. Therefore, in order to improve the accuracy of the data analysis model, the data analysis model needs to be tested for accuracy before executing it.

[0153] In one embodiment, the data analysis unit 204 may specifically include:

[0154] A testing subunit, used to test the accuracy of the data analysis model;

[0155] The output subunit is used to output the data analysis model to the database to execute and obtain the data analysis result when the data analysis model passes the accuracy test.

[0156] In summary, the present application performs an accuracy test on the data analysis model before executing the data analysis model, and when the data analysis model passes the accuracy test, the data analysis model is output to the database to execute and obtain the data analysis results, thereby improving the accuracy of the data analysis results.

[0157] When users use table aliases in data analysis models, they usually do not actively bring filtering conditions to narrow the query scope. In addition, in order to control the user's viewing permissions, the analysis platform needs to automatically add permission control conditions to the data analysis model. Based on this, the present application has adopted a unified model compiler suitable for a variety of databases. The unified model compiler parses, verifies, maps and compiles the data analysis model, and compiles the data analysis model into content that is actually executed by the database, taking into account permission control and underlying isolation, and has multiple characteristics such as scalability and easy maintenance.

[0158] It should be noted that the unified model compiler in this application supports multiple syntax type models, and different syntax type models correspond to different types of databases. Among them, the multiple syntax type models at least include: Spark-SQL model, Spark operator model and Oracle-SQL model. In practical applications, the multiple syntax type models can also include models of other syntax types to support multiple databases.

[0159] In one embodiment, the test subunit may be specifically used for:

[0160] Inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value, wherein the compilation result test value includes the actual execution content of the database;

[0161] Output the compilation result test value to the test library for execution;

[0162] If the test library executes the compilation result test value to obtain a data test result, it is determined that the data analysis model passes the accuracy test;

[0163] If the test library executes the compilation result test value and fails to obtain the data test result, it is determined that the data analysis model fails the accuracy test.

[0164] In one embodiment, the output subunit may be specifically used to: when the data analysis model passes the accuracy test, input the data analysis model into the unified model compiler in the execution mode for processing to obtain the true value of the compilation result;

[0165] The true value of the compilation result is input into the database for execution to obtain the data analysis result.

[0166] Preferably, the unified model compiler supports multiple syntax type models, and different syntax type models correspond to different types of databases, wherein the multiple syntax type models include at least: Spark-SQL model, Spark operator model and Oracle-SQL model.

[0167] In one embodiment, the test subunit may also be used for:

[0168] When the data analysis model is a SQL model, parsing the data analysis model into an abstract syntax tree;

[0169] Acquire a first relevant table alias when constructing the SQL model from the abstract syntax tree, wherein the first relevant table alias includes an original table alias, an external table alias, and a model table alias;

[0170] Verifying the abstract syntax tree;

[0171] When the abstract syntax tree passes the verification, mapping and matching the first related table alias with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias;

[0172] When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the first target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement;

[0173] When there is no variable parameter in the abstract syntax tree, determining the first target table real name as the compilation result test value;

[0174] When the variable parameter exists in the abstract syntax tree, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the first target table to obtain the compilation result test value.

[0175] In one embodiment, the test subunit may also be used for:

[0176] Performing a validity check on the statement operation type in the abstract syntax tree;

[0177] When the statement operation type is legal, the data range in the abstract syntax tree is checked, and when the data range passes the check, a permission control condition is added after the table alias that needs to be controlled permission through the abstract syntax tree.

[0178] In one embodiment, the test subunit may also be used for:

[0179] When the data analysis model is a graphical model, performing node parsing on the graphical model, and converting the graphical model from a fixed format text into a database operator execution code;

[0180] Verifying the statement type and data range in the database operator execution code;

[0181] When the statement type and the data range pass the verification, the second related table alias when constructing the graphical model in the database operator execution code is mapped and matched with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias;

[0182] When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the second target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement;

[0183] When there is no variable parameter in the database operator execution code, determining the second target table real name as the compilation result test value;

[0184] When the variable parameter exists in the database operator execution code, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the second target table to obtain the compilation result test value.

[0185] It should be noted that, for the specific working principles of each component in the device embodiment, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0186] Corresponding to the above embodiment, Figure 4 As shown, the present invention also provides an electronic device, which may include: a processor 1 and a memory 2;

[0187] The processor 1 and the memory 2 communicate with each other via a communication bus 3;

[0188] Processor 1, configured to execute at least one instruction;

[0189] Memory 2, used to store at least one instruction;

[0190] The processor 1 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0191] The memory 2 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0192] Among them, the processor executes at least one instruction to implement the steps shown in the data analysis method embodiment.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0194] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0195] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data analysis method, characterized in that: include: Obtaining a table alias of a table data source from a table mapping middle layer, wherein the table mapping middle layer serves as an isolation layer between the user side and the database and stores a correspondence between the table alias and the table real name; According to the table real name corresponding to the table alias in the table mapping middle layer, obtaining the table data corresponding to the table real name from the database; According to the analysis syntax supported by the database type of the database, a model statement for analyzing and processing the table data is written to obtain a data analysis model; The data analysis model is output to the database for execution to obtain data analysis results.

2. The data analysis method according to claim 1, characterized in that: Outputting the data analysis model to the database for execution to obtain data analysis results includes: Performing accuracy testing on the data analysis model; When the data analysis model passes the accuracy test, the data analysis model is output to the database for execution to obtain the data analysis result.

3. The data analysis method according to claim 2, characterized in that: The accuracy test of the data analysis model includes: Inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value, wherein the compilation result test value includes the actual execution content of the database; Output the compilation result test value to the test library for execution; If the test library executes the compilation result test value to obtain a data test result, it is determined that the data analysis model passes the accuracy test; If the test library executes the compilation result test value and fails to obtain the data test result, it is determined that the data analysis model fails the accuracy test.

4. The data analysis method according to claim 3, characterized in that: When the data analysis model passes the accuracy test, the data analysis model is output to the database for execution to obtain the data analysis result, including: When the data analysis model passes the accuracy test, the data analysis model is input into the unified model compiler in the execution mode for processing to obtain a true value of the compilation result; The true value of the compilation result is input into the database for execution to obtain the data analysis result.

5. The data analysis method according to claim 3 or 4, characterized in that: The unified model compiler supports multiple syntax type models, and different syntax type models correspond to different types of databases, wherein the multiple syntax type models at least include: Spark-SQL model, Spark operator model and Oracle-SQL model.

6. The data analysis method according to claim 3, characterized in that: When the data analysis model is a structured query language SQL model, the step of inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value includes: Parsing the data analysis model into an abstract syntax tree; Acquire a first relevant table alias when constructing the SQL model from the abstract syntax tree, wherein the first relevant table alias includes an original table alias, an external table alias, and a model table alias; Verifying the abstract syntax tree; When the abstract syntax tree passes the verification, mapping and matching the first related table alias with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias; When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the first target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement; When there is no variable parameter in the abstract syntax tree, determining the first target table real name as the compilation result test value; When the variable parameter exists in the abstract syntax tree, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the first target table to obtain the compilation result test value.

7. The data analysis method according to claim 6, characterized in that: The checking of the abstract syntax tree includes: Performing a validity check on the statement operation type in the abstract syntax tree; When the statement operation type is legal, the data range in the abstract syntax tree is checked, and when the data range passes the check, a permission control condition is added after the table alias that needs to be controlled permission through the abstract syntax tree.

8. The data analysis method according to claim 3, characterized in that: When the data analysis model is a graphical model, the step of inputting the data analysis model into a unified model compiler in a test mode for processing to obtain a compilation result test value includes: Performing node parsing on the graphical model, converting the graphical model from a fixed format text into a database operator execution code; Verifying the statement type and data range in the database operator execution code; When the statement type and the data range pass the verification, the second related table alias when constructing the graphical model in the database operator execution code is mapped and matched with each table alias in the table mapping intermediate layer to perform table permission verification on the first related table alias; When the first related table alias is found from the table mapping middle layer, it is determined that the table authority passes the verification, and the second target table real name corresponding to the first related table alias is obtained from the table mapping middle layer for replacement; When there is no variable parameter in the database operator execution code, determining the second target table real name as the compilation result test value; When the variable parameter exists in the database operator execution code, the variable parameter is replaced with the real value of the variable, and is compiled in combination with the real name of the second target table to obtain the compilation result test value.

9. A data analysis device, characterized in that: include: A table alias acquisition unit, used to acquire a table alias of a table data source from a table mapping middle layer, wherein the table mapping middle layer serves as an isolation layer between a user side and a database and stores a correspondence between a table alias and a table real name; A table real name determining unit, configured to obtain table data corresponding to the table real name from the database according to the table real name corresponding to the table alias in the table mapping intermediate layer; A model writing unit, used to write a model statement for analyzing and processing the table data according to the analysis syntax supported by the database type of the database, so as to obtain a data analysis model; The data analysis unit is used to output the data analysis model to the database to execute and obtain the data analysis results.

10. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the data analysis method as described in any one of claims 1 to 8.

Images