Data analysis method, device and storage medium

Abstract

The application discloses a data analysis method, device and storage medium, wherein the data analysis method comprises the following steps: analyzing based on function information of a function related to a project to be analyzed, and determining a target table of the function; querying the function information of the function based on a database tracking plug-in, obtaining first source information of the function irrelevant to dynamic SQL, and analyzing the function information of the function based on a first regular expression, and obtaining second source information of the function related to dynamic SQL; wherein the first source information and the second source information at least comprise a source table of the function; and integrating the first source information, the second source information and the target table belonging to the same function, and obtaining a blood relationship of data in the project to be analyzed. The above scheme can improve the accuracy of data analysis and improve the efficiency of data analysis.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a data analysis method, device and storage medium. Background Technology

[0002] For projects such as data warehouse projects and data analysis projects, due to personnel changes or asynchronous document management, it is often necessary to analyze the source tables of data from database functions.

[0003] Currently, SQL is often parsed from logs to analyze the lineage relationships between SQL statements; alternatively, the processed SQL is pushed onto a stack, then a syntax tree is abstracted, and lineage relationships are analyzed by iteratively parsing the syntax tree. However, these methods are time-consuming and sometimes require manual intervention, making it difficult to ensure the accuracy of data analysis. Therefore, how to improve the efficiency of data analysis while simultaneously enhancing its accuracy has become a pressing issue. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a data analysis method, device, and storage medium that can improve the accuracy and efficiency of data analysis.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a data analysis method, comprising: parsing function information related to functions in the project to be analyzed to determine the target table of the functions; querying function information based on a database tracking plugin to obtain first source information unrelated to dynamic SQL, and parsing function information based on a first regular expression to obtain second source information related to dynamic SQL; wherein the first source information and the second source information at least include the source table of the functions; and integrating the first source information, the second source information, and the target table belonging to the same function to obtain the lineage relationship of the data in the project to be analyzed.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a data analysis device, including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the data analysis method described in the first aspect.

[0007] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the data analysis method of the first aspect described above.

[0008] The above scheme determines the target table of functions by parsing the function information involved in the project under analysis; then, it obtains the first source information of functions unrelated to dynamic SQL by querying function information using a database tracing plugin, and obtains the second source information of functions related to dynamic SQL by parsing function information using a first regular expression. The first and second source information at least include the source table of the function. The scheme integrates the first and second source information of functions belonging to the same category with the target table to obtain the lineage relationship of data in the project under analysis. On the one hand, parsing the function information involved in the project under analysis to determine the target table of each function improves the accuracy of the target table; on the other hand, querying function information using a database tracing plugin or parsing function information using a first regular expression to obtain the source information of functions helps to improve the speed of obtaining function information and the comprehensiveness of the source information. Furthermore, integrating the function information of the same category to determine the lineage relationship of data in the project under analysis helps to improve the accuracy of the lineage relationship. Therefore, it can improve the efficiency of data analysis while improving the accuracy of the analysis.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0011] Figure 1 This is a flowchart illustrating an embodiment of the data analysis method of this application;

[0012] Figure 2 This is a schematic diagram of an embodiment of function information;

[0013] Figure 3 This is a schematic diagram of an embodiment of the target table of a function;

[0014] Figure 4 This is a schematic diagram of an embodiment of the first source information;

[0015] Figure 5 This is a schematic diagram of an embodiment of the hierarchical relationship of the source table corresponding to a function;

[0016] Figure 6 This is a schematic diagram of an embodiment of the lineage relationship of data in the project to be analyzed;

[0017] Figure 7 This is a schematic diagram of the framework of an embodiment of the data analysis method of this application;

[0018] Figure 8 This is a schematic diagram of the framework of an embodiment of the data analysis device of this application;

[0019] Figure 9 This is a schematic diagram of the framework of an embodiment of the data analysis device of this application;

[0020] Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. "Several" means at least one. The terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data analysis method of this application.

[0025] Specifically, this may include the following steps:

[0026] Step S11: Based on the function information of the functions involved in the project to be analyzed, perform analysis to determine the target table of the functions.

[0027] In one implementation scenario, before parsing the function information of the functions involved in the project to be analyzed and determining the target table of the functions, several metadata relationship tables of the project to be analyzed can be linked to obtain the first information of the functions involved in the project. This first information includes at least the function identifier. Specifically, the several metadata relationship tables may include, but are not limited to: business tables, detail tables, summary tables, and marketplace tables. The metadata relationship tables can be determined according to the actual situation and are not specifically limited here. After linking the several metadata relationship tables, the first information of the functions involved in the project to be analyzed is obtained. The first information may include the function identifier, the database corresponding to the function, the schema corresponding to the function, the function name, the function parameters, and the function content. For example, the function identifier may be the function OID (Object identifier). Then, the function identifier is linked with the function parameter table to obtain the second information of the function. The second information may be related to the function parameters. Based on this, the first and second information of the function are integrated to obtain the function information of the functions involved in the project to be analyzed. For an example, please refer to [link to example]. Figure 2 , Figure 2 This is a schematic diagram of one embodiment of function information. The metadata relationship table includes two relationship tables: pg_proc and routes. The pg_proc and routes tables are associated to obtain the first information of the function. This first information includes the function identifier (fun_oid), the database name corresponding to the function (fun_catalog), the schema name corresponding to the function (fun_schemata), the function name (fun_name), the function parameters (fun_par), and the function content (fun_text). Then, based on the function identifier, the function parameter table is associated with the function parameter table. Specifically, the function parameters can be row-transformed according to the function identifier to obtain the second information of the function. The first and second information of the function are further integrated to obtain the function information, which includes the function identifier, the schema name corresponding to the function, etc. The function identifier is 116672, and the schema name corresponding to the function is dmcw. The above method obtains the first information about the functions involved in the project by associating them based on the data relationship table of the project to be analyzed. Then, it further associates the function identifiers and function parameter tables to obtain the second information. In this process, obtaining function information based on two different association methods helps to improve the comprehensiveness of the obtained function information. Finally, the obtained function information is integrated to improve the accuracy of the function information while increasing the speed of obtaining function information.

[0028] In one implementation scenario, the function information includes the function content. To determine the target table, the function content can be parsed using a second regular expression to identify the data table adjacent to the keyword representing the target editing operation within the function content. This data table is then used as the target table. The target editing operation may include, but is not limited to, Insert, delete, truncate table, etc., and does not include update. For an example, please refer to [link to example]. Figure 3 , Figure 3 This is a schematic diagram of one embodiment of the target table for a function. Based on parsing the function content using a second regular expression, the data table following keywords such as `Insert into`, `delete`, and `truncate table` in the function content is determined as the target table for the function. Furthermore, since `update` operations are almost nonexistent, the table following `update` is not parsed as the target table for the function. Therefore, the target table (tar_tab_name) is obtained as `dwcw.tw_cw_rz_comp_kpi_m`. This method, using a second regular expression to parse the function content and obtain the target table, helps improve the efficiency of obtaining the target table, thereby improving the data analysis speed and accuracy.

[0029] Step S12: Based on the function information of the query function of the database tracking plugin, obtain the first source information of the function that is not related to the dynamic SQL, and parse the function information of the function based on the first regular expression to obtain the second source information of the function that is related to the dynamic SQL.

[0030] In this embodiment, the first source information and the second source information at least include a source table of functions, and may further include the source table name, source function, source parameter time, update time, etc. The first source information and the second source information can be determined according to the actual situation, and are not specifically limited here.

[0031] In one implementation scenario, function information includes the function's content. To obtain primary source information about the function, independent of dynamic SQL, we can first query the function's content using a database tracing plugin to determine the result type of each data item within that content. The database tracing plugin can be determined based on the database type; for example, in a PostgreSQL database, the database tracing plugin is `plpgsql check`. Furthermore, data items can include, but are not limited to, functions and tables. Then, based on the result type of the data item, we determine whether to select the data item as the source table or source function to obtain the primary source information. For example, the result type of a data item can include `relation`, `function`, etc. If the result type of a data item is `relation`, then the data item is selected as the source table; if the result type of a data item is `function`, then the data item is selected as the source function. In addition, databases can add filtering conditions. These conditions can filter specific functions. For example, they can filter functions that write logs (e.g., "f_etl_event_log"), system functions in "pg_catalog" mode, target temporary tables prefixed with "tp", and functions like 'nvl', 'decode', 'f_etl_create_random', 'f_etl_drop_temp_table', '_pg_expandarrange', and 'plpgsql_show_dependency_tb'. For an example, please refer to [link to example]. Figure 4 , Figure 4This is a schematic diagram of one embodiment of the first source information. In a PostgreSQL database, the result type of each data item in the function content can be determined by querying the function content using the `plpgsql show dependency tb` command in the `plpgsql check` command. The result type of the data item can include relation, function, etc. If the result type of the data item is relation, the data item is selected as the source table; if the result type of the data item is function, the data item is selected as the source function. Finally, the first source information is obtained. The first source information can include the schema (tar_tab_schemata) corresponding to the target table, the name of the target table (tar_tab_name), the identifier of the source function (src_oid), the schema (src_tab_schemata) corresponding to the source table, the name of the source table (src_tab_name), the name of the source function (src_fun_name), and the parameters of the source function (src_fun_par). The above method, by using a database tracing plugin to query and then determining the content of the selected data item based on the result type of the data item, helps to improve the speed of obtaining the first source information and further ensures the scope and quality of data analysis.

[0032] In one implementation scenario, to obtain secondary source information about the function-related dynamic SQL, we can first parse the data table adjacent to any of the keywords in the `from` or `join` clause of the dynamic SQL using a first regular expression. Specifically, we can determine which data table to use as the source table based on whether the data table adjacent to any of the keywords in `from` or `join` is related to the supplementary year in the year table. Then, we determine whether to use the data table as the source table based on whether the data table adjacent to any of the keywords in `from` or `join` is related to the year. For example, we parse the data table corresponding to the keywords `from` and `join` in the dynamic SQL using the first regular expression. If it is a supplementary year in the year table, then we obtain the source table corresponding to the function-related part of the dynamic SQL. This method, by parsing the data table corresponding to the keywords in the dynamic SQL using the first regular expression and determining the source table based on the parsing result, thus obtaining secondary source information about the function-related dynamic SQL, helps to improve the data information of the dynamic SQL, thereby improving the efficiency and accuracy of data analysis.

[0033] Step S13: Integrate the first source information, the second source information, and the target table that belong to the same function to obtain the lineage relationship of the data in the project to be analyzed.

[0034] In one implementation scenario, the first source information also includes the source functions of the functions. To obtain the lineage of the data in the project to be analyzed, it is necessary to first determine the hierarchical relationship of the source tables corresponding to the functions. Before determining the hierarchical relationship of the source functions, the first and second source information can be deduplicated. This method ensures the correctness and accuracy of the lineage of the data in the project to be analyzed by deduplicating the first and second source information, which may contain duplicate information. After deduplicating the first and second source information, a recursive loop is performed based on the source functions corresponding to each function to determine the hierarchical relationship of the source functions corresponding to each function, and then a recursive loop is performed based on the source tables corresponding to each function to determine the hierarchical relationship of the source tables corresponding to each function. For example, based on function B, we obtain the source tables of function B, resulting in source table 3 and source table 4. Source table 3 is the target table of function A. Function A's source tables include source table 1 and source table 2, and source tables 1 and 2 have no source tables. Therefore, the hierarchy of function A is determined to be 0, and the hierarchy of function B is determined to be 2. Other cases can be deduced similarly, thus determining the hierarchy of the source tables corresponding to the functions. For an example, please refer to [link to example]. Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of the hierarchical relationship of the source tables corresponding to a function. Here, `fun_oid` represents the function identifier, `fun_catalog` represents the database name corresponding to the function, `fun_schemata` represents the schema name corresponding to the function, `fun_name` represents the function name, `fun_par` represents the function parameters, `tar_tab_name` represents the target table name, `src_tab_name` represents the source table name, and `level_num` represents the hierarchical relationship value of the source tables corresponding to the function. After obtaining the hierarchical relationship of the source tables corresponding to the function, further, based on the source function and its hierarchical relationship, as well as the target table, source table, and their hierarchical relationship, the lineage relationship of the data in the project to be analyzed is obtained. For an example, please refer to [link to example]. Figure 6 , Figure 6This is a schematic diagram illustrating an embodiment of the lineage relationship of data in the project to be analyzed. `fun_oid` represents the function identifier, `fun_catalog` represents the database name corresponding to the function, `fun_schemata` represents the schema name corresponding to the function, `fun_name` represents the function name, `fun_par` represents the function parameters, `src_fun_name` represents the source function name, and `level_num` represents the lineage relationship of the data in the project to be analyzed. This method, by recursively iterating based on the source functions or source tables corresponding to each function, determines the hierarchical relationship of the source tables corresponding to the functions, which helps improve the accuracy of the hierarchical relationship of the source tables. Furthermore, based on the source functions and their hierarchical relationships, as well as the target table, source table, and their hierarchical relationships, the lineage relationship of the data in the project to be analyzed is obtained, further improving the accuracy of the lineage relationship of the data in the project to be analyzed.

[0035] The above scheme determines the target table of functions by parsing the function information involved in the project under analysis; then, it obtains the first source information of functions unrelated to dynamic SQL by querying function information using a database tracing plugin, and obtains the second source information of functions related to dynamic SQL by parsing function information using a first regular expression. The first and second source information at least include the source table of the function. The scheme integrates the first and second source information of functions belonging to the same category with the target table to obtain the lineage relationship of data in the project under analysis. On the one hand, parsing the function information involved in the project under analysis to determine the target table of each function improves the accuracy of the target table; on the other hand, querying function information using a database tracing plugin or parsing function information using a first regular expression to obtain the source information of functions helps to improve the speed of obtaining function information and the comprehensiveness of the source information. Furthermore, integrating the function information of the same category to determine the lineage relationship of data in the project under analysis helps to improve the accuracy of the lineage relationship. Therefore, it can improve the efficiency of data analysis while improving the accuracy of the analysis.

[0036] Please see Figure 7 , Figure 7This is a schematic diagram of the framework of an embodiment of the data analysis method of this application. After determining the project to be analyzed, function information parsing is performed first, that is, the function information of the functions involved in the project to be analyzed is parsed to determine the target table of the function; then, function information of the function is obtained in different ways, that is, the function information of the function is queried based on the database tracking plugin to obtain the first source information of the function that is not related to the dynamic SQL, and the function information of the function is parsed based on the first regular expression to obtain the second source information of the function that is related to the dynamic SQL; after obtaining the first source information and the second source information, information deduplication is performed on the first source information and the second source information; and the function information is integrated, that is, the first source information, the second source information and the target table belonging to the same function are integrated to finally obtain the lineage relationship of the data in the project to be analyzed. For example, in the KPI function of the financing market layer, in order to obtain the lineage relationship of the KPI function of the financing market layer, specifically, firstly, based on several metadata relationship tables of the project to be analyzed, the first information of the function involved in the project to be analyzed is obtained by association, and based on the function identifier and function parameter table of the function, the second information of the function is obtained. Then, the first information and the second information of the function are integrated to obtain the function information of the function involved in the project to be analyzed. Further, based on the function information of the function involved in the project to be analyzed, the target table of the function is determined, and the first source information and the second source information of the function are obtained through two methods. Based on the first source information, the second source information and the target table of the same function, the lineage relationship of the data in the project to be analyzed is obtained, and the analyzed lineage relationship is integrated into a large table.

[0037] The above scheme determines the target table of functions by parsing the function information involved in the project under analysis; then, it obtains the first source information of functions unrelated to dynamic SQL by querying function information using a database tracing plugin, and obtains the second source information of functions related to dynamic SQL by parsing function information using a first regular expression. The first and second source information at least include the source table of the function. The scheme integrates the first and second source information of functions belonging to the same category with the target table to obtain the lineage relationship of data in the project under analysis. On the one hand, parsing the function information involved in the project under analysis to determine the target table of each function improves the accuracy of the target table; on the other hand, querying function information using a database tracing plugin or parsing function information using a first regular expression to obtain the source information of functions helps to improve the speed of obtaining function information and the comprehensiveness of the source information. Furthermore, integrating the function information of the same category to determine the lineage relationship of data in the project under analysis helps to improve the accuracy of the lineage relationship. Therefore, it can improve the efficiency of data analysis while improving the accuracy of the analysis.

[0038] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0039] Please see Figure 8 , Figure 8 This is a schematic diagram of the framework of an embodiment of the data analysis device of this application. The data analysis device 80 includes a determining module 81, an acquiring module 82, and an integrating module 83. The determining module 81 is used to parse the function information of the functions involved in the project to be analyzed to determine the target table of the functions; the acquiring module 82 is used to query the function information of the functions based on the database tracking plugin to obtain first source information of the functions that is unrelated to dynamic SQL, and to parse the function information of the functions based on a first regular expression to obtain second source information of the functions that is related to dynamic SQL; wherein the first source information and the second source information at least include the source table of the functions; the integrating module 83 is used to integrate the first source information, the second source information, and the target table belonging to the same function to obtain the lineage relationship of the data in the project to be analyzed.

[0040] The above scheme determines the target table of functions by parsing the function information involved in the project under analysis; then, it obtains the first source information of functions unrelated to dynamic SQL by querying function information using a database tracing plugin, and obtains the second source information of functions related to dynamic SQL by parsing function information using a first regular expression. The first and second source information at least include the source table of the function. The scheme integrates the first and second source information of functions belonging to the same category with the target table to obtain the lineage relationship of data in the project under analysis. On the one hand, parsing the function information involved in the project under analysis to determine the target table of each function improves the accuracy of the target table; on the other hand, querying function information using a database tracing plugin or parsing function information using a first regular expression to obtain the source information of functions helps to improve the speed of obtaining function information and the comprehensiveness of the source information. Furthermore, integrating the function information of the same category to determine the lineage relationship of data in the project under analysis helps to improve the accuracy of the lineage relationship. Therefore, it can improve the efficiency of data analysis while improving the accuracy of the analysis.

[0041] In some disclosed embodiments, the integration module 83 includes a determining submodule and an obtaining submodule. The determining submodule is used to perform a recursive loop based on the source functions corresponding to each function to determine the hierarchical relationship of the source functions corresponding to each function, and to perform a recursive loop based on the source tables corresponding to each function to determine the hierarchical relationship of the source tables corresponding to each function; the obtaining submodule is used to obtain the lineage relationship of the data in the project to be analyzed based on the source functions of the functions and their hierarchical relationship, as well as the target table, source table and their hierarchical relationship of the functions.

[0042] Therefore, by recursively iterating based on the source functions or source tables corresponding to each function, the hierarchical relationship of the source tables corresponding to the functions can be determined, which helps to improve the accuracy of the hierarchical relationship of the source tables. Furthermore, based on the source functions and their hierarchical relationships, as well as the target tables, source tables and their hierarchical relationships, the lineage relationship of the data in the project to be analyzed can be obtained, further improving the accuracy of the lineage relationship of the data in the project to be analyzed.

[0043] In some disclosed embodiments, the integration module 83 further includes a deduplication submodule, which is used to deduplicatize the first source information and the second source information.

[0044] Therefore, since there may be duplicate information between the first and second source information, deduplication of the first and second source information is performed to ensure the correctness and accuracy of the lineage of the data in the project to be analyzed.

[0045] In some disclosed embodiments, the data analysis device 80 includes a first association module, a second association module, and an information integration module. The first association module is used to perform association based on several metadata relationship tables of the project to be analyzed to obtain first information of the functions involved in the project to be analyzed; wherein, the first information includes at least the function identifier of the function; the second association module is used to perform association based on the function identifier of the function and the function parameter table to obtain second information of the function; the information integration module is used to integrate the first information and the second information of the function to obtain function information of the functions involved in the project to be analyzed.

[0046] Therefore, by associating data based on the data relationship table of the project to be analyzed, the first information of the functions involved in the project to be analyzed is obtained. Then, by further associating the function identifier and function parameter table, the second information is obtained. In this process, obtaining function information based on two different association methods helps to improve the comprehensiveness of the obtained function information. Then, the obtained function information is integrated, which improves the speed of obtaining function information and the accuracy of the function information.

[0047] In some disclosed embodiments, the function information of the function includes the function content of the function, and the determining module 81 includes a determining submodule, which is used to parse the function content of the function based on a second regular expression and determine the data table in the function content that is adjacent to the keyword representing the target editing operation, as the target table of the function.

[0048] Therefore, parsing the function content using the second regular expression to obtain the target table of the function helps improve the efficiency of obtaining the target table, thereby improving the data analysis speed and accuracy.

[0049] In some disclosed embodiments, the function information of a function includes the function content of the function, and the acquisition module 82 includes a determination submodule and a judgment submodule. The determination submodule is used to query the function content of the function based on the database tracking plugin and determine the result type of each data item in the function content; the judgment submodule is used to determine whether to select a data item as a source table or source function based on the result type of the data item, and obtain the first source information.

[0050] Therefore, querying using a database tracking plugin and then determining the content of selected data items based on the result type of the data items helps to improve the speed of obtaining information from primary sources and further ensures the scope and quality of data analysis.

[0051] In some disclosed embodiments, the acquisition module 82 further includes a parsing submodule and a selection submodule. The parsing submodule is used to parse the data table in the dynamic SQL that is adjacent to any key in the from or join clause based on a first regular expression; the selection submodule is used to determine whether to select the data table as the source table based on whether the data table adjacent to any key in the from or join clause is related to a year.

[0052] Therefore, parsing the data table corresponding to the keywords in the dynamic SQL using the first regular expression, and determining the source table based on the parsing result, can lead to obtaining the second source information related to the function in the dynamic SQL. This helps to improve the data information of the dynamic SQL, thereby improving the efficiency and accuracy of data analysis.

[0053] In some disclosed embodiments, the selection submodule includes a response unit for determining the selection as the source table in response to a data table adjacent to either the from or join key and a supplementary year of the year table.

[0054] Please see Figure 9 , Figure 9This is a schematic diagram of a framework of an embodiment of the data analysis device of this application. The data analysis device 90 includes a memory 91 and a processor 92 coupled to each other. The memory 91 stores program instructions, and the processor 92 is used to execute the program instructions to implement the steps in any of the above-described data analysis method embodiments. Specifically, the data analysis device 90 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablets, etc., and is not limited thereto.

[0055] Specifically, processor 92 controls itself and memory 91 to implement the steps in any of the above data analysis method embodiments. Processor 92 can also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 92 can be implemented using integrated circuit chips.

[0056] The above scheme, on the one hand, improves the accuracy of the target tables of each function by parsing the function information involved in the project under analysis. On the other hand, it improves the speed and comprehensiveness of function information acquisition by querying function information through a database tracking plugin or parsing function information through a first regular expression. Furthermore, it integrates the function information of the same function to determine the lineage of data in the project under analysis, which helps to improve the accuracy of the lineage of data in the project under analysis. Therefore, it can improve the efficiency of data analysis while improving the accuracy of data analysis.

[0057] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 100 stores program instructions 110 that can be executed by a processor, the program instructions 110 being used to implement the steps in any of the above-described data analysis method embodiments.

[0058] The above scheme, on the one hand, improves the accuracy of the target tables of each function by parsing the function information involved in the project under analysis. On the other hand, it improves the speed and comprehensiveness of function information acquisition by querying function information through a database tracking plugin or parsing function information through a first regular expression. Furthermore, it integrates the function information of the same function to determine the lineage of data in the project under analysis, which helps to improve the accuracy of the lineage of data in the project under analysis. Therefore, it can improve the efficiency of data analysis while improving the accuracy of data analysis.

[0059] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0060] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0061] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0063] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0064] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A data analysis method, characterized by, The method comprises the following steps: Based on the function information of the functions involved in the project to be analyzed, the target table of the functions is determined; Based on the database tracking plug-in, the function information of the functions is queried to obtain the first source information of the functions irrelevant to dynamic SQL, and based on the first regular expression, the function information of the functions is analyzed to obtain the second source information of the functions related to dynamic SQL; wherein, the first source information and the second source information at least include the source table of the functions, and the database tracking plug-in is determined based on the type of the database; Based on the first source information, the second source information and the target table belonging to the same function, the blood relationship of the data in the project to be analyzed is obtained.

2. The method of claim 1, wherein, The first source information further includes the source function of the function, and the blood relationship of the data in the project to be analyzed is obtained based on the first source information, the second source information and the target table belonging to the same function, comprising: Based on the source function corresponding to each function respectively, the hierarchical relationship of the source function corresponding to the function is determined, and based on the source table corresponding to each function respectively, the hierarchical relationship of the source table corresponding to the function is determined; Based on the source function and the hierarchical relationship of the function, and the target table, the source table and the hierarchical relationship of the function, the blood relationship of the data in the project to be analyzed is obtained.

3. The method of claim 2, wherein, Before the hierarchical relationship of the source function corresponding to the function is determined based on the recursive cycle of the source function corresponding to each function respectively, the method further comprises: The first source information and the second source information are de-duplicated.

4. The method of claim 1, wherein, Before the target table of the functions is determined based on the function information of the functions involved in the project to be analyzed, the method further comprises: Based on the plurality of metadata relationship tables of the project to be analyzed, the first information of the functions involved in the project to be analyzed is obtained; wherein, the first information at least includes the function identifier of the functions; Based on the function identifier of the functions and the function parameter table, the second information of the functions is obtained; Based on the first information and the second information of the functions, the function information of the functions involved in the project to be analyzed is integrated.

5. The method of claim 1, wherein, The function information of the functions includes the function content of the functions, and the target table of the functions is determined based on the function information of the functions involved in the project to be analyzed, comprising: Based on the second regular expression, the data table adjacent to the keyword representing the target editing operation in the function content of the functions is determined as the target table of the functions.

6. The method of claim 1, wherein, The function information of the functions includes the function content of the functions, and the first source information of the functions irrelevant to dynamic SQL is obtained based on the database tracking plug-in, comprising: Based on the database tracking plug-in, the result type of each data item in the function content of the functions is determined; Based on the result type of the data item, it is determined whether to select the data item as the source table or the source function to obtain the first source information.

7. The method of claim 1, wherein, The function information of the function is parsed based on the first regular expression to obtain second source information of the function related to the dynamic SQL, and the second source information comprises: A data table adjacent to any keyword of from or join in the dynamic SQL is parsed based on the first regular expression; Whether to select as the source table is determined based on whether the data table adjacent to any keyword of from or join is related to a year.

8. The method of claim 7, wherein, The determination of whether to select as the source table based on whether the data table adjacent to any keyword of from or join is related to a year comprises at least: In response to the data table adjacent to any keyword of from or join being related to a supplementary year of a year table, it is determined to select as the source table.

9. A data analysis device, characterized by, The data analysis method comprises a memory and a processor coupled to each other, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the data analysis method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The data analysis method comprises program instructions capable of being executed by a processor, and the program instructions are configured to implement the data analysis method of any one of claims 1 to 8.

Images