Data reference chain construction and utilization method based on data analysis application

Abstract

The invention relates to a data reference chain construction and utilization method based on data analysis application, and belongs to the technical field of computer information technology related data analysis application. The method comprises the following steps: constructing a data reference chain record table; defining a data reference chain expression format; generating data reference chain initial node information in the initial layer data query result set; splicing subsequent node information of the data reference chain in the intermediate layer data query result set; storing the finally spliced data reference chain information and the corresponding application ID in a data reference chain record table in a terminal layer display data set; and when in use, obtaining a data reference relationship related to the application or the data table by querying the data reference chain record table. Through the method, the operation and maintenance efficiency is greatly improved, the skill requirement for operation and maintenance personnel is reduced, great convenience is brought to operation and maintenance work of data analysis application, and the method is easy to apply and popularize.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis applications related to computer information technology, and specifically relates to a data reference chain construction and utilization method based on data analysis applications. Background Art

[0002] In enterprise big data application scenarios, a data analysis application usually requires multi-layer data query, aggregation, statistics and other processing. From the terminal display, the final data generation process and dependency relationships are concealed. As data analysis applications are continuously built and put into practical use, it becomes difficult to trace the data reference relationship of data analysis applications and identify the application scope of specific data tables referenced during operation and maintenance. Typical examples are as follows:

[0003] Example 1: When the terminal data of a data analysis application is wrong (or questioned by the user), it is necessary to trace back the reference data layer by layer. Usually, the designer (or developers with the same technical level and familiarity) can only perform data reference troubleshooting through code design logic. This operation and maintenance condition is obviously difficult to promote and maintain.

[0004] Case 2: When a business data table at the ODS layer of the data warehouse needs to be adjusted due to changes in the requirements of the related business system, it is necessary to evaluate whether all data analysis applications that reference it are affected. If one can only determine whether there is a reference to the table based on the design logic of the data analysis application, then this is almost an impossible, time-consuming and labor-intensive task.

[0005] Therefore, how to overcome the shortcomings of the existing technology is an urgent problem to be solved in the field of computer information technology related data analysis application technology. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and to provide a method for constructing and utilizing a data reference chain based on a data analysis application, thereby improving the efficiency and quality of operation and maintenance by constructing and utilizing a data reference chain corresponding to the data analysis application.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A method for constructing and utilizing a data reference chain based on a data analysis application comprises the following steps:

[0009] Step 1: Construct a data reference chain record table;

[0010] Step 2: Define the data reference chain expression format;

[0011] Step 3: Generate the initial node information of the data reference chain in the initial layer data query result set;

[0012] Step 4: Join the subsequent node information of the data reference chain in the middle-layer data query result set;

[0013] Step 5: In the terminal layer display data set, the final spliced ​​data reference chain information and the corresponding application ID are stored in the data reference chain record table;

[0014] Step 6: When using, obtain the data reference relationship related to the application or data table by querying the data reference chain record table.

[0015] Further, preferably, in step 1, the data reference chain record table includes application ID, application name, application description, and data reference chain information fields; the data reference chain record table is used to store a constructed data reference chain information record corresponding to a data analysis application.

[0016] Furthermore, preferably, in step 2, the data reference chain expression format is nested using brackets; the innermost brackets are the first layer, the outermost brackets are the last layer, the content of the first layer is the name of the initial layer of data table, the content of the first layer is the name of the second layer of data table, and so on.

[0017] Further, preferably, in step 3, the initial layer data query result set is the result data set obtained by the initial layer data query integration operation executed according to the data retrieval logic when building the data analysis application; the specific method of generating the initial node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code, and the value of the field is set according to the data reference chain expression format, specifically set to the table name of the initial layer of data table referenced by the data analysis logic.

[0018] Further, preferably, in step 4, the intermediate layer data query result set is a data query result set generated in all data processing layers except the initial layer and the terminal layer during the construction of the data analysis application;

[0019] The specific method of splicing the subsequent node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code. In terms of field value construction, it is necessary to first inherit the data reference chain information in the result set generated by the previous level, wrap it as a whole with brackets, and then combine it with other data tables referenced by this level according to the data reference chain expression format to generate new data reference chain information.

[0020] Furthermore, preferably, in step 5, the terminal layer display data set is the result set provided after the last intermediate layer data is processed, which is used for terminal display; the application ID of the current data analysis application and the complete data reference chain information are stored in the data reference chain record table created in step 1 for subsequent retrieval.

[0021] Further, preferably, in step 6, there are two utilization methods, specifically:

[0022] (1) Retrieve the data reference chain record information corresponding to a specific application according to the application ID, and parse all referenced data tables and hierarchical dependencies;

[0023] (2) Construct a search based on the table name to obtain all data reference chain records and corresponding application information that contain the table in the data application chain, thereby determining the data analysis application scope and impact depth affected by the specific table.

[0024] In the operation and maintenance of data analysis applications, it is typically necessary to trace the various levels of data sources referenced by the terminal display data, or to identify the scope of data analysis applications affected by changes in a data table structure definition. To handle these problems conveniently and efficiently, it is first necessary to obtain the reference dependency relationship of the data analysis application on the data. Therefore, the present invention provides a method for constructing and utilizing a data reference chain based on a data analysis application.

[0025] Working principle of the present invention: In the process of building a data analysis application, the hierarchical data reference relationship actually exists, but it does not exist in the form of entity data. It must be manifested through technical means before it can be used. Since the material reflecting the data reference (specifically the referenced data table in this solution) is directly and clearly defined in the data processing logic of each layer in the process of building the data analysis application, the construction of the "data reference chain" is the simplest and most reasonable way to synchronously insert the "data reference chain" into the data processing actions at each level in the process of building the data analysis application. Through layer-by-layer superposition and splicing, the construction of the "data reference chain" is completed at the same time as the construction of the data analysis application is completed. After storage, it can be used as entity data, which greatly facilitates the operation and maintenance of the data analysis application.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] In the operation and maintenance work of data analysis applications, it is typically necessary to trace the references to various levels of data sources based on the terminal display data, or to identify the scope of data analysis applications affected by changes in a certain data table structure definition. Faced with this problem, at present, it is usually only possible for technicians familiar with design and code to track from the application code level, which is very time-consuming and labor-intensive, and the threshold for operation and maintenance personnel is high, making operation and maintenance work more difficult. After implementing the scheme of the present invention, the above problems can be solved by simple information retrieval. According to the operation and maintenance practice effect, compared with before the implementation of the scheme, the time spent on processing operation and maintenance events is shortened by about 50% to 70%, and the personnel who can participate in the operation and maintenance event processing are expanded from technicians familiar with application design to general operation and maintenance personnel who have mastered relevant skills, so that the rapid response capability of event processing is increased by more than 5 times (related to the available human resources of the operation and maintenance team), greatly improving the operation and maintenance efficiency and reducing the skill requirements for operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for constructing and utilizing a data reference chain based on data analysis applications of the present invention;

[0029] Figure 2 The figure is a flow chart of an application example of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with embodiments.

[0031] Those skilled in the art will appreciate that the following examples are only used to illustrate the present invention and should not be considered to limit the scope of the present invention. If no specific techniques or conditions are specified in the examples, the techniques or conditions described in the literature in the art or the product specifications are used. If the manufacturer of the materials or equipment used is not specified, they are all conventional products that can be purchased.

[0032] Example 1

[0033] like Figure 1 As shown, a method for constructing and utilizing a data reference chain based on a data analysis application includes the following steps:

[0034] Step 1: Construct a data reference chain record table;

[0035] Step 2: Define the data reference chain expression format;

[0036] Step 3: Generate the initial node information of the data reference chain in the initial layer data query result set;

[0037] Step 4: Join the subsequent node information of the data reference chain in the middle-layer data query result set;

[0038] Step 5: In the terminal layer display data set, the final spliced ​​data reference chain information and the corresponding application ID are stored in the data reference chain record table;

[0039] Step 6: When using, obtain the data reference relationship related to the application or data table by querying the data reference chain record table.

[0040] Example 2

[0041] like Figure 1 As shown, a method for constructing and utilizing a data reference chain based on a data analysis application includes the following steps:

[0042] Step 1: Construct a data reference chain record table;

[0043] Step 2: Define the data reference chain expression format;

[0044] Step 3: Generate the initial node information of the data reference chain in the initial layer data query result set;

[0045] Step 4: Join the subsequent node information of the data reference chain in the middle-layer data query result set;

[0046] Step 5: In the terminal layer display data set, the final spliced ​​data reference chain information and the corresponding application ID are stored in the data reference chain record table;

[0047] Step 6: When using, obtain the data reference relationship related to the application or data table by querying the data reference chain record table.

[0048] In step 1, the data reference chain record table includes application ID, application name, application description, and data reference chain information fields; the data reference chain record table is used to store a constructed data reference chain information record corresponding to a data analysis application.

[0049] In step 2, the data reference chain expression format is nested in brackets; let the innermost brackets be the first layer, and the outermost brackets be the last layer. The content of the first layer is the name of the initial layer of data table, and the content of the first layer is the name of the second layer of data table, and so on.

[0050] In step 3, the initial layer data query result set is the result data set obtained by the initial layer data query integration operation executed according to the data retrieval logic when building the data analysis application; the specific method of generating the initial node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code, and the value is set according to the data reference chain expression format, specifically set to the name of the initial layer of data table referenced by the data analysis logic.

[0051] In step 4, the intermediate layer data query result set is the data query result set generated in all data processing layers except the initial layer and the terminal layer during the construction of the data analysis application;

[0052] The specific method of splicing the subsequent node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code. In terms of field value construction, it is necessary to first inherit the data reference chain information in the result set generated by the previous level, wrap it as a whole with brackets, and then combine it with other data tables referenced by this level according to the data reference chain expression format to generate new data reference chain information.

[0053] In step 5, the terminal layer display data set is the result set provided after the last intermediate layer data is processed, which is used for terminal display; the application ID of the current data analysis application and the complete data reference chain information are stored in the data reference chain record table created in step 1 for subsequent retrieval.

[0054] In step 6, there are two ways to use it, specifically:

[0055] (1) Retrieve the data reference chain record information corresponding to a specific application according to the application ID, and parse all referenced data tables and hierarchical dependencies;

[0056] (2) Construct a search based on the table name to obtain all data reference chain records and corresponding application information that contain the table in the data application chain, thereby determining the data analysis application scope and impact depth affected by the specific table.

[0057] Example 3

[0058] like Figure 1 As shown, the method for constructing and utilizing a data reference chain based on a data analysis application includes the following steps:

[0059] Step 1: Construct a data reference chain record table;

[0060] Step 2: Define the data reference chain expression format;

[0061] Step 3: Generate the initial node information of the data reference chain in the initial layer data query result set;

[0062] Step 4: Join the subsequent node information of the data reference chain in the middle-layer data query result set;

[0063] Step 5: In the terminal layer display data set, the final spliced ​​data reference chain information and the corresponding application ID are stored in the data reference chain record table;

[0064] Step 6: When using, obtain the data reference relationship related to the application or data table by querying the data reference chain record table;

[0065] The above method steps are further described as follows:

[0066] In step 1, the "data reference chain record table" is a common relational database data table, which mainly includes application ID (record unique identifier), application name, application description, and data reference chain information fields. "Data reference chain" information refers to a hierarchical combination expression of several data table names referenced in the process of building a data analysis application described in a specific format. The "data reference chain record table" is used to store a constructed data reference chain information record corresponding to a data analysis application, and is also used to provide retrieval and utilization of data reference relationships related to applications or data tables. You can retrieve the corresponding data reference chain information based on the application ID, and then obtain the data table and reference hierarchy relationship referenced in the application; you can also search based on the data table name to obtain all data analysis applications that have references to the data table, and clarify the scope of influence of the data table.

[0067] In step 2, the "data reference chain expression format" is used as a construction specification for constructing data reference chain information. The definition format mainly considers the convenience of describing the hierarchical combination relationship of data references, parsing and identification, and storage. The definition format example of the present invention is: "((T1; T2); T3; T4); T5", brackets are used to distinguish the reference levels, the nesting depth of brackets reflects the reference level depth (or data table dependency order), and the same-level data tables are separated and arranged with semicolons. The order of front and back can imply the primary and secondary relationship of the data tables (optional).

[0068] Specifically:

[0069] (1) In the format example, T1, T2, T3, T4, and T5 represent the names of five data tables, indicating that the data analysis application references the data in these five tables.

[0070] (2) Brackets are used in the format examples to express a data reference hierarchy. Brackets can be nested, and the nesting depth reflects the depth of the data reference hierarchy. The table reference in the innermost brackets represents the first and lowest level of data analysis operations. The data analysis operations contained in the outer brackets are based on the results of the data analysis operations in the inner brackets. Brackets can be omitted at the outermost level of the expression. Data tables of the same level contained in a pair of brackets are separated by semicolons.

[0071] (3) The innermost brackets (T1; T2) in the format example indicate that the data analysis application is based on the T1 and T2 tables.

[0072] (4) In the format example ((T1; T2); T3; T4) means that the data analysis applies an intermediate level (level 2) of data analysis, based on the results of the (T1; T2) analysis and T3 and T4.

[0073] (5) In the format example, ((T1; T2); T3; T4); T5 is the third layer of the data analysis application, and is also the final layer of data analysis operation. It is composed of the analysis result of ((T1; T2); T3; T4) and T5. At this point, the data reference chain is completed.

[0074] In step 3, the "initial layer data query result set" refers to the result data set obtained from the initial (most basic) layer of data query integration operation performed according to the data retrieval logic when building a data analysis application. The specific method of generating the initial node information of the data reference chain is to add a field for storing "data reference chain information" in the output column based on the original query SQL code, and its value is the expression of the table name referenced by the query logic described in the "data reference chain expression format". Since the referenced table name itself is provided in the original query SQL code, the value does not need to be calculated and is spelled into the SQL statement as a direct quantity. This expansion method is lossless to the original query SQL logic and will not change the data results originally required to be obtained in the original code logic.

[0075] In step 4, the "middle layer data query result set" refers to the data query result set generated in all data processing layers except the initial layer and the terminal layer during the construction of the data analysis application. The number of middle layers of different data analysis applications may be different. In the scheme of the present invention, the processing performed by each middle layer is the same, so it is described as a step, but when executed, this step should be repeatedly executed according to the number of middle layers. When splicing the subsequent node information of the data reference chain in the middle layer, a field for storing the "data reference chain information" is added to the query SQL code output column in the same way as step 3. The difference lies mainly in the field value construction. It is necessary to first inherit the "data reference chain information" in the result set generated by the previous level, wrap it as a whole with brackets, and then combine it with other data tables referenced by this layer according to the "data reference chain expression format" to generate new "data reference chain information". In this way, as the middle layers are superimposed and progressive, the corresponding "data reference chain information" is also continuously spliced ​​and updated until it is complete.

[0076] In step 5, the "terminal layer display data set" is the result set provided after the last intermediate layer data is processed, which can be used for terminal display. The "data reference chain information" contained in the data set is spliced ​​when the entire data processing process is completed, reflecting the data reference relationship of the entire process. At this time, the application ID (unique identifier) ​​of the current data analysis application and the complete "data reference chain information" are stored in the "data reference chain record table" created in step 1 for subsequent retrieval.

[0077] Step 6 indicates obtaining the data reference relationship by querying the "data reference chain record table" for utilization. The utilization methods are mainly as follows: 1. Retrieve the "data reference chain record" information corresponding to a specific application according to the application ID, and parse all referenced data tables and hierarchical dependencies. 2. Construct a search according to the table name to obtain all "data reference chain records" and corresponding application information containing the table in the data application chain, so as to determine the data analysis application scope and impact depth affected by the specific table.

[0078] Application Examples

[0079] like Figure 2 As shown, a method for constructing and utilizing a data reference chain based on a data analysis application includes the following steps:

[0080] In step 1, create D4 [Data Reference Chain Record Table];

[0081] In step 2, define the data reference chain expression format and determine the splicing and combination rules of the data reference chain information.

[0082] In step 3, execute the P1 [Generate data reference chain initial node information] action to add the data reference chain field output in D1 [Initial layer data query result set].

[0083] In step 4, execute the P2 [join data reference chain subsequent node information] action to add the data reference chain field output in D2 [middle layer data query result set].

[0084] In step 5, execute the P3 [Generate Data Reference Chain Record] action to store the data reference chain field output value contained in D3 [Terminal Layer Display Data Set] in combination with the application ID into D4 [Data Reference Chain Record Table].

[0085] In step 6, the user searches D4 [Data Reference Chain Record Table] through P4 [Query Data Reference Relationship] operation to obtain the required data reference information.

[0086] like Figure 2 This is a schematic diagram of the application structure of the method described in the application example of the present invention.

[0087] Among them, D1: represents the initial layer data query result set, which includes the fields required for the original data query logic output and the extended fields for storing data reference chain information.

[0088] D2: represents the middle-layer data query result set, which includes the fields required for the original data query logic output and the extended fields for storing data reference chain information.

[0089] D3: represents the data set displayed at the terminal layer, which includes the fields required for the original data query logic output and the extended fields for storing data reference chain information.

[0090] D4: represents a relational database table used to store data reference chain records.

[0091] P1: represents the processing action of generating the initial node information of the data reference chain and assigning it to the data reference chain field in D1.

[0092] P2: represents the processing action of concatenating the subsequent node information of the data reference chain and assigning it to the data reference chain field in D2.

[0093] P3: represents the action of obtaining the complete data reference chain information from the data reference chain field in D3 and storing it in the D4 table to generate a data reference chain record.

[0094] Specifically:

[0095] Step 1 Create D4 [Data Application Chain Record Table], as shown in Table 1:

[0096] Table 1

[0097]

[0098] The expression format of the "data reference chain" defined in step 2 is as follows: "((T1; T2); T3; T4); T5". Brackets are used to represent hierarchies, and the nesting of brackets reflects hierarchical dependencies. Tables at the same level are separated by semicolons.

[0099] In step 3, the DATA_REF_CHAIN ​​(data reference chain information) output in the result set is constructed as: "table1; table2";

[0100] The original query SQL code example of the initial layer is:

[0101]

[0102] The output result set is shown in Table 2.

[0103] Table 2

[0104]

[0105] In step 4, the query statement code transformation method is similar to the above step 3. The first intermediate layer data processing is based on the initial layer result set, and table3 and table4 are newly referenced. The DATA_REF_CHAIN ​​(data reference chain information) output in the result set is constructed as: "(table1; table2); table3; table4";

[0106] The second middle-layer data processing is based on the first middle-layer result set and references table5. The DATA_REF_CHAIN ​​(data reference chain information) output in the result set is constructed as: "((table1; table2); table3; table4); table5". In this example, the data set required for terminal display has been obtained, and the data reference chain information construction is complete.

[0107] In step 5, the data reference chain field content is obtained from the D3 data set and written into the D4 data table for storage. An example is shown in Table 3.

[0108] Table 3

[0109]

[0110] In step 6, you can search and use the "Data Reference Chain Record Table" by appid or table name, as shown below:

[0111] Select * from [data application chain record table] where APPID = 'dataApp001' or select * from [data application chain record table] where DATA_REF_CHAIN ​​LIKE '%table3%'

[0112] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for constructing and utilizing a data reference chain based on a data analysis application, characterized in that: The steps include: Step 1: Construct a data reference chain record table; Step 2: Define the data reference chain expression format; Step 3: Generate the initial node information of the data reference chain in the initial layer data query result set; Step 4: Splice the subsequent node information of the data reference chain in the middle-layer data query result set; Step 5: In the terminal layer display data set, the final spliced ​​data reference chain information and the corresponding application ID are stored in the data reference chain record table; Step 6: When using, obtain the data reference relationship related to the application or data table by querying the data reference chain record table.

2. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 1, characterized in that: In step 1, the data reference chain record table includes application ID, application name, application description, and data reference chain information fields; the data reference chain record table is used to store a constructed data reference chain information record corresponding to a data analysis application.

3. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 1, characterized in that: In step 2, the data reference chain expression format is nested in brackets; let the innermost brackets be the first layer, and the outermost brackets be the last layer. The content of the first layer is the name of the initial layer of data table, and the content of the first layer is the name of the second layer of data table, and so on.

4. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 3, characterized in that: In step 3, the initial layer data query result set is the result data set obtained by the initial layer data query integration operation executed according to the data retrieval logic when building the data analysis application; the specific method of generating the initial node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code, and the value is set according to the data reference chain expression format, specifically set to the name of the initial layer of data table referenced by the data analysis logic.

5. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 1, characterized in that: In step 4, the intermediate layer data query result set is the data query result set generated in all data processing layers except the initial layer and the terminal layer during the construction of the data analysis application; The specific method of splicing the subsequent node information of the data reference chain is to add a field for storing the data reference chain information in the output column based on the original query SQL code. In terms of field value construction, it is necessary to first inherit the data reference chain information in the result set generated by the previous level, wrap it as a whole with brackets, and then combine it with other data tables referenced by this level according to the data reference chain expression format to generate new data reference chain information.

6. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 1, characterized in that: In step 5, the terminal layer display data set is the result set provided after the last intermediate layer data is processed, which is used for terminal display; the application ID of the current data analysis application and the complete data reference chain information are stored in the data reference chain record table created in step 1 for subsequent retrieval.

7. The method for constructing and utilizing a data reference chain based on a data analysis application according to claim 1, characterized in that: In step 6, there are two ways to use it, specifically: (1) Retrieve the data reference chain record information corresponding to a specific application according to the application ID, and parse all referenced data tables and hierarchical dependencies; (2) Construct a search based on the table name to obtain all data reference chain records and corresponding application information that contain the table in the data application chain, so as to determine the scope and depth of data analysis applications affected by the specific table.

Images