A scene-driven hierarchical fusion data bloodline analysis method

Abstract

The application discloses a scene-driven layered fusion data blood relationship analysis method, and belongs to the field of data management.The method is driven by a business scene, and after a data analysis scene is determined, the method analyzes a database, sorts out business information systems covered by the data analysis scene, and determines data blood relationship at the database level of the information system.According to primary key and foreign key information of the information system data table, the data blood relationship between the data tables is further sorted out, and the data blood relationship at the system database level and the data table level is expressed.The application proposes a novel data blood relationship analysis method based on the scene-driven and layered fusion strategy, realizes data fusion of multiple information systems, lays a foundation for data analysis and mining, and is suitable for multi-system big data application in which information system related documents are insufficient, data table and data field information are not standardized and clear, and data tracing and cleaning are difficult.

Description

TECHNICAL FIELD

[0001] The application discloses a scene-driven hierarchical fusion data blood relationship analysis method, belongs to the field of data science, and is used for a multi-system big data analysis application scene in which documents are deficient and design is not standard, leading to difficult data tracing and cleaning. BACKGROUND

[0002] Data blood relationship analysis is a comprehensive analysis of enterprise data flow and information flow, and is a core basis for realizing data integration and big data application. Whether data blood relationship can be accurately, clearly and completely described significantly affects the effectiveness of big data application (Jin Yong, 2019). Data blood relationship analysis technology has become a research focus in the field of big data analysis and has attracted more and more researchers' attention.

[0003] There are many previous reports on data blood relationship analysis. For example, Mao Ruixue et al. divide existing data blood relationship analysis methods into two categories: a workflow tracing method based on business information and a data tracing method based on query inversion (Mao Ruixue, 2012). The former analyzes data blood relationship by combing the business flow of an information system from the business layer, clarifying data flow and information flow. The latter analyzes data blood relationship by combing the data flow and information flow of a new system from the code layer with the help of information system design documents. The workflow tracing method and the data tracing method comb data flow and information flow from different aspects to realize data blood relationship analysis. However, due to the fact that workflows are divided by systems, systems are greatly modified, abandoned or newly built as the business develops, system personnel are lost, documents are scattered (Li Xufeng, 2016), and data table and field information are not clear (Pan Feng, 2016), the existing workflow tracing and data tracing methods cannot meet the needs of data blood relationship analysis in the big data background.

[0004] The application proposes a data blood relationship automatic analysis technology based on business scene driving and data hierarchical fusion. The technology analyzes business theme data by adopting a scene driving strategy based on business scene characteristics, and analyzes the blood relationship between systems and data tables by adopting a hierarchical fusion method, so as to achieve the purpose of fusing data between different systems. SUMMARY

[0005] The application aims to provide a scene-driven hierarchical fusion data blood relationship analysis method.

[0006] To achieve the above object, the technical scheme adopted by the application is as follows:

[0007] A scene-driven hierarchical fusion data blood relationship analysis method includes the following three steps:

[0008] Step 1: Extract the data analysis scene theme keywords, construct the SQL query statement, query the theme keyword master data code from the master data management system, and obtain the theme keyword master data information of the scene. Analyze the requirements and goals of the data analysis application from the business level, confirm through expert discussion, and extract the theme keywords of the data analysis application. Construct the SQL query statement, query the theme keyword master data code table from the master data management system, and obtain the theme keyword master data information set A of the scene.

[0009] Step 2: System-level blood relationship analysis, using the master data information of the data analysis scene theme keywords, combing the information management system covering the analysis scene theme, and clearly defining the system-level blood relationship based on the analysis scene theme. During the system-level blood relationship analysis process, the system-level blood relationship based on the analysis scene theme is clearly defined in the following way: if there is at least one data table in a certain information system, and the data value in a certain column of the data table contains elements in set A, and all other elements are null values, then it is considered that the system contains the data required by the data analysis scene; otherwise, it is considered that the data does not contain the data required by the data analysis scene. The set of information systems containing the data required by the data analysis scene is defined as netSys. In the case of lack of information system related documents, non-standard and unclear data table and data field information, the specific process of determining the set of information systems containing the data required by the data analysis scene is as follows:

[0010] 1) Step 1: Initialization, set netSys = φ, i = j = k = 1.

[0011] 2) Step 2: Determine whether all information systems S = [S1, …, S n ] have been processed, if all have been processed, end the analysis, the information systems in netSys set are the information systems containing the data required by the data analysis scene; otherwise, execute Step 3.

[0012] 3) Step 3: Determine whether all information systems have been combed, if there are still information systems that have not been combed, select the information system S i that has not been combed, set its database as DB i , its data table set as T i = [T i1 , …, T ij , …, T im ], set j = k = 1; otherwise, end the information system combing, and output the netSys set as the information system set containing the data required by the data analysis scene.

[0013] 4) Step 4: Determine whether the information system S iIs there any unprocessed data table? If there is any unprocessed data table, select the unprocessed data table T ij , whose field set is F ij . ij1 , …, F ijp ; otherwise, i = i + 1, go to Step 3.

[0014] 5) Step 5: judge whether the data values in the field F ijk contain elements in the set A and are all null values except the elements in the set A? If yes, put the information system S i into the set netSys, and i = i + 1, go to Step 3; otherwise, continue to judge whether all data columns of the data table T ij have been processed. If there are still unprocessed data columns, k = k + 1, go to Step 5; if there are still data columns to be judged, j = j + 1, go to Step 4.

[0015] Step 3: data layer blood relationship analysis, based on the primary key and foreign key information of the data table in the information management system, further sort out the blood relationship between the data tables in the information system related to the data analysis scene.

[0016] netSys represents a set of information systems related to the data analysis scene, and the blood relationship network between the data tables in the information system S i ∈netSys is constructed as follows:

[0017] 1) Step 1: set netField = φ, and the set of subject business management tables in the information system S i is T = [T i1 , …, T im ].

[0018] 2) Step 2: optionally select a data table T ij ∈T, and set the primary key set of the data table T ij as and the foreign key set as

[0019] 3) Step 3: judge whether FP = φ? If yes, go to Step 6; otherwise, for any primary key FP p ∈FP, judge whether the primary key FP p is the foreign key of any data table (assuming TP) in S i . If yes, execute Step 4, otherwise go to Step 5.

[0020] 4) Step 4: judge the triple <T ij , TP, FP pnetField?, if yes, go to Step 5; otherwise, add the triple <T ij , TP, FP p > to the set netField, and add the data table TP to the set T.

[0021] 5) Step 5: delete the primary key FP from the set FP p , and go to Step 3.

[0022] 6) Step 6: judge whether FQ = φ? If yes, execute Step 9; otherwise, for any foreign key FQ q ∈ FQ, judge whether the foreign key FQ q is the primary key of any data table (supposed to be TQ) in S i , if yes, execute Step 7, otherwise, go to Step 8.

[0023] 7) Step 7: judge whether the triple <TQ, T ij , FQ q > ∈ netField? If yes, go to Step 8; otherwise, add the triple <TQ, T ij , FQ q > to the list netField, and add the data table TQ to the set T.

[0024] 8) Step 8: delete the foreign key FQ from the set FQ q , and go to Step 6.

[0025] 9) Step 9: delete T ij from the set T, judge whether T = φ? If yes, execute Step 10; otherwise, go to Step 2.

[0026] 10) Step 10: output netField.

[0027] Compared with the prior art, the scene-driven hierarchical fusion data bloodline analysis method has the following excellent effects: the business flow and information flow information between systems are solved, the definition of the data table field does not need to be explicitly known, the related development documents of the information management system do not need to be used, only the data table record information needs to be analyzed, whether the information system contains the data required by the data analysis application can be determined, and the method is suitable for the big data application scene with complex business scenes, disordered data flow and incomplete information system document resources. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a specific process of the scene-driven hierarchical fusion data bloodline analysis method.

[0029] Figure 2For the embodiment, part of the data in the cloud platform.

[0030] Figure 3 For the embodiment, the "material management" main data coding part of the information.

[0031] Figure 4 For the embodiment, the system layer blood relationship of the "material management" theme data.

[0032] Figure 5 For the embodiment, the data layer blood relationship example in the raw material integrated management platform information system. DETAILED DESCRIPTION

[0033] The specific process of the present application will be further described in detail below in combination with the drawings:

[0034] Embodiment background: A domestic tobacco company's cigarette R&D data system platform, which integrates 29 billion structured data from 13 information systems such as raw material integrated management and main data management, and PB semi-structured and structured data. Cigarette R&D business is complex, the number of related information systems is large, the data structure and interface lack unified standards, and part of the documents are not detailed, making it difficult to directly sort and extract material management related data in each information system, affecting the effectiveness of data-driven material management business improvement. Part of the data in the cloud platform of the company is shown in the accompanying Figure 2

[0035] Embodiment purpose: In the "data-driven material management business improvement" data analysis application scenario, the application scenario-driven and data layer fusion strategy is adopted to determine the "material management" analysis theme keyword, and the blood relationship of the "material management" theme related data in the current platform is sorted out. 13 information systems.

[0036] Firstly, through the analysis of the data analysis scene "data-driven material management business improvement", the data analysis scene keyword "material coding" is determined, the "material coding" data from the main data management system in the platform is extracted, and the main data coding table of the "material management" theme data blood relationship is constructed, as shown in the accompanying Figure 3

[0037] Secondly, based on the hierarchical fusion data blood analysis method proposed in this paper, the system layer blood relationship of the "material management" theme data in the 13 information systems in the cloud platform is sorted out, and the analysis result is shown in the accompanying Figure 4 The analysis result shows that the main data system, raw material integrated management platform, lean R&D system, raw material base system and raw material quality control system in the cigarette R&D data system cover part of the "material management" business. Among them, the raw material procurement order table, raw material warehouse record table, outsourcing processing warehouse table and raw material allocation flow plan table of the raw material integrated management platform store part of the "material management" business data.​​

[0038] Finally, in several information management systems related to the theme, taking the raw material integrated management platform as an example, the data layer blood relationship analysis results of the "material management" theme in the system are as shown in the accompanying Figure 5 The data table layer blood relationship describes the theme data table and the topological relationship of the associated data table in the platform through the form of primary key and foreign key, and is used for analyzing the data table and business related to theme management in the platform. Figure 5 As shown in the accompanying

[0039] Through the analysis results of the embodiments, the scene-driven layered fusion data blood relationship analysis method can exhibit the correlation between the multi-business systems, data tables and data fields related to the "business theme" in the platform in a layered network manner, and can realize the data blood topological relationship analysis of the "business theme". From the analysis results, the method can better obtain the implicit data rules and characteristics between the systems. The scene-driven layered fusion data blood relationship analysis method can realize the blood topological relationship analysis of the data related to the business theme through the analysis of the data table and data field information, without the need to understand the business and system relationship or detailed document support, and can satisfy the data blood relationship analysis requirements under the big data background.

Claims

1. A scenario-driven hierarchical fusion data lineage analysis method, characterized in that: Driven by data analysis scenarios, hierarchical analysis of business information systems and data related to business scenarios includes the following steps: Step 1: Extract the data analysis scenario theme keywords, construct the SQL query statement, query the theme keyword master data code from the master data management system, and obtain the theme keyword master data information of the scene; The specific method of step 1 is as follows: From the business level, the demand and goal of the data analysis application are analyzed, and the theme keywords of the data analysis application are extracted through expert discussion confirmation; an SQL query statement is constructed to query the theme keyword master data coding table from the master data management system to obtain the theme keyword master data information set of the scene ; Step 2: System layer blood relationship analysis, using the master data information of the data analysis scenario theme keywords, combing the information management system covering the analysis scene theme, and clearly defining the system layer blood relationship based on the analysis scene theme; Step 3: Data layer blood relationship analysis, based on the primary key and foreign key information of the data table in the information management system, further combing the blood relationship between the data tables in the information system related to the data analysis scene; In the system layer blood relationship analysis process, the system layer blood relationship based on the analysis scene theme is determined in the following manner: if there is at least one data table in an information system, the data values in a column of the data table contain the elements in the set , and all values are except the elements in the set , it is indicated that the information system contains the data required by the data analysis scene; otherwise, it is considered that the information system does not contain the data required by the data analysis scene; the set of information systems containing the data required by the data analysis scene is defined as , and the system layer blood relationship analysis is performed according to the information systems contained in the set . 2.The scene-driven layered fusion data bloodline analysis method of claim 1, wherein, The specific method of step 2 is as follows: In the case of lack of information system related documents, non-standard and unclear data table and data field information, the specific process to determine the data information system set containing the data analysis scene is as follows: 1) Step 1: Initialization, setup , ; 2) Step2: judge all information systems in current set If all information systems have been processed, end the analysis, The information system in the set is the information system containing the data required by the data analysis scene; otherwise, Step3 is executed; 3) Step3: judge whether all information systems are combed, if there are information systems that have not been combed, select the information system that has not been combed , the database of which is , the data table set of which is , set ; otherwise, end the combing of information systems, and output The set of information systems required by the data analysis scene is the set of data required by the data analysis scene 4) Step4: judge whether there is unprocessed data table, if there is unprocessed data table, select unprocessed data table , whose field set is ;​ Otherwise, go to Step 3. 5) Step 5: Determine the field The data values ​​in the middle contain a set Elements in, and excluding sets Apart from the central element, are they all... If the value is positive, then the information system will be configured accordingly. Add to collection Inside, and Proceed to Step 3; otherwise, continue evaluating the data table. Have all data columns been processed? If there are still unprocessed data columns, then... Proceed to Step 5; if there are still unprocessed data columns, then... Proceed to Step 4. 3.The scene-driven layered fusion data bloodline analysis method of claim 1, wherein, The specific method of step 3 is as follows: representing association information system collection with data analysis scenario theme information system The construction process of the blood relationship network between data tables in the information system is as follows: 1) Step 1: Setup , information system in which the subject business management table set is ; 2) Step 2: Select any of the data tables Let the data table be... The primary key set is Its foreign key set is ; 3) Step3: judging , if yes, jump to Step6; otherwise, for any primary key , judging whether the primary key is the foreign key of any data table (assuming ) in , if yes, execute Step4, otherwise, jump to Step5; 4) Step4: judging the triple , if yes, jump to Step5; otherwise, add the triple to the set and add the data table to the set ; 5) Step5: Remove the primary key from the set and go to Step3;​ 6) Step6: judging , if yes, then executing Step9; otherwise, for any foreign key , judging whether the foreign key is the primary key of any data table (assuming ) in , if yes, then executing Step7, otherwise, jumping to Step8; 7) Step 7: judging the triple , if yes, jump to Step 8; Otherwise, the triple is added to the list and the data table is added to the set ; 8) Step 8: Remove foreign key from set , go to Step 6;​ 9) Step 9: ... From the set Delete, determine If so, proceed to Step 10; Otherwise, loop to Step2; 10) Step 10: Output ; The elements in the output list are triples , representing an edge in the kinship network , with the edge's parameters being ; through the same process described above, the data field kinship analysis of other information systems in can be completed.

Images