Data blood relationship mining method based on ABAP program analysis

Abstract

The application discloses a data blood relationship mining method based on ABAP program analysis, and the main design concept of the application is that, when data development is carried out in a SAP data auxiliary decision-making platform, data blood needs to be acquired in real time, and a wide coverage range is required, therefore, the ABAP language is analyzed, data flow transfer and subordination in ABAP code are mined based on semantic rules, then, context understanding operation is carried out on the mined data flow transfer and subordination, and data blood relationships are constructed accordingly, finally, data nodes and blood relationships between the nodes are mined from an expression dictionary and stored in a graph database, so that visual display can be carried out. The application utilizes machine to automatically mine various data flow transfer relationships in ABAP code, improves data analysis efficiency, and has the characteristics of high efficiency, wide coverage, and strong practicability, so that data traceability can be conveniently tracked, data value evaluation and safety control can be carried out.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for mining data lineage based on ABAP program parsing. Background Technology

[0002] In response to the informatization efforts of the tobacco and cigarette industry, most tobacco companies have adopted advanced data management tools and processing methods. These include the ABAP high-level language and data lineage analysis. ABAP is a widely used high-level development language for the SAP management software platform, frequently used to help large companies build platforms for raw material management and financial accounting management on mainframes. Furthermore, with massive amounts of data being collected, stored, and processed by digital production services, data lineage, as an efficient method for describing data itself and its relationships, can effectively help enterprise staff analyze and model the relationships between data, such as their generation, calculation, flow, and reference. This reduces the complexity of business logic implementation, application maintenance costs, and scalability costs.

[0003] Against this backdrop, exploring the ABAP language and its data lineage is a task with significant application value.

[0004] However, this task still faces many challenges: 1) As a high-level development language, ABAP has rich syntax rules, and there is no open-source method to fully parse the ABAP program source code; 2) SAP programs often contain intricate business logic; 3) Existing methods lack redundancy removal, strong correlation and high-dimensional visualization of data lineage nodes. Summary of the Invention

[0005] In view of the above, the present invention aims to provide a data lineage mining method based on ABAP program understanding to solve the aforementioned technical problems.

[0006] The technical solution adopted in this invention is as follows:

[0007] This invention provides a data lineage mining method based on ABAP program understanding, including:

[0008] Based on all the data flows and hierarchical relationships involved in the SAP platform, the corresponding rules are described in the form of regular expressions;

[0009] Based on the regular expression, the ABAP program involved is parsed using regular expression matching to mine and match the data flow and subordinate relationships within it.

[0010] Contextual understanding is performed on the data flow and subordinate relationships mined in the ABAP program, and the pre-defined specific data relationships are extended based on the context.

[0011] Data lineage is constructed based on the extended processing results, and the data lineage relationship is visualized through a preset graph structure.

[0012] In at least one possible implementation, parsing the ABAP program using regular expression matching includes:

[0013] Initialize the preset first result list;

[0014] Obtain the ABAP code needed to mine blood relations, and segment the code based on spaces;

[0015] Based on several regular expressions obtained from the grammatical description rules based on data flow and subordinate relationships, construct several finite automata.

[0016] The finite automaton is used to automatically match the segmented ABAP code, and the mined data relationship matching results are added to the first result list.

[0017] In at least one possible implementation, the automatic matching of the segmented ABAP code includes: describing the data relationships involving data flow and subordination in the segmented ABAP code in a preset dictionary format.

[0018] In at least one possible implementation, the extended processing includes:

[0019] Traverse the first result list, store the relationships between data using an adjacency matrix, and create a data dictionary to mark the data nodes that have appeared;

[0020] When the result type of the iteration is an expression, retrieve the corresponding data field;

[0021] Determine whether a data field contains a preset first-target operator;

[0022] If included, then perform secondary word segmentation based on the first target operator;

[0023] Iterate through the results of the secondary word segmentation and, in conjunction with the precedence of the first target operator, parse the content of the current operation expression;

[0024] Based on the parsing result of the current operation expression, a new data field is created to represent the relationship between the data nodes on both sides of the first target operator;

[0025] Delete the results of type expression from the first result list and add the new data field to the first result list.

[0026] In at least one possible implementation, the extended processing includes:

[0027] Traverse the first result list, store the relationships between data using an adjacency matrix, and create a data dictionary to mark the data nodes that have appeared;

[0028] When the result type encountered during iteration is a conditional statement, retrieve the corresponding data field and condition field.

[0029] The content of the condition field is segmented into words to obtain a condition list;

[0030] Iterate through the condition list and determine whether each item in the condition list includes a preset second target operator;

[0031] If included, then based on the data dictionary, determine whether the left and right sides of the second target operator are data nodes;

[0032] If it is a data node, then establish a new relationship between the condition field and the data field;

[0033] Delete the results of type "judgment" from the first result list, and add the new relationship to the first result list.

[0034] In at least one possible implementation, the step of constructing data lineage based on the extended processing results and visually representing the data lineage relationship through a preset graph structure includes:

[0035] Iterate through the expanded processing results and convert the data fields of each result into data nodes;

[0036] Based on the relational information of each result, the relationships between the data nodes are established, and the data nodes with relationships are connected in a directed graph.

[0037] In at least one possible implementation, describing the corresponding rule as a regular expression includes:

[0038] According to the syntax rules of ABAP, the definition relationship, reference relationship, subordination relationship, operation expression relationship, or judgment relationship of data flow are described as regular expressions that can extract corresponding elements.

[0039] In at least one possible implementation, the data lineage mining method further includes: organizing the obtained data lineage relationships and deleting redundant data relationships.

[0040] The main design concept of this invention addresses the challenge of real-time data lineage acquisition and broad coverage required during data development in the SAP data-assisted decision-making platform. It proposes parsing the ABAP language and mining data flow and dependency relationships within the ABAP code based on semantic rules. Next, contextual understanding operations are performed on the mined data flow and dependency relationships to construct data lineage relationships. Finally, data nodes and their inter-node lineage relationships are mined from an expression dictionary and stored in a graph database for visualization. This invention utilizes machine learning to automatically mine various data flow relationships in ABAP code, improving data analysis efficiency. It features high efficiency, broad coverage, and strong practicality, facilitating data traceability, value assessment, and security management. Attached Figure Description

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0042] Figure 1 A flowchart of a data lineage mining method based on ABAP program parsing provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the results of parsing the overall data lineage in ABAP code, as provided in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram illustrating the results of selecting data flow conditions for parsing ABAP code, as provided in an embodiment of the present invention. Detailed Implementation

[0045] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0046] This invention proposes an embodiment of a data lineage mining method based on ABAP program understanding, specifically, as follows: Figure 1 As shown, it includes:

[0047] Step S1: Organize all data flows and hierarchical relationships involved in the SAP platform operation, and describe the corresponding rules in the form of regular expressions.

[0048] The rules described here, based on the data flow and hierarchical relationships involved in operations within the SAP platform, are expressed in the form of regular expressions, including:

[0049] According to the syntax rules of ABAP, the definition relationship, reference relationship, subordination relationship, operation expression relationship, or judgment relationship of data flow are described as regular expressions that can extract corresponding elements.

[0050] Specifically, according to the syntax rules of ABAP, the definition relationship of data is described as a regular expression that can be mined out (name, type, initial value);

[0051] For example, in ABAP syntax, data is typically defined as:

[0052] DATA:data_name TYPE:type_name VALUE:value_data.

[0053] To extract data definitions, it can be described as a regular expression:

[0054] DATA:(.+?)TYPE:(.+?)VALUE:(.+?).

[0055] Using the regular expression above, you can mine data name (data_name), data type (type), and initial value (VALUE);

[0056] Similarly, according to ABAP syntax rules, the data referencing relationship can be described as a regular expression that can be mined as (data, reference, data); the data dependency relationship can be described as a regular expression that can be mined as (data, dependency, data); the data operation expression relationship can be described as a regular expression that can be mined as (data, operator, data); and the judgment relationship involving data flow can be described as a regular expression that can be mined as (data, condition, data).

[0057] The examples above are common data flow and data hierarchy descriptions in the SAP platform. If there are other data lineages that need to be mined, they can be described using regular expressions according to the syntax rules of ABAP.

[0058] Step S2: Based on the regular expression, parse the ABAP program involved by regular expression matching, and mine and match the data flow and subordinate relationships.

[0059] In practice, this can be achieved through the following specific process:

[0060] Initialize the preset first result list;

[0061] Obtain the ABAP code needed to mine blood relations, and segment the code based on spaces;

[0062] Based on several regular expressions obtained from grammatical description rules based on data flow and subordinate relationships, construct several finite automata.

[0063] Based on the constructed finite automata, the segmented ABAP code is automatically matched, and the mined data relationship matching results are added to the first result list.

[0064] Specifically, the automatic matching of segmented ABAP code described here includes: describing the data relationships involving data flow and subordination in the segmented ABAP code as a preset dictionary format.

[0065] For example, in the dictionary format {data1:“”,data2:“”,relation:“”,condition:“”,type:“”}, the “data1” field stores the name of the first matched data; the “data2” field stores the name of the second matched data; the “relation” field stores the relationship between the matched data 1 and data 2; and the “condition” field stores the condition required for data 1 and data 2 to satisfy a certain relationship. If this field is empty, it means there is no condition. After completing the dictionary description, these matched relationship results can be added to the first result list.

[0066] Step S3: Perform contextual understanding on the data flow and subordinate relationships mined in the ABAP program, and expand the pre-set specific data relationships based on the context.

[0067] In practice, based on the previous example, the extended processing can be implemented using the following two specific procedures:

[0068] (1) Process 1

[0069] Traverse the first result list, store the relationships between data using an adjacency matrix, and create a data dictionary to mark the data nodes that have appeared;

[0070] When the result type (i.e., the type field) encountered during iteration is an expression, the corresponding data fields are retrieved, such as the "data1" field and the "data2" field.

[0071] Determine whether the data field contains a preset first target operator, such as whether the "data1" field and the "data2" field contain the ["+", "-", "*", " / "] operator;

[0072] If included, then perform secondary word segmentation based on the first target operator;

[0073] The results of the secondary word segmentation are traversed, and the content of the current operation expression is parsed in combination with the priority of the first target operator. The content of the operation expression here includes: variable name, field type, and the content on both sides of the preset target operator.

[0074] Based on the parsing result of the current operation expression, a new data field is created to represent the relationship between the data nodes on both sides of the first target operator. The name of this new data field can be defined as "operator left node name_operator_operator right node name", that is, to establish the relationship between this node and the left and right data nodes. The relationship name is the name of the corresponding first target operator. Afterwards, it can be converted into the aforementioned dictionary form, and the new data field is added to the first result list.

[0075] (2) Process Two

[0076] Traverse the first result list, store the relationships between data using an adjacency matrix, and create a data dictionary to mark the data nodes that have appeared;

[0077] When the result type encountered during iteration (i.e., the type field) is a condition, the corresponding data field and condition field are retrieved, such as the "data1" field, the "data2" field, and the "condition" field.

[0078] The content of the condition field is segmented into words to obtain a condition list. For example, the content of the condition field is segmented into words according to ["AND", "OR", "NOT"] to obtain a Condition list.

[0079] Iterate through the condition list and determine whether each item in the condition list includes a preset second target operator. For example, iterate through the Condition list and determine whether each item in the Condition list includes the ["==", ">=", "<="] operator.

[0080] If included, then based on the data dictionary, determine whether the left and right sides of the second target operator are data nodes;

[0081] If it is a data node, a new relationship is established between the condition field and the data field, and the relationship name is determined. It can then be converted to the aforementioned dictionary form, and the new data field is added to the first result list. For example, a relationship is established between the condition node and the data node "data2", with the relationship name "depend", and after conversion to the aforementioned dictionary form, it is added to the first result list.

[0082] After the above two processes are completed, that is, after the processing of the results of type expression and judgment is completed, the results of type expression and judgment in the first result list are deleted, and the remaining part is used as the second result list.

[0083] Step S4: Construct data lineage based on the extended processing results, and visualize the data lineage relationship through a preset graph structure.

[0084] Building upon the previous example, we iterate through the second result list, converting the data fields of each result, such as "data1" and "data2", into data nodes. The data lineage will be reflected in the relationships between these nodes. Based on the "relation" field of each result, we establish the relationships between the aforementioned data nodes and connect them using a directed graph.

[0085] The structure of the data node can be represented as follows:

[0086] {Id:'NodeId', Name:'NodeName', Value:'Node V alue', Type:'NodeType'}

[0087] Here, `Id` is a randomly generated, unique string, while the `Type` of data nodes is mainly divided into `Constant`, `Variable`, `Class`, `Attribute`, and `Operate`. Therefore, the conversion method into data nodes can be referenced as follows:

[0088] When the content of the second result list is a number or a string, a data node is created for that number or string. The Name and Value of the data node are both the actual value of the number or string, and the Type of the data node is Constant.

[0089] When iterating through the second result list, if the content of the data field is a data field, search according to the name of the data field. If an existing node is found, return the node directly; if no existing node is found, create a new data node with the Name of the new data node being the actual name of the data field and the Type of the new data node being Variable, and return the newly created data node.

[0090] To facilitate understanding the representation of directed graphs, combined with Figure 2As shown, when using graph structures to represent data lineage, specific data tasks are introduced. By demonstrating the generation process of relevant data involved in the specific task, the relationships between data nodes are used to describe the flow of data. When converting "judgment" type data lineage relationships into graph structures for representation, the following example is used... Figure 3 As shown, in Figure 3 In the table, data 1 has different numerical values ​​(such as value 1, value 2, ..., value n) under different conditions, and the conditions depend on field 1 of table 1. The data flow relationship under different conditions can be clearly seen through the above representation. Furthermore, it can be added that in actual operation, after completing the initial creation of data nodes and data lineage mining, the data lineage relationship can also be represented as (node ​​n... i Relationship r i，j Node n j The system identifies triple relationships and organizes the data lineage between data nodes, storing the nodes and relationships in a graph database (such as Neo4j).

[0091] Finally, it should be noted that the obtained data lineage relationships are organized, and redundant data relationships are removed. Specifically, each node in the graph is traversed, and the in-degree of each node is checked. If the in-degree of a node is greater than 2, it is checked whether this node has in-degree edges with two different relationships. If so, the in-degree edges of this node are obtained, and the shortest path along the different in-degree edges is obtained starting from this node (Dijkstra's algorithm can be used). The edge with the smallest in-degree in the shortest path is then deleted.

[0092] The main design concept of this invention addresses the challenge of real-time data lineage acquisition and broad coverage required during data development in the SAP data-assisted decision-making platform. It proposes parsing the ABAP language and mining data flow and dependency relationships within the ABAP code based on semantic rules. Next, contextual understanding operations are performed on the mined data flow and dependency relationships to construct data lineage relationships. Finally, data nodes and their inter-node lineage relationships are mined from an expression dictionary and stored in a graph database for visualization. This invention utilizes machine learning to automatically mine various data flow relationships in ABAP code, improving data analysis efficiency. It features high efficiency, broad coverage, and strong practicality, facilitating data traceability, value assessment, and security management.

[0093] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0094] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for mining data lineage based on ABAP program parsing, characterized in that, include: Based on all the data flows and hierarchical relationships involved in the SAP platform, the corresponding rules are described in the form of regular expressions; Based on the regular expression, the ABAP program involved is parsed by regular expression matching, and the data flow and subordinate relationships are mined and matched. The mined data relationship matching results are added to the first result list. Contextual understanding is performed on the data flow and subordinate relationships mined in the ABAP program, and the pre-defined specific data relationships are extended based on the context. The extended processing includes: traversing the first result list, storing the relationships between data using an adjacency matrix, and creating a data dictionary to mark the data nodes that have appeared; when the traversed result type is an expression, obtaining the corresponding data field; determining whether the data field includes a preset first target operator; if so, performing secondary word segmentation based on the first target operator; traversing the secondary word segmentation results and parsing the content of the current operation expression based on the priority of the first target operator; creating a new data field to represent the relationship between data nodes on both sides of the first target operator based on the parsing result of the current operation expression; deleting the results of type expression in the first result list and adding the new data field to the first result list; Alternatively, the extended processing includes: traversing the first result list, storing the relationships between data using an adjacency matrix, and creating a data dictionary to mark the data nodes that have appeared; when the traversed result type is a judgment, obtaining the corresponding data field and condition field; segmenting the content of the condition field to obtain a condition list; traversing the condition list and determining whether each item in the condition list includes a preset second target operator; if it includes, determining whether the left and right sides of the second target operator are data nodes according to the data dictionary; if they are data nodes, establishing a new relationship between the condition field and the data field; deleting the results of type judgment in the first result list and adding the new relationship to the first result list; Data lineage is constructed based on the extended processing results, and the data lineage relationship is visualized through a preset graph structure.

2. The data lineage mining method based on ABAP program parsing according to claim 1, characterized in that, The parsing of the ABAP program using regular expression matching includes: Initialize the preset first result list; Obtain the ABAP code needed to mine blood relations, and segment the code based on spaces; Based on several regular expressions obtained from the grammatical description rules based on data flow and subordinate relationships, construct several finite automata. The finite automaton is used to automatically match the segmented ABAP code, and the mined data relationship matching results are added to the first result list.

3. The data lineage mining method based on ABAP program parsing according to claim 2, characterized in that, The automatic matching of the segmented ABAP code includes: describing the data relationships involving data flow and subordination in the segmented ABAP code in a preset dictionary format.

4. The data lineage mining method based on ABAP program parsing according to claim 1, characterized in that, The step of constructing data lineage based on the extended processing results and visually representing the data lineage relationship through a preset graph structure includes: Iterate through the expanded processing results and convert the data fields of each result into data nodes; Based on the relational information of each result, the relationships between the data nodes are established, and the data nodes with relationships are connected in a directed graph.

5. The data lineage mining method based on ABAP program parsing according to claim 1, characterized in that, The description of the corresponding rule in the form of a regular expression includes: According to the syntax rules of ABAP, the definition relationship, reference relationship, subordination relationship, operation expression relationship, or judgment relationship of data flow are described as regular expressions that can extract corresponding elements.

6. The data lineage mining method based on ABAP program parsing according to any one of claims 1 to 5, characterized in that, The data lineage mining method further includes: organizing the obtained data lineage relationships and deleting redundant data relationships.

Images