A data lineage mining method and device for a high-level enterprise application programming language

Abstract

The application discloses a data blood relationship mining method and device of an advanced enterprise application programming language, and the data blood relationship mining method comprises the following steps: performing word segmentation processing on the code of the advanced enterprise application programming language to obtain a mark list; wherein the word segmentation processing is performed in the minimum unit of words and symbols; the mark list is parsed according to the grammar rules of the Backus Normal Form to obtain a semantic expression list, and the semantic expression list comprises grammar keywords and grammar contents; the semantic expression list is formatted according to the grammar rules of the Backus Normal Form to obtain a semantic expression dictionary list; the semantic expression dictionary list is parsed, nodes are created, the blood relationship between different nodes is found and saved. The ABAP program code is subjected to fine-grained word segmentation processing, and semantic analysis is performed on the basis of the fine-grained word segmentation processing, so that the association relationship between data is finally obtained, and the blood relationship between nodes is determined.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method and apparatus for data lineage mining using a high-level enterprise application programming language. Background Technology

[0002] Advanced Business Application Programming (ABAP) is a high-level development language based on the widely used management software SAP platform. It is frequently used to help large companies build business applications for raw material management and financial accounting on mainframes. On the other hand, with massive amounts of data being collected, stored, and processed by digital production services, data lineage, as an efficient method for describing the data itself and its relationships, can effectively help enterprise staff analyze and model the relationships between data, such as their generation, calculation, flow, and referencing. This reduces the complexity of implementing business logic, application maintenance costs, and scalability costs. Therefore, mining ABAP code and its data lineage relationships is a highly commercially valuable task.

[0003] As a high-level programming language, ABAP has rich syntax rules, but there is currently no open-source method to fully parse the ABAP program source code to obtain the lineage relationship between data. Summary of the Invention

[0004] This application provides a data lineage mining method and apparatus for advanced enterprise application programming languages, which performs fine-grained word segmentation on ABAP program code, and performs semantic parsing based on this to finally obtain the association between data and determine the lineage between nodes.

[0005] This application provides a data lineage mining method using a high-level enterprise application programming language, including:

[0006] The code of a high-level enterprise application programming language is segmented to obtain a list of tags; the segmentation process uses words and symbols as the smallest units.

[0007] The list of tags is parsed according to the syntax rules of Backus paradigm to obtain a list of semantic expressions, which includes syntactic keywords and syntactic content;

[0008] The list of semantic expressions is formatted according to the syntax rules of Backus paradigm to obtain a list of semantic expression dictionaries;

[0009] Parse the list of semantic expression dictionaries, create nodes, find the lineage relationships between different nodes, and save them.

[0010] Preferably, the code of a high-level enterprise application programming language is segmented to obtain a list of tags, specifically including:

[0011] It iterates through the code of a high-level enterprise application programming language word by word and symbol by symbol, using the specified symbol as the token node, and combines the words between two adjacent specified nodes and the specified node that follows them into a tag or a comment, and adds it to the tag list. The specified symbols do not include newline characters.

[0012] Preferably, when encountering ambiguous symbols, the semantic distance of the operator is compared based on the context and the corpus, and the segmentation method corresponding to the smallest semantic distance is taken as the segmentation result.

[0013] Preferably, the list of tags is parsed according to the syntax rules of Backus paradigm to obtain a list of semantic expressions, specifically including:

[0014] For code containing terminators, the terminator is used as the semantic segmentation node. The code is decomposed into at least one semantic expression according to the syntax rules of Backus paradigm, and then added to the list of semantic expressions.

[0015] Preferably, the list of semantic expressions is formatted according to the syntax rules of Backus paradigm to obtain a list of semantic expression dictionaries, specifically including:

[0016] Traverse the list of semantic expressions and form a list of keyword dictionaries that includes multiple keyword dictionaries according to the syntax rules of Backus paradigm. The keyword dictionaries include descriptions of the syntax rules related to the keywords.

[0017] The semantic expression list is standardized according to the keyword dictionary list to form corresponding formatted expressions, which are then added to the semantic expression dictionary list.

[0018] Preferably, parsing the list of semantic expression dictionaries, creating nodes, and finding the lineage relationships between different nodes specifically includes:

[0019] The parsing path of the semantic expression dictionary is determined based on the types of the keywords in the semantic expression dictionary;

[0020] The semantic expression dictionary is parsed according to the parsing path, nodes corresponding to the types of keywords are created, and the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries are mined according to the meaning of the keywords, forming node-relationship-node triple relationships.

[0021] This application also provides a data lineage mining device for a high-level enterprise application programming language, including a word segmentation module, a semantic expression generation module, a semantic expression dictionary generation module, and a data lineage mining module;

[0022] The word segmentation module is used to segment code from advanced enterprise application programming languages ​​to obtain a list of tags; word segmentation is performed using words and symbols as the smallest units.

[0023] The semantic expression generation module is used to parse the tag list according to the syntax rules of Backus paradigm to obtain a semantic expression list, which includes syntax keywords and syntax content;

[0024] The semantic expression dictionary generation module is used to format the list of semantic expressions according to the syntax rules of Backus paradigm to obtain a list of semantic expression dictionaries;

[0025] The data lineage mining module is used to parse a list of semantic expression dictionaries, create nodes, find and save the lineage relationships between different nodes.

[0026] Preferably, the word segmentation module includes a node partitioning module, which is used to traverse the code of the high-level enterprise application programming language word by word and symbol by symbol. The specified symbol is used as the word segmentation node. The words between two adjacent specified nodes and the specified node after them are combined into a tag or a comment and added to the tag list. The specified symbols do not include newline characters.

[0027] Preferably, the semantic expression dictionary generation module includes a keyword dictionary list acquisition module and a standardization module;

[0028] The keyword dictionary list acquisition module is used to traverse the list of semantic expressions and form a keyword dictionary list that includes multiple keyword dictionaries according to the syntax rules of Backus paradigm. The keyword dictionary includes a description of the syntax rules related to the keywords.

[0029] The standardization module is used to standardize the list of semantic expressions according to the keyword dictionary list, forming corresponding formatted expressions, which are then added to the semantic expression dictionary list.

[0030] Preferably, the data lineage mining module includes a parsing path determination module and a parsing module;

[0031] The parsing path determination module is used to determine the parsing path of the semantic expression dictionary based on the type of the keywords in the semantic expression dictionary;

[0032] The parsing module is used to parse the semantic expression dictionary according to the parsing path, create nodes corresponding to the types of keywords, and mine the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries based on the meaning of the keywords, forming node-relationship-node triple relationships.

[0033] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0035] Figure 1 A flowchart of the data lineage mining method for the high-level enterprise application programming language provided in this application;

[0036] Figure 2 A flowchart illustrating a preferred embodiment of obtaining a list of semantic expressions without including terminators, as provided in this application;

[0037] Figure 3 This is a diagram illustrating the lineage of data in a data expression.

[0038] Figure 4 This is a diagram illustrating the data lineage of nodes within a data class.

[0039] Figure 5 A schematic diagram of the data lineage mining device for the advanced enterprise application programming language provided in this application. Detailed Implementation

[0040] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application.

[0041] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0042] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0043] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0044] This application provides a data lineage mining method and apparatus for advanced enterprise application programming languages, which performs fine-grained word segmentation on ABAP program code, and performs semantic parsing based on this to finally obtain the association between data and determine the lineage between nodes.

[0045] Example 1

[0046] like Figure 1 As shown, data lineage mining methods for advanced enterprise application programming languages ​​include:

[0047] S110: Perform word segmentation on the code of advanced enterprise application programming languages ​​to obtain a list of tags; where word segmentation is divided into words and symbols as the smallest units.

[0048] It should be noted that traditional word segmentation methods are difficult to achieve complete word segmentation, as shown in the following code:<l_record_no> =<_ys_sc_1>-record, traditional word segmentation methods divide this sentence code into (<l_record_no> The code consists of five parts: , =, <_ys_sc_1>, -, and -record. During subsequent data lineage analysis, the lineage of this code segment will be analyzed as "<l_record_no> The field is obtained by subtracting the "record" field from the "<_ys_sc_1>" field, and the actual semantics of this code is "<l_record_no> The field is equal to the "record" value of the "<_ys_sc_1>" field.

[0049] Based on the above considerations, in S110 of this application, the code of the high-level enterprise application programming language is traversed word by word and symbol by symbol. The specified symbol is used as the word segmentation node. The words between two adjacent specified nodes and the specified node after them are combined into a tag or a comment and added to the tag list. The specified symbol does not include the newline character.

[0050] As an example, S110 includes the following steps:

[0051] S1101: Initialize the tag capturer, the Chinese comment capturer, and the result list. The tag capturer is used to capture code items other than comment content, and the Chinese comment capturer is used to capture Chinese comment content. In this step, initialization means updating the tag capturer, the Chinese comment capturer, or the result list to an empty set.

[0052] S1102: Traverses the code of a high-level enterprise application programming language word by word and symbol by symbol, dividing and organizing words and symbols according to specified rules. Specifically:

[0053] 1. When a space is encountered, the tag capturer adds the captured word to the tag list and reinitializes the tag capturer. Thus, the word before the currently captured space forms a tag.

[0054] 2. When traversing to "," or "." (both are designated symbols), add the captured word and "," or "." to the token list, and reinitialize the Token Capturer. Thus, the word currently captured by the token capturer and "," or "." form a token.

[0055] 3. When encountering unambiguous operators such as "+", "*", " / ", and "=>" (which are designated symbols), add the captured words and operators to the token list and reinitialize the token capturer. Thus, these operators and the words currently captured by the token capturer form a token.

[0056] 4. When encountering ambiguous operators such as "-" and "->", compare the semantic distance of the various segmentation methods for the operator and take the segmentation method with the smallest semantic distance as the segmentation result.

[0057] When making a judgment, considering the context and the existing ABAP code content corpus, the judgment method can be simplified as follows:

[0058]

[0059] in:

[0060] Dis(Vk i )=∑(X ki -X kil ) 2

[0061] In this scenario, assuming there are n possible word segmentation methods in case of ambiguity, denoted as 1, 2, 3…n, and Vk is the k-th word segmentation method, each method consists of a list of words. i For the i-th word in the k-th word segmentation method, iterate through each word in each segmentation method and obtain the distance Dis(Vk) between each word and its context. i The sum of the distances between all words and their context in each segmentation method is the distance for that segmentation method. Result represents the final segmentation method selected after distance comparison.

[0062] Dis(Vk i In ), X ki X represents the position of the i-th word in the k-th word segmentation method. kil When X was initially defined ki The position of the l-th context (e.g., the preceding and following words) in the text. This method fully considers the contextual relationships of the code and avoids parsing ambiguous statements.

[0063] 5. When the newline character "\n" is encountered, it is added to the tag list, thus forming a tag.

[0064] 6. When the generated Chinese comment content is encountered, the Chinese comment capturer is used to identify and capture the corresponding Chinese characters. When the "*" (which belongs to the specified symbol) is encountered, the Chinese comment is added to the tag list and the Chinese comment capturer is initialized. Thus, the Chinese content currently captured by the Chinese comment capturer forms a comment.

[0065] Once all words and symbols in the code have been traversed and divided, a list of tags can be formed.

[0066] S120: Parse the list of tags according to the syntax rules of Backus Normal Form (BNF) to obtain a list of semantic expressions, which includes syntactic keywords and syntactic content.

[0067] In this step, the token list is segmented, grouping code items belonging to the same ABAP statement into the same semantic expression, ultimately generating several lists of semantic expressions. The first element of each semantic expression is a syntax keyword, followed by the syntax expression content. If comments exist, the last item in the semantic expression is a code comment.

[0068] Since different statements have different semantic relationships and can be parsed to reveal different data lineages, it is necessary to normalize the tag list.

[0069] Specifically, the semantic expression list, the expression list, and the prefix list are first initialized. Then, the tags and comments in the tag list are traversed sequentially. The expression list is used to temporarily store the words obtained during the formation of a semantic expression. After the requirements of the semantic expression are met, the obtained words are combined into a semantic expression and transmitted to the semantic expression list.

[0070] Specifically, as an example, for code containing a terminator, the terminator is used as a semantic segmentation node, and the code is decomposed into at least one semantic expression according to the syntax rules of Backus paradigm, and added to the list of semantic expressions.

[0071] Specifically, within a tag (e.g., a tag defining parameters), when the colon ":" is encountered, the word preceding ":" is added to the prefix list, and the expression list is made equal to the prefix list. The expression list is then initialized, and the iteration continues with the content following ":". When other characters (e.g., words) are encountered, the characters are directly added to the expression list.

[0072] When the symbol "," is encountered, the prefix list and the expression list (which stores the words between the symbols ":" and ",") are merged to form a semantic expression. The semantic expression is then added to the semantic expression list, and the expression list is initialized.

[0073] When the symbol "." is encountered, the prefix list and expression list are merged to form a semantic expression. The semantic expression is then added to the semantic expression list, and the expression list and prefix list are initialized.

[0074] Traditionally, word segmentation occurs line by line, with line endings marked by symbols such as commas, periods, or colons. However, in ABAP, some statements end with commas or periods, while others do not, continuing with a newline. For example:

[0075]

[0076]

[0077] This code snippet contains only a single terminator ".". If the traditional method is used, this code snippet will be parsed as a semantic expression, failing to correctly identify keywords and corresponding statements in the code. Code statements with different lineages will not be parsed separately, resulting in the loss of data lineage information.

[0078] For the reasons stated above, in this application, if Figure 2 As shown, for code that does not contain a terminator, a list of semantic expressions is obtained, including:

[0079] S1201: Treat the target word and the words between it and the nearest terminator as a semantic analysis sequence.

[0080] S1202: Determine whether the semantic analysis sequence satisfies the syntax rules of Backus normal form. If yes, execute S1206: Treat the semantic analysis sequence as a semantic expression; otherwise, execute S1203.

[0081] S1203: Perform semantic matching from right to left. Specifically, reduce the number of words from right to left, and at the same time determine whether the first sequence after reduction satisfies the grammatical rules, until a sequence that satisfies the grammatical rules is matched, to obtain at least one first semantic expression and at least one first remaining word.

[0082] Specifically, starting from the right side of the semantic analysis sequence, each time a word is removed, it is determined whether the resulting new sequence satisfies a certain grammatical description of the Backus paradigm. This process is repeated until the last word is matched or a sequence that satisfies the grammatical rules is found, thereby generating a semantic expression.

[0083] As an example, after obtaining at least one first semantic expression and at least one first residual word, the at least one first semantic expression is added to the semantic expression list.

[0084] To avoid excessive matching, i.e., when matching semantic expressions, the selectivity of the syntax affects the generation of the next semantic expression, preferably, if the semantic analysis sequence does not satisfy the grammatical rules of Backus paradigm, it also includes:

[0085] S1204: Perform semantic matching from left to right. Specifically, reduce the number of words word by word from left to right, while simultaneously determining whether the reduced second sequence satisfies the grammatical rules, until a sequence that satisfies the grammatical rules is matched, obtaining at least one second semantic expression and at least one second remainder word. The method for this step is the same as in S1203.

[0086] S1205: Add the semantic expression obtained by matching the lesser of the first and second remainder words to the semantic expression list.

[0087] Understandably, in this preferred embodiment, steps S1203 and S1204 can be performed in reverse order or simultaneously.

[0088] In this step, the word segmentation results are initially formatted and parsed according to grammatical rules, transforming the code statements containing data lineage into a list of simple semantic expressions.

[0089] S130: Format the list of semantic expressions according to the syntax rules of Backus paradigm to obtain a list of semantic expression dictionaries.

[0090] In this step, the list of semantic expressions is further formatted according to the ABAP syntax rules, and converted into a corresponding semantic expression dictionary to facilitate data lineage mining.

[0091] Specifically, the semantic expressions in the semantic expression list are traversed sequentially first, a keyword dictionary list is generated according to the syntax rules, and the semantic expression dictionary list is initialized.

[0092] Specifically, for keywords, the syntax rules described by Backus paradigm are used for normalization to generate a keyword dictionary list.

[0093] For example, in ABAP syntax, the syntax rules for describing data definitions are as follows:

[0094] DATA{var TYPE{{REF TO ref}|{{STANDARD TABLE OF table}|{TABLE OFtable}}|abap_type}[LENGTH len][DECIMALS dec]}

[0095] [VALUE val|{IS INITIAL+initial}]

[0096] [READ-ONLY+readonly].

[0097] In the description of this grammar rule, the content in "[]" represents optional items, the content in "{}" represents items that can be repeated 0 to infinitely, and "|" represents any item that can be selected from either side.

[0098] When parsing this grammar rule, the grammar rule is segmented into words according to spaces, and each word is traversed. Matching is performed according to the rules described above, and the result is converted into a corresponding keyword dictionary, as shown below:

[0099] {'DATA':['DATA',['var','TYPE',[('|',('|',['REF','TO','ref'],[('|',['STANDARD','TABLE','OF','table'],['TABLE','OF','table'])]),'abap_type')] ,('[',['LENGTH','len']),('[',['DECIMALS','dec'])],('[',['VALUE',('|','val',['IS','INITIAL','+initial'])]),('[',['READ-ONLY','+readonly'])]}

[0100] In the keyword dictionary, "DATA" is a keyword, and the content corresponding to this keyword is a description of the syntax rules corresponding to the keyword "DATA".

[0101] After parsing all the grammatical descriptions, a list of keyword dictionaries will be generated, consisting of multiple keyword dictionaries.

[0102] When traversing the list of semantic expressions obtained in S120, the list of semantic expressions is standardized according to the list of keyword dictionaries to form corresponding formatted expressions, which are used as semantic expression dictionaries and added to the list of semantic expression dictionaries.

[0103] Specifically, the first element (syntax keyword) of each semantic expression is extracted, and the corresponding syntax rule description is selected to format the semantic expression based on the generated keyword dictionary.

[0104] During formatting, for the remaining part of the semantic expression excluding the first element, a backtracking method is used for matching based on the obtained syntax rule description. That is, for the remaining part of the semantic expression excluding the first element, matching is performed starting from the first element of the syntax rule against the remaining part. When multiple matching schemes exist, one scheme is selected for matching. If this scheme matches completely, the corresponding matching result is returned; otherwise, the process returns to the initial selection state and another scheme is selected for matching, until the remaining part of the semantic expression completely matches the syntax rule. Successfully matched formatting expressions are added to the semantic expression dictionary list as part of the semantic expression dictionary.

[0105] The matching process specifically includes:

[0106] 1. Determine if the last item in the semantic expression list is a code comment. If so, establish a dictionary relationship between the comment content and the keyword, thus forming a semantic expression dictionary.

[0107] 2. Determine if the semantic expression contains the symbol "=". If so, use a recursive method to format the content on the left and right sides of the "=" sign, thereby forming a semantic expression dictionary.

[0108] For example, for ABAP code statements

[0109] Result = x*x + y.

[0110] This statement describes the relationship between the data field Result, data field x, and data field y. In this step, this line of code will be formatted as follows:

[0111] ('=',{'lhs':['result'],'rhs':['x','*','x','+','y']})

[0112] In this context, “=" is the keyword, “lhs” represents the content on the left side of the equals sign, and “rhs” represents the content on the right side of the equals sign.

[0113] 3. The definition of data often contains the initialization content of data nodes, for example, in ABAP code statements:

[0114] x type i

[0115] This step involves standardizing it as follows:

[0116] ('data',{'var':'x','abap_type':'i'})

[0117] In this context, "data" is the keyword, "var" is the variable name, and "abap_type" is the field type.

[0118] Similarly, for other ABAP semantic expressions, this step formats them into the corresponding semantic expression dictionary.

[0119] S140: Parse the list of semantic expression dictionaries, create nodes, find and save the lineage relationships between different nodes. Initialize nodes and mine data lineage based on the keywords and contents of the semantic expression dictionary.

[0120] Parsing a dictionary of semantic expressions involves creating nodes from a list of semantic expression dictionaries and finding the lineage relationships between different nodes. Specifically, this includes:

[0121] S1401: Determine the parsing path of the semantic expression dictionary based on the types of the keywords in the semantic expression dictionary.

[0122] S1402: Parse the semantic expression dictionary according to the parsing path, create nodes corresponding to the types of keywords, and mine the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries based on the meaning of the keywords, forming node-relationship-node triplet relationships.

[0123] For example, when the key encountered during iteration is "data", the dictionary is used to retrieve attributes such as data type, variable name, reference, whether it is unique, and whether it is read-only, and data nodes are created based on these attributes.

[0124] If the semantic expression dictionary contains a "table" field, a table data node is created based on the value of the "table" field. A reference relationship is established between the data node and the table data node, forming a data lineage.

[0125] If the semantic expression dictionary contains a "ref" field, a relationship is established between the data node and the class corresponding to ref based on the value of the "ref" field, forming a data lineage relationship.

[0126] If the semantic expression dictionary has a "value" field, create and initialize a constant node based on the value of the "value" field, and establish the relationship between the data node and the constant node to form a data lineage.

[0127] When the keyword encountered during iteration is "data_begin", the structure name is retrieved from the semantic expression dictionary, a structure data node is initialized and created, and the semantic expression dictionary of the sub-data in the structure is traversed, data nodes are initialized and created, and a subordinate relationship is established between the structure data node and the sub-data nodes in the structure, forming a data lineage relationship.

[0128] When the keyword encountered during iteration is "move", the data nodes of the original field and the target field of "move" in the semantic expression dictionary are obtained, and the assignment relationship between the two data nodes is established to form a data lineage relationship.

[0129] When the key encountered during iteration is "=", the data nodes corresponding to the left and right sides of the equal sign are recursively obtained respectively, and an equality relationship is established between the two data nodes to form a data lineage relationship.

[0130] When parsing complex arithmetic expressions, the complex operations are formatted according to the operation priority.

[0131] When processing operations between two data nodes, first recursively obtain the data nodes on the left and right sides of the operation, then obtain the operation type, and create a new data node named "left data name_operation name_right data name". Establish the relationship between this data node and the left and right data nodes, with the relationship name being the name of the operation, and return this newly created node. Figure 3 The diagram illustrates the data lineage in a data expression within ABAP code. Three data nodes, Data1, Data2, and Data3, undergo a certain operation to produce a Result data node. The operations between Data3 and Data2 have priority.

[0132] When the traversed keyword is "add", "subtract", "multiply", or "divide", the data nodes of the original field and the target field in the semantic expression dictionary are obtained. The relationship between the two data nodes is established based on the keyword type, forming a data lineage relationship.

[0133] When the keyword encountered during iteration is "if", retrieve all the conditional blocks and execution content of the if statement in the field, iterate through all conditional blocks, obtain the truth value of each conditional block, and recursively execute the corresponding content for each true conditional block to uncover the corresponding data lineage.

[0134] When the keyword encountered during iteration is "while" or "do", obtain the number of iterations or conditions, and explore the data lineage within the loop, adding the number of iterations to the data lineage content.

[0135] When the keyword encountered during traversal is "class", a new class data node is initialized and created. The subordinate relationship between the class node and the public and private data nodes in the class is established, forming a data lineage.

[0136] Figure 4 This illustrates the relationship between a data node class and its corresponding public and private data nodes. Within this data node class, there are data nodes with public attributes such as data1, data2, ..., dataK, and data nodes with private attributes such as dataA, dataB, ..., dataN.

[0137] After completing the initialization and creation of data nodes and the mining of data lineage, the data lineage is described as (node ​​n) i Relationship r i,j Node n j The relationship between triples is determined, the data lineage between data nodes is organized, and the nodes and relationships are saved.

[0138] Preferably, data nodes and the lineage relationships between data nodes are stored using a graph database (such as Neo4j).

[0139] Example 2

[0140] Based on the above-described data lineage mining method, this application also provides a data lineage mining apparatus using a high-level enterprise application programming language. For example... Figure 5 As shown, the data lineage mining device includes a word segmentation module 510, a semantic expression generation module 520, a semantic expression dictionary generation module 530, and a data lineage mining module 540.

[0141] The word segmentation module 510 is used to segment code from advanced enterprise application programming languages ​​to obtain a list of tags; the word segmentation process uses words and symbols as the smallest units.

[0142] The semantic expression generation module 520 is used to parse the tag list according to the syntax rules of Backus paradigm to obtain a semantic expression list, which includes syntax keywords and syntax content.

[0143] The semantic expression dictionary generation module 530 is used to format the list of semantic expressions according to the syntax rules of Backus paradigm to obtain a list of semantic expression dictionaries.

[0144] The data lineage mining module 540 is used to parse the list of semantic expression dictionaries, create nodes, find the lineage relationships between different nodes, and save them.

[0145] As an example, the word segmentation module 510 includes a node partitioning module 5101. The node partitioning module 5101 is used to traverse the code of the high-level enterprise application programming language word by word and symbol by symbol. Using the specified symbol as the word segmentation node, the word between two adjacent specified nodes and the specified node after it are combined into a tag or a comment and added to the tag list. The specified symbol does not include the newline character.

[0146] As another embodiment, the word segmentation module 510 also includes an ambiguous symbol processing module 5102. The ambiguous symbol processing module 5102 is used to compare the semantic distance of the operator's various word segmentation methods based on the context content and the corpus when an ambiguous symbol is encountered, and take the word segmentation method corresponding to the minimum semantic distance as the word segmentation result.

[0147] As one embodiment, the semantic expression dictionary generation module 530 includes a keyword dictionary list acquisition module 5301 and a standardization module 5302.

[0148] The keyword dictionary list acquisition module 5301 is used to traverse the list of semantic expressions and form a keyword dictionary list that includes multiple keyword dictionaries according to the syntax rules of Backus paradigm. The keyword dictionary includes a description of the syntax rules related to the keywords.

[0149] The standardization module 5302 is used to standardize the semantic expression list according to the keyword dictionary list, form the corresponding formatted expression, which is used as the semantic expression dictionary and added to the semantic expression dictionary list.

[0150] As an example, the data lineage mining module 540 includes a parsing path determination module 5401 and a parsing module 5402.

[0151] The parsing path determination module 5401 is used to determine the parsing path of the semantic expression dictionary based on the type of the keywords in the semantic expression dictionary.

[0152] The parsing module 5402 is used to parse the semantic expression dictionary according to the parsing path, create nodes corresponding to the types of keywords, and mine the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries based on the meaning of the keywords, forming node-relationship-node triplet relationships.

[0153] This application addresses the diverse and complex data flow patterns in SAP data-assisted decision-making platforms by proposing a method to parse and mine data lineage from ABAP code. This method can automatically uncover the flow relationships of various types of data within ABAP code, improving the efficiency of data tracing, analysis, and expansion. Furthermore, this application utilizes a graph database for data storage, facilitating visualization.

[0154] While some specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A data lineage mining method using a high-level enterprise application programming language, characterized in that, include: The code of a high-level enterprise application programming language is segmented to obtain a list of tags; the segmentation process uses words and symbols as the smallest units. The list of tags is parsed according to the syntax rules of Backus paradigm to obtain a list of semantic expressions, which includes syntactic keywords and syntactic content; The semantic expression list is formatted according to the syntax rules of Backus paradigm to obtain a semantic expression dictionary list; Parse the list of semantic expression dictionaries, create nodes, find the lineage relationships between different nodes, and save them.

2. The data lineage mining method for high-level enterprise application programming languages ​​according to claim 1, characterized in that, The code of a high-level enterprise application programming language is segmented to obtain a list of tags, specifically including: The code of the advanced enterprise application programming language is traversed word by word and symbol by symbol. Using the specified symbol as the word segmentation node, the words between two adjacent specified nodes and the specified node after them are combined into a tag or a comment and added to the tag list. The specified symbol does not include newline characters.

3. The data lineage mining method for high-level enterprise application programming languages ​​according to claim 2, characterized in that, When encountering ambiguous symbols, the semantic distance of the operator is compared based on the context and the corpus, and the segmentation method with the smallest semantic distance is taken as the segmentation result.

4. The data lineage mining method for high-level enterprise application programming languages ​​according to claim 1, characterized in that, The list of tags is parsed according to the syntax rules of Backus paradigm to obtain a list of semantic expressions, specifically including: For code containing a terminator, the terminator is used as a semantic segmentation node. The code is decomposed into at least one semantic expression according to the syntax rules of Backus paradigm, and then added to the list of semantic expressions.

5. The data lineage mining method for high-level enterprise application programming languages ​​according to claim 1, characterized in that, The semantic expression list is formatted according to the Backus paradigm's grammar rules to obtain a semantic expression dictionary list, specifically including: Traverse the list of semantic expressions and form a list of keyword dictionaries that includes multiple keyword dictionaries according to the syntax rules of Backus paradigm. The keyword dictionaries include descriptions of syntax rules related to the keywords. The semantic expression list is standardized according to the keyword dictionary list to form corresponding formatted expressions, which are then added to the semantic expression dictionary list.

6. The data lineage mining method for high-level enterprise application programming languages ​​according to claim 1, characterized in that, Parse the list of semantic expression dictionaries, create nodes, and find the lineage relationships between different nodes, specifically including: The parsing path of the semantic expression dictionary is determined based on the type of the keywords in the semantic expression dictionary; The semantic expression dictionary is parsed according to the parsing path, nodes corresponding to the types of the keywords are created, and the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries are mined according to the meaning of the keywords, forming node-relationship-node triplet relationships.

7. A data lineage mining device for a high-level enterprise application programming language, characterized in that, It includes a word segmentation module, a semantic expression generation module, a semantic expression dictionary generation module, and a data lineage mining module; The word segmentation module is used to segment the code of a high-level enterprise application programming language to obtain a list of tags; wherein, the word segmentation process uses words and symbols as the smallest units. The semantic expression generation module is used to parse the tag list according to the syntax rules of Backus paradigm to obtain a semantic expression list, which includes syntax keywords and syntax content; The semantic expression dictionary generation module is used to format the semantic expression list according to the syntax rules of Backus paradigm to obtain a semantic expression dictionary list; The data lineage mining module is used to parse the semantic expression dictionary list, create nodes, find the lineage relationships between different nodes, and save them.

8. The data lineage mining device for high-level enterprise application programming languages ​​according to claim 7, characterized in that, The word segmentation module includes a node partitioning module, which is used to traverse the code of the advanced enterprise application programming language word by word and symbol by symbol. Using a specified symbol as a word segmentation node, the word between two adjacent specified nodes and the specified node after it are combined into a tag or a comment and added to the tag list. The specified symbol does not include a newline character.

9. The data lineage mining device for high-level enterprise application programming languages ​​according to claim 7, characterized in that, The semantic expression dictionary generation module includes a keyword dictionary list acquisition module and a standardization module; The keyword dictionary list acquisition module is used to traverse the semantic expression list and form a keyword dictionary list including multiple keyword dictionaries according to the syntax rules of Backus paradigm. The keyword dictionary includes a description of the syntax rules related to the keywords. The semantic expression list is standardized according to the keyword dictionary list to form corresponding formatted expressions, which are then added to the semantic expression dictionary list.

10. The data lineage mining device for high-level enterprise application programming languages ​​according to claim 7, characterized in that, The data lineage mining module includes a parsing path determination module and a parsing module; The parsing path determination module is used to determine the parsing path of the semantic expression dictionary based on the type of the keywords in the semantic expression dictionary; The parsing module is used to parse the semantic expression dictionary according to the parsing path, create nodes corresponding to the types of the keywords, and mine the data lineage relationships between different nodes within the semantic expression dictionary and between different semantic expression dictionaries according to the meaning of the keywords, forming node-relationship-node triplet relationships.

Images