A method and device for algorithmic lineage management based on feature fusion

Abstract

The present invention discloses a method for algorithm lineage management based on feature fusion, comprising: step 1, obtaining algorithm information and corresponding source code data, parsing the source code data, obtaining a control flow graph and an abstract syntax tree corresponding to the node information in the control flow graph; step 2, obtaining the node information corresponding to the source code data in the control flow graph, and calculating and obtaining a node embedding vector; step 3, obtaining a sequence identifier corresponding to the source code data in the abstract syntax tree, and calculating and obtaining a word embedding vector; step 4, fusing the node embedding vector and the word embedding vector to obtain a corresponding fused feature vector; step 5, visually analyzing the fused feature vector based on the algorithm information and the fused feature vector, and obtaining an algorithm lineage map. The present invention also provides an algorithm lineage management device. The method provided by the present invention solves the problem that the source of the algorithm is difficult to trace and manage by fusing the multi-dimensional features of the algorithm.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to an algorithmic lineage management method and device based on feature fusion. Background Art

[0002] In recent years, with the rapid development of the software field, more and more application systems, websites, etc. are connected and accessed by humans through the Internet. The software supply chain has become more complex and diverse. The complex software supply chain will introduce a series of security issues, such as serious algorithm island phenomenon, difficulty in tracing the source of algorithms, and difficulty in ensuring the quality of algorithms. The quality of software systems has become a research focus at home and abroad, and code security has gradually become a hot topic in academia and industry.

[0003] To facilitate algorithm traceability analysis and management, it's necessary to visualize the algorithm's lineage relationships, clearly presenting them to users. However, among existing visualization analysis methods, lexical representations offer simplicity and speed, but ignore the structural characteristics of the algorithm. Syntax-based representations address the shortcomings of lexical representations but ignore the dependencies between algorithm contexts. Semantic representations address the shortcomings of grammar-based representations and facilitate understanding of program execution, but the resulting models are extremely complex.

[0004] Patent document CN112783857A discloses a data lineage management method, apparatus, electronic device, and storage medium. The method includes: obtaining a workflow from target metadata and storing the workflow at a target node; the workflow includes at least one process component; parsing the process components in the workflow to determine the source table and target table for each process component, as well as the relationship between the source field in the source table and the target field in the target table; and managing data lineage using the target node's attribute information and branching diagram. This method addresses the entire data process, from data generation to data expiration, requiring real-time updates to the data stream content.

[0005] Patent document CN110363985A discloses a traffic data analysis method, apparatus, storage medium, and device. The method includes: acquiring traffic data; classifying and calculating the traffic data using a lineage management algorithm to construct a thematic database containing traffic data of different categories; analyzing and calculating the traffic data using a multidimensional analysis algorithm to construct a thematic database containing traffic data of different dimensions; and generating traffic data analysis results based on the thematic and thematic databases. This method classifies all data using the lineage management algorithm and uses the classification results to perform multidimensional predictions, thereby improving the overall effectiveness of the final decision.

[0006] Traditional data lineage is used to study the relationship between tables in the data flow. In the prediction process, it focuses on the functional relationship between data. The algorithm lineage proposed in this application is aimed at the unified management of the algorithm's syntax and functions. However, at this stage, the algorithm lineage is only presented through grammatical expression, but the logical order between the corresponding source code data cannot be well expressed, which affects the difficulty of subsequent source code data management. Summary of the Invention

[0007] In order to solve the above problems, the present invention provides an algorithm lineage management method based on feature fusion, which solves the problem that the source of the algorithm is difficult to trace and manage by fusing the multi-dimensional features of the algorithm.

[0008] An algorithmic lineage management method based on feature fusion, comprising:

[0009] Step 1: Acquire algorithm information and corresponding source code data, parse the source code data, and obtain a control flow graph and an abstract syntax tree corresponding to the node information in the control flow graph;

[0010] Step 2: Obtain the node information corresponding to the source code data in the control flow graph, and calculate the node embedding vector corresponding to the node information;

[0011] Step 3: Obtain a sequence identifier corresponding to the source code data in the abstract syntax tree, and calculate a word embedding vector corresponding to the sequence identifier;

[0012] Step 4: Fuse the node embedding vector obtained in step 2 and the word embedding vector obtained in step 3 to obtain a fused feature vector corresponding to the source code data;

[0013] Step 5: Based on the algorithm information obtained in step 1 and the fusion feature vector obtained in step 4, the fusion feature vector corresponding to the source code data is visualized and analyzed to obtain an algorithm lineage map for tracing and managing source code data.

[0014] The present invention extracts features from the abstract syntax tree containing grammatical information and the control flow graph containing semantic information, splices and fuses the extracted features to obtain fusion features that have both semantic information and grammatical information, and finally uses algorithm information as attributes to perform visual analysis on the fusion features corresponding to the algorithm to obtain an algorithm lineage map. The visual algorithm lineage map can facilitate staff to trace the source analysis and management of source code data.

[0015] Specifically, the algorithm information in step 1 includes the original file path, model name, function name, programming language, code comments, input and output types, and functional information.

[0016] Preferably, after obtaining the control flow graph in step 1, the control flow graph needs to be preprocessed, and the preprocessing includes splicing and replacing node information, and feature extraction is performed on the preprocessed control flow graph.

[0017] Specifically, the expression of the control flow graph is as follows:

[0018] G=(V,E)

[0019] Where V represents a set of nodes, each of which contains at least one statement, and E represents a set of directed edges. A control flow graph composed of statements has only one entry and exit node.

[0020] Specifically, the splicing and replacement of node information has the following specific process: based on the function call relationship of the algorithm, the control flow graph node corresponding to the source code data is replaced with the control flow graph corresponding to the called function, that is, the parent node of the node is connected to the entry node of the replaced control flow graph, and the child node of the node is connected to the exit node of the replaced control flow graph, until a more complete control flow graph is constructed.

[0021] Preferably, after obtaining the abstract syntax tree in step 1, the abstract syntax tree is preprocessed. The preprocessing includes normalizing the variables in the program and adding a global auto-increment number. Feature extraction is performed on the preprocessed abstract syntax tree. The normalization is to unify the formats of variables of the same type and replace them according to their types. At the same time, variables with decimals are uniformly used. <double>It indicates that the dimensions of the final extracted features are guaranteed to be the same, which is convenient for the subsequent splicing process.

[0022] Specifically, the sequence identifier of the abstract syntax tree is generated by traversing the preprocessed abstract syntax tree from the root node according to the depth-first search algorithm (DFS), and its specific expression is as follows:

[0023] Seq AST ={ident1,ident2,...,ident n }

[0024] In the formula, ident n Represents the symbol or variable name in the nth AST node.

[0025] Specifically, the node embedding vector in step 2 is based on the asymmetric transitivity of the control flow graph, and the HOPE algorithm is used for feature extraction. The length of the node embedding vector is 4.

[0026] Specifically, the word embedding vector in step 3 uses the Word2Vec algorithm to extract features from the sequence identifier, and the length of the word embedding vector is 50.

[0027] Specifically, the fused feature vector in step 4 is obtained by concatenating the word embedding vectors of all sequence identifiers in the current node and the node embedding vector of the node. Its specific expression is as follows:

[0028]

[0029] Where, E CFG Node embedding vector representing source code data, The word embedding vector representing the kth identifier within the node.

[0030] Specifically, the visualization analysis of step 5 is based on the fusion feature vector, uses the Euclidean distance to calculate the similarity between the source code data, classifies the source code data with similarity less than the threshold, and uses the algorithm information as the retrieval attribute to construct an algorithm lineage map.

[0031] The present invention also provides an algorithm lineage management device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the above-mentioned algorithm lineage management method based on fusion features is executed in the computer memory; when the computer processor executes the computer program, the following steps are implemented: inputting the source code data of the algorithm to be retrieved, searching through a visualized algorithm lineage map, and outputting an algorithm data set that has an affinity relationship with the target algorithm.

[0032] Specifically, the affinity relationship refers to the existence of identical or similar source code data between the target algorithm and the corresponding algorithm data.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The node information of the control flow graph is used to represent the semantic information, and the sequence identifier of the abstract syntax tree is used to represent the grammatical information, thereby obtaining a fusion feature that has both grammatical and semantic features.

[0035] (2) Preprocess the data of the control flow graph and the abstract syntax tree to ensure that the features of two different dimensions can be integrated together.

[0036] (3) Generate a visual source code data lineage map, which can quickly obtain the algorithm database that has an affinity with the target algorithm, thereby facilitating the subsequent daily maintenance of the source code data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of an algorithmic bloodline management method based on feature fusion provided by the present invention. DETAILED DESCRIPTION

[0038] Due to the rapid development of computer algorithms, more and more systems and websites will be built using many different algorithms. However, algorithms with different syntax and semantics will cause a series of security issues in the subsequent daily management and maintenance process: algorithm island phenomenon, difficulty in tracing the source of algorithms, difficulty in ensuring algorithm quality, etc.

[0039] In order to facilitate the traceability analysis and management of algorithms and corresponding source code data, this embodiment proposes an algorithm lineage management method based on feature fusion, such as Figure 1 As shown, including:

[0040] Step 1: Obtain algorithm information and corresponding source code data, parse the source code data, and obtain a control flow graph and an abstract syntax tree corresponding to the node information in the control flow graph;

[0041] Preprocess the control flow graph. According to the function call relationship of the algorithm, replace the control flow graph node corresponding to the source code data with the control flow graph corresponding to the called function. That is, connect the parent node of the node to the entry node of the replaced control flow graph, and connect the child node of the node to the exit node of the replaced control flow graph until a more complete control flow graph is constructed. Perform feature extraction on the preprocessed control flow graph. The expression is as follows:

[0042] G=(V,E)

[0043] Where V represents a set of nodes, each of which contains at least one statement, and E represents a set of directed edges. A control flow graph composed of statements has only one entry and exit node.

[0044] Preprocess the abstract syntax tree, which includes normalizing the variables in the program and adding a global auto-increment number. Then, traverse the preprocessed abstract syntax tree from the root node using the depth-first search algorithm (DFS) to generate a node sequence identifier. Normalization is to unify the formats of variables of the same type according to their type, and to use a unified format for variables with decimals. <double>Indicates that, the specific expression for generating a node sequence identifier is as follows:

[0045] Seq AST ={ident1,ident2,...,ident n }

[0046] In the formula, ident n Represents the symbol or variable name in the nth AST node.

[0047] Step 2: Obtain the node information corresponding to the source code data in the control flow graph. Based on the asymmetric transitivity of the control flow graph, the HOPE algorithm is used for feature extraction, where the node embedding vector E CFG The length of is 4.

[0048] Step 3: Obtain the sequence identifier corresponding to the source code data in the abstract syntax tree, and use the Word2Vec algorithm to extract features from the sequence identifier. The word embedding vector E ident The length is 50.

[0049] Step 4: The final fusion feature is obtained by concatenating the word embedding vectors of all sequence identifiers in the current node and the node embedding vector of the node. The specific expression is as follows:

[0050]

[0051] Where, E CFG Node embedding vector representing source code data, The word embedding vector representing the kth identifier within the node.

[0052] Step 5: Based on the algorithm information obtained in step 1 and the fusion feature vector obtained in step 4, the fusion feature vector corresponding to the source code data is converted to As an entity, the similarity S index is used as a relationship, and the algorithm information is used as the attribute A and the value a to construct an algorithm lineage map for traceability management of source code data:

[0053]

[0054]

[0055] In the formula, Represents the node fusion feature vector corresponding to source data i, Represents the node fusion feature vector corresponding to source code data j, S ij A represents the similarity of the fusion feature vectors between source data i and source data j; ik Indicates the kth attribute corresponding to source data i, a ik Indicates the value of the kth attribute corresponding to source data i.

[0056] This embodiment also provides an algorithmic lineage management device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the above-mentioned algorithmic lineage management method based on feature fusion is executed in the computer memory.

[0057] When performing routine maintenance of the software, you only need to input the source code data of the algorithm to be updated, trace the algorithm through a visual algorithm lineage map, and output the algorithm data set with the same or similar source code data as the target algorithm. Then, use the software to replace and update these algorithm data sets as a whole, thereby quickly completing the daily management and update of the source code data.< / double> < / double>

Claims

1. A lineage management method based on feature fusion algorithm, characterized in that: include: Step 1: Acquire algorithm information and corresponding source code data, parse the source code data, and obtain a control flow graph and an abstract syntax tree corresponding to the node information in the control flow graph; Step 2: Obtain the node information corresponding to the source code data in the control flow graph, and calculate the node embedding vector corresponding to the node information; Step 3: Obtain the sequence identifier corresponding to the source code data in the abstract syntax tree, and calculate the word embedding vector corresponding to the sequence identifier; Step 4: Fuse the node embedding vector obtained in step 2 and the word embedding vector obtained in step 3 to obtain a fused feature vector of the corresponding source code data. The fused feature vector in step 4 is obtained by concatenating the word embedding vectors of all sequence identifiers in the current node and the node embedding vector of the node. Its specific expression is as follows: Where, E CFG Node embedding vector representing source code data, The word embedding vector representing the kth identifier within the node; Step 5: Based on the algorithm information obtained in step 1 and the fused feature vector obtained in step 4, the fused feature vector corresponding to the source code data is visualized and analyzed to obtain an algorithm lineage map for tracing and managing source code data. The visualization analysis is based on the fused feature vector and uses Euclidean distance to calculate the similarity between source code data. Source code data with similarity less than a threshold are classified, and the algorithm information is used as the retrieval attribute to construct an algorithm lineage map. The specific process is as follows: According to the algorithm information obtained in step 1 and the fusion feature vector obtained in step 4, the fusion feature vector corresponding to the source code data is converted into As an entity, the similarity S index is used as a relationship, and the algorithm information is used as the attribute A and the value a to construct an algorithm lineage map for traceability management of source code data: In the formula, Represents the node fusion feature vector corresponding to source data i, Represents the node fusion feature vector corresponding to source code data j, S ij A represents the similarity of the fusion feature vectors between source data i and source data j; ik Indicates the kth attribute corresponding to source data i, a ik Indicates the value of the kth attribute corresponding to source data i.

2. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: The algorithm information in step 1 includes the original file path, model name, function name, programming language, code comments, input and output types, and functional information.

3. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: After the control flow graph is obtained in step 1, the control flow graph needs to be preprocessed. The preprocessing includes splicing and replacing node information, and feature extraction is performed on the preprocessed control flow graph.

4. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: After the abstract syntax tree is obtained in step 1, the abstract syntax tree needs to be preprocessed. The preprocessing includes normalizing the variables in the program, adding a global auto-increment number, and performing feature extraction on the preprocessed abstract syntax tree.

5. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: The node embedding vector in step 2 is based on the asymmetric transitivity of the control flow graph, and the HOPE algorithm is used for feature extraction. The length of the node embedding vector is 4.

6. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: The sequence identifier of the abstract syntax tree in step 3 is generated by traversing the preprocessed abstract syntax tree starting from the root node through a depth-first search algorithm.

7. The algorithmic lineage management method based on feature fusion according to claim 1 is characterized in that: The word embedding vector in step 3 is obtained by extracting features from the sequence identifier using the Word2Vec algorithm, and the length of the word embedding vector is 50.

8. An algorithmic lineage management device, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory executes the algorithm lineage management method based on fusion features as described in any one of claims 1 to 7; when the computer processor executes the computer program, the following steps are implemented: inputting the source code data of the algorithm to be retrieved, searching through a visualized algorithm lineage map, and outputting an algorithm data set that has an affinity relationship with the target algorithm.

Images