A symbol-based large-scale code data feature extraction and retrieval method

Abstract

This invention proposes a symbol-based method for large-scale code data feature extraction and retrieval, relating to the field of code detection technology. The method includes: sequentially performing lexical analysis, syntactic analysis, and semantic analysis on the source code to construct an abstract syntax tree, symbol table, and control flow graph; establishing a variable definition-use chain; calculating the influence propagation coefficient for each variable definition point in the definition-use chain; extracting the statistical features of the influence propagation coefficients of all definition points as data flow enhancement features; calculating the control-data flow coupling strength of nodes; extracting control flow graph perceptual features; obtaining data flow feature vectors and enhanced semantic feature vectors; acquiring the detection task to obtain the final feature vector; training a classifier using the final feature vector; and using the classifier to predict the code to be detected to obtain the detection result. This invention can improve the discriminative and expressive capabilities of code features, thereby enhancing the accuracy and effectiveness of static code detection.

Description

Technical Field

[0001] This invention relates to the field of code detection technology, and in particular to a method for feature extraction and retrieval of large-scale code data based on symbols. Background Technology

[0002] In static code analysis, the accuracy and completeness of code feature extraction directly affect the reliability of the detection results. Code feature extraction extracts key information and features from source code for tasks such as code quality inspection, security vulnerability discovery, performance issue assessment, code style checking, and code complexity evaluation. Most existing feature extraction techniques generate hash or MD5 values ​​by deleting or replacing useless parts of the code. This approach is overly simplistic, ignoring semantic information such as code context, function call relationships, and file associations. It fails to perform multi-dimensional detection such as semantic analysis, data flow analysis, and code metrics, resulting in high false negative and false positive rates in static analysis. Existing technologies also include methods that perform feature extraction through lexical analysis, syntax analysis, semantic analysis, and data flow analysis. These methods extract features based on statistical analysis after completing the analysis steps, primarily extracting features such as identifier frequency and naming conventions, function complexity and call relationships, and dependencies between classes. However, these methods only define statistical variables in data flow analysis—using simple metrics such as the number of chains and average chain length—without considering the propagation effect of data flow in control flow; in feature fusion, they only use simple splicing or weighted summation methods, without considering the interaction relationship between features of different analysis dimensions; in feature engineering, they only use traditional metrics such as the number of lines of code and cyclomatic complexity, without fully exploring the structural characteristics of the control flow graph, resulting in weak discriminative ability of the extracted code features.

[0003] Chinese patent CN115906086A discloses a method for detecting web page backdoors based on code attribute graphs. This method constructs an abstract syntax tree (AST) based on test samples, and then builds a control flow graph, a program dependency graph, and a function call graph based on the AST. Syntactic attributes from the AST are extracted as nodes. Control flow in the control flow graph, data flow in the program dependency graph, and inter-function relationships in the function call graph are extracted respectively. These are combined to construct a code attribute graph, which is stored in a code attribute graph database. Backdoor detection is performed based on tainted path search through reverse traversal of the code attribute graph database. Although this patent combines control flow and data flow information to construct the code attribute graph, it is mainly applied to malicious code detection scenarios based on tainted paths. It does not quantitatively analyze the propagation impact of data flow in the control flow, nor does it design an adaptive feature fusion mechanism for different detection tasks. Summary of the Invention

[0004] In view of this, the present invention provides a symbol-based method for large-scale code data feature extraction and retrieval. By introducing an influence propagation coefficient to quantify the propagation influence of data flow in control flow, designing the interaction relationship between control dependency and data flow dependency of control-data flow coupling strength measurement nodes, mining the structural features of control flow graph, and adopting a task-specific attention mechanism to achieve adaptive feature fusion, the method improves the discriminative and expressive capabilities of code features, thereby enhancing the accuracy and effectiveness of static code detection.

[0005] The technical solution of this invention is implemented as follows: This invention provides a method for feature extraction and retrieval of large-scale code data based on symbols, including: S1. Perform lexical analysis, syntax analysis and semantic analysis on the source code in sequence, and construct an abstract syntax tree, symbol table and control flow graph. The nodes of the control flow graph represent basic code blocks and the edges represent control flow transfer relationships. S2. Perform data flow analysis based on symbol tables and control flow graphs, and establish variable definitions using chains; S3. For each variable definition point in the definition-use chain, calculate the influence propagation coefficient of the definition point based on the path characteristics from the definition point to the use point, the control flow complexity, and the semantic importance of the use point; extract the statistical features of the influence propagation coefficients of all definition points as data flow enhancement features; S4. For nodes in the control flow graph, calculate the control-data flow coupling strength of the node based on the product relationship between control dependency depth and data flow dependency depth; extract the statistical features of coupling strength and the structural features of the control flow graph as control flow graph perception features; S5. Extract lexical feature vectors, syntactic feature vectors, semantic feature vectors, and basic data flow feature vectors from lexical analysis, syntax analysis, semantic analysis, and data flow analysis. Merge the data flow enhanced features with the basic data flow feature vectors to obtain the data flow feature vectors. Merge the control flow graph-aware features with the semantic feature vectors to obtain the enhanced semantic feature vectors. S6. Obtain the detection task, use the attention mechanism to calculate the task-specific attention weights of lexical feature vector, syntactic feature vector, enhanced semantic feature vector and data flow feature vector, and perform weighted fusion of the four feature vectors based on the attention weights to obtain the fused feature vector. Concatenate the fused feature vector with the four feature vectors to obtain the final feature vector. S7. Use the final feature vector to train a classifier, and use the classifier to predict the code to be detected to obtain the detection result.

[0006] Preferably, the calculation of the influence propagation coefficient in step S3 includes: For each variable's definition point, obtain the set of all usage points for that definition point from the definition-usage chain. For each usage point, calculate the path reachability weight, control flow complexity weight, and usage point semantic importance weight. Multiply and sum these three weights to obtain the influence propagation coefficient of that definition point. : ; In the formula, v is a variable and d is the definition point. Let u be the set of all points that are used to define the point, and u be the point that is used. For path reachability weights, To control the flow complexity weights, To use point semantic importance weights.

[0007] Preferably, the path reachability weight is calculated using a breadth-first search algorithm to determine the shortest path length from the defined point to the point of use on the control flow graph. The path length is expressed in units of the number of edges in the control flow graph, and the path reachability weight is calculated using an exponential decay function. : ; In the formula To define the shortest path length from point d to point u, The attenuation coefficient has a value range of 0.1 to 0.5.

[0008] Preferably, the control flow complexity weight is calculated based on the number of branch nodes and loop nodes along the path from the definition point to the usage point. A branch node is a node in the control flow graph with multiple successor nodes, and a loop node is the head node of a loop structure in the control flow graph. The control flow complexity weight... The calculation formula is: ; In the formula This represents the number of branch nodes traversed along the path. This represents the number of loop nodes traversed along the path. The value range is 0.3 to 0.8. The value range is 1.0 to 2.0.

[0009] Preferably, the semantic importance weight of the usage point is determined according to the node type of the usage point in the abstract syntax tree. The weight of the function call node is 2.0, the weight of the return statement node is 1.8, the weight of the conditional expression node is 1.5, the weight of the loop condition node is 1.3, and the weight of other types of nodes is 1.0.

[0010] Preferably, the calculation of the control-data flow coupling strength in step S4 includes: For nodes in the control flow graph, calculate the control dependency depth and data flow dependency depth. The control dependency depth is obtained by traversing the control dependency set of the node, assigning a weight to each control node according to its type, and summing the weights. The weight of a simple branch node is 1.0, the weight of a loop condition node is 2.0, the weight of a switch statement node is 1.5, and the weight of a short-circuit logical operator node is 0.5. The data flow dependency depth is defined by the variable used in the node - using the average chain length; The data flow dependency depth is smoothed using a logarithmic function, and the control-data flow coupling strength of node v is adjusted. The calculation formula is: ; In the formula To control dependency depth, For data flow dependency depth.

[0011] Preferably, the structural features of the control flow graph in step S4 include branch balance, and the calculation of branch balance includes: For a branch node with k successor branches, a control flow subgraph is constructed for each branch, extending from the branch origin to the branch confluence point or function exit. The product of the number of nodes in the subgraph and the average execution frequency of the subgraph is used as the branch weight. The balance of the branch node is calculated using the Gini impurity formula. : ; In the formula For the m-th branch, The weight of the branch; Extract the average balance of all branch nodes as the branch balance feature.

[0012] Preferably, the structural features of the control flow graph in step S4 include loop complexity, and the calculation of loop complexity includes: For loop structures, we calculate the number of loop exits, nesting depth, and number of function calls within the loop body. Loop exits include normal exits, break statements, continue statements, and return statements. The loop complexity of loop structure L is calculated. The calculation formula is: ; In the formula This represents the number of exports in the cycle. The nesting depth of the loop, The number of function calls within the loop body; Extract the maximum and average loop complexity of all loops as loop complexity features.

[0013] Preferably, the calculation of the attention mechanism in step S6 includes: The lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector are all normalized using the L2 norm. The normalized feature vectors are then linearly transformed using a mapping matrix and followed by a hyperbolic tangent activation function. The attention scores are then weighted using the query vector of the detection task. The attention scores of the four feature vectors are normalized using the softmax function to obtain task-specific attention weights. For a detection task t and feature dimension i, the attention score is... and attention weight The calculation formula is: ; ; In the formula Let be the normalized feature vector of feature dimension i, where i takes the values ​​lex, syn, sem-enh, or data, representing the feature dimensions of the lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector, respectively. For the mapping matrix, For bias vectors, Let be the query vector for task t.

[0014] Preferably, the calculation of the fused feature vector in step S6 includes: For the four feature dimensions—lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector—a projection matrix is ​​used to map the normalized feature vectors to a unified dimensional space. The mapped feature vectors are then weighted and summed according to task-specific attention weights to obtain the fused feature vector for task t. : ; In the formula The projection matrix; The four normalized feature vectors are concatenated to obtain the concatenated feature vector, which is then concatenated with the fused feature vector to obtain the final feature vector.

[0015] The present invention has the following advantages over the prior art: (1) By introducing an influence propagation coefficient, this invention comprehensively considers the path reachability from the definition point to the use point, the complexity of the control flow, and the semantic importance of the use point. It adopts an exponential decay function and a weighted accumulation method to quantify the influence intensity of variable definition on the use point. Compared with the existing technology that only counts the number of definition-use chains and the average chain length, it can more accurately characterize the propagation characteristics of data flow in the control flow and improve the discrimination ability of data flow features. (2) The control-data flow coupling strength index designed in this invention reflects the interaction effect of control flow and data flow through the product relationship between control dependency depth and data flow dependency depth. The logarithmic function is applied to the data flow dependency depth for smoothing, which effectively captures the complexity of code nodes in control dependency and data flow dependency. Compared with existing methods that extract control flow or data flow features from only a single dimension, it can more comprehensively reflect the complexity and coupling degree of the code. (3) The structural features of the control flow graph extracted by this invention include branch balance and loop complexity. The branch balance is calculated by combining the Gini impurity formula with the number of nodes in the branch subgraph and the average execution frequency. The loop complexity comprehensively considers the number of loop exits, nesting depth and the number of function calls in the loop body. Compared with traditional coarse-grained measures such as cyclomatic complexity, it can more finely characterize the structural features of the control flow graph and enhance the ability of code features to express the complexity of control flow. (4) The present invention adopts a task-adaptive feature fusion method based on attention mechanism. It calculates task-specific attention weights for four feature dimensions: lexical, syntactic, enhanced semantic and data flow through mapping matrix, hyperbolic tangent activation function and task query vector. Compared with the fusion method of simple concatenation or fixed weight weight summation, it can automatically adjust the importance of each feature dimension according to different detection tasks, improve the flexibility and task adaptability of feature fusion, and thus improve the accuracy of different detection tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram illustrating the technical implementation of the present invention; Figure 3 This is a schematic diagram of the control flow graph structure features of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figures 1-3 As shown, this invention provides a symbol-based method for large-scale code data feature extraction and retrieval, comprising: S1, performing lexical analysis, syntactic analysis, and semantic analysis on the source code sequentially to construct an abstract syntax tree, a symbol table, and a control flow graph, where nodes in the control flow graph represent basic code blocks and edges represent control flow transition relationships; S2, performing data flow analysis based on the symbol table and control flow graph to establish variable definition-use chains; S3, for each variable definition point in the definition-use chain, calculating the influence propagation coefficient of the definition point based on the path characteristics from the definition point to the use point, the control flow complexity, and the semantic importance of the use point; extracting the statistical features of the influence propagation coefficients of all definition points as data flow enhancement features; S4, for nodes in the control flow graph, calculating the control-data flow coupling strength of the nodes based on the product relationship between the control dependency depth and the data flow dependency depth; extracting the statistical features of the coupling strength. The structural features of the features and control flow graph are used as control flow graph perceptual features; S5, extract lexical feature vectors, syntactic feature vectors, semantic feature vectors, and basic data flow feature vectors from lexical analysis, syntax analysis, semantic analysis, and data flow analysis. Merge the data flow enhanced features with the basic data flow feature vectors to obtain the data flow feature vector, and merge the control flow graph perceptual features with the semantic feature vectors to obtain the enhanced semantic feature vector; S6, obtain the detection task, use an attention mechanism to calculate the task-specific attention weights of the lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector, and perform weighted fusion of the four feature vectors based on the attention weights to obtain the fused feature vector. Concatenate the fused feature vector with the four feature vectors to obtain the final feature vector; S7, use the final feature vector to train a classifier, use the classifier to predict the code to be detected, and obtain the detection result.

[0020] In one embodiment of the present invention, step S1 includes: First, the source code is preprocessed by collecting a source code dataset and removing whitespace characters and standardizing newline characters to prepare for subsequent analysis. A lexical analyzer is used to perform lexical analysis on the source code. A lexical rule table is defined according to the lexical rules of the programming language. Each lexical rule contains a pattern and a corresponding lexical unit type. The pattern is represented by a regular expression, and the lexical unit type includes identifiers, keywords, operators, constants, etc. The source code file is read, and the source code is read character by character and matched with the lexical rule table. The matched character sequence is identified as a lexical unit and added to the lexical unit list. After the lexical analysis is completed, the lexical unit list is returned as the output result. A parser is used to perform lexical unit sequence analysis on the lexical analysis. Syntax rules are defined according to the syntax specification of the programming language, including non-terminal symbols and terminal symbols. Non-terminal symbols represent combinations of syntactic structures, and terminal symbols represent lexical units. A syntax rule table is constructed, and each syntax rule contains a left-hand non-terminal symbol and a right-hand symbol sequence. The parser is constructed using either top-down recursive descent parsing or bottom-up LR parsing. The lexical unit sequence is provided as input to the parser. The parser matches the lexical unit sequence according to the syntax rule table and constructs an abstract syntax tree. After the parsing is completed, the abstract syntax tree is returned as the output. The semantic analyzer performs semantic analysis on the abstract syntax tree. First, a symbol table is created to store information about variables, functions, classes, etc. The abstract syntax tree is traversed, and when semantic structures such as variable declarations and function definitions are encountered, their relevant information is added to the symbol table. Type checks are performed on expressions and assignment statements in the code. The abstract syntax tree is traversed to perform type inference on each expression, and the type matching between operators and operands is checked. The type information in the symbol table is used to check whether the use of variables conforms to semantic rules. Analyze the scope in the code, maintain a scope stack to track variables in the current scope, handle the syntax structure of variable declaration and scope entry and exit, and add variables to the current scope or remove them from the scope stack. Analyze the control flow structure in the code, such as conditional statements and loop statements, and construct a control flow graph by traversing the abstract syntax tree. The nodes of the control flow graph represent basic code blocks, and the edges represent control flow transition relationships. After semantic analysis, output the symbol table and control flow graph.

[0021] In one embodiment of the present invention, step S2 includes: The control flow graph is analyzed using a data flow analyzer. Data flow information is initialized at the entry and exit points of each basic block of the control flow graph, and a symbol table is used to store the definition, use, and transmission of variables. In each basic block, the data flow information is iteratively computed until convergence. The data flow information is updated using the data flow equation and iterative algorithm, where the data flow equation includes the data flow values ​​at the inlet and outlet as well as the data flow transfer function. Mark the definition and usage points of variables in the control flow graph, establish the definition-use chain of variables, determine the transmission path of variables between different statements, and output a control flow graph containing definition-use chain information after the data flow analysis is completed.

[0022] In one embodiment of the present invention, step S3 includes: Based on the definitions obtained from data flow analysis, use chains to calculate the influence propagation coefficient for each variable definition point; For a definition point d of variable v, obtain the set of all usage points of that definition point from the definition-usage chain. For each usage point u, calculate the path reachability weight, control flow complexity weight, and usage point semantic importance weight. Multiply and sum the three weights to obtain the influence propagation coefficient of that defined point. : ; in For path reachability weights, To control the flow complexity weights, To use point semantic importance weights; Path reachability weight The shortest path length from the definition point d to the usage point u on the control flow graph is calculated using a breadth-first search algorithm. The path length is calculated using an exponential decay function, with the number of edges in the control flow graph as the unit. ; in The attenuation coefficient, ranging from 0.1 to 0.5, is used to control the rate at which the influence of path length decays; the longer the path, the lower the weight. The exponential decay form is chosen because, in the code, the longer the path from the defined point to the point of use, the weaker the propagation effect of the data flow. The exponential function can better simulate this non-linear decay relationship. Control flow complexity weight The calculation is based on the number of branch nodes and loop nodes along the path from point d to point u. Branch nodes are nodes in the control flow graph with multiple successor nodes, corresponding to conditional statements like if and switch statements in the code. Loop nodes are the head nodes of loop structures in the control flow graph, corresponding to loop statements like for and while loops in the code. The calculation formula is: ; in This represents the number of branch nodes traversed along the path. This represents the number of loop nodes traversed along the path. The value range is 0.3 to 0.8. The value range is 1.0~2.0; the square root function is used for the number of branch nodes because the impact of branches on control flow complexity increases sublinearly, and the complexity increment of increasing the number of branches from 1 to 2 is different from that of increasing from 10 to 11; the linear function is used for the number of loop nodes because loops have a more significant impact on control flow complexity, and each additional layer of nested loops will significantly increase the execution complexity and understanding difficulty of the code. ; FunctionCall represents the function call node, ReturnStmt represents the return statement node, ConditionExpr represents the conditional expression node, and LoopCondition represents the loop condition node. These weight values ​​are set based on code analysis experience. The use of variables at the function call point has the greatest impact on code behavior, the return statement determines the function output, and the conditional expression and loop condition affect the direction of control flow, so they are given higher weights. After calculating the influence propagation coefficients for all definitions of all variables, the following statistical features are extracted as data flow augmentation features: the average influence is calculated as the arithmetic mean of the influence propagation coefficients for all definitions; the influence variance is calculated as the variance of the influence propagation coefficients for all definitions; the maximum influence is the maximum value of the influence propagation coefficients for all definitions; and the influence concentration is calculated as the proportion of the total influence to the sum of the influence of the top 10% of definitions with the highest influence propagation coefficients. The idea behind introducing an influence propagation coefficient is that traditional data flow analysis only counts the number and average length of definition-use chains, failing to reflect the propagation characteristics of data flow in control flow. This invention comprehensively considers three dimensions: path reachability, control flow complexity, and semantic importance of use points. It uses exponential decay and weighted accumulation to quantify the influence strength of variable definitions on use points, which can more accurately characterize the propagation characteristics of data flow in control flow and improve the discriminative ability of data flow features.

[0023] In one embodiment of the present invention, step S4 includes: Extract control flow graph perception features from the control flow graph obtained by semantic analysis; First, static execution frequency estimation is performed on each node of the control flow graph. Execution frequency estimation is achieved by performing a forward traversal of the control flow graph. Let the execution frequency of the entry node of the control flow graph be 1.0. For node v in the control flow graph, its execution frequency... Calculated based on the execution frequency of its predecessor node and the execution probability from the predecessor to v; For edges executed sequentially, the execution probability is 1.0; for branch edges, in the absence of runtime data, it is assumed that the execution probability of each branch is equal, and for two branches, it is assumed to be 0.5; for loop-back edges, the average number of iterations of the loop is used. As the probability of execution, The value is 10; for exception handling edges, the execution probability is set to 0.1. For loop structures, first identify the loop head node h and the set of loop body nodes. The execution frequency of the loop head node is equal to the execution frequency of its outer predecessor node, and the execution frequency of the nodes in the loop body is the execution frequency of the loop head node multiplied by the average number of loop iterations. ; Compute the criticality of node v The formula is: ; in The execution frequency of node v. Let v be the out-degree of node v, i.e., the number of its successor nodes. To control the maximum out-degree of all nodes in the flow graph for normalization, For indicator functions, when node v belongs to a special set of nodes. The value is 1 if the condition is met, and 0 otherwise. Special nodes include function call nodes, return statement nodes, and exception throwing nodes. and These are weighting coefficients, with values ​​of 0.5 and 0.3 respectively; The criticality score takes into account the execution frequency of a node, the complexity of the control flow branches, and the semantic importance of the node. Nodes with high execution frequency and multiple successor branches or belonging to special node types have higher criticality scores. The statistical features of criticality scores extracted include average criticality score, criticality score variance, maximum criticality score, and the proportion of critical nodes. Control dependency depth of compute node v Depth of data flow ; Control dependency depth is defined as the weighted sum of whether all control nodes v have been executed, for a control dependency set. Each node u in the algorithm is assigned a weight based on its type. Simple branch nodes, such as if and else statements, have a weight of 1.0; loop condition nodes, such as condition judgment nodes in for and while loops, have a weight of 2.0; switch statement nodes have a weight of 1.5; and short-circuit logical operator nodes, such as logical AND and logical OR, have a weight of 0.5. The control dependency depth is calculated as the sum of the weights of all nodes in the control dependency set. Loop conditions have the highest weight because they control multiple executions, while short-circuit logical operators have the lowest weight because they provide local control. Data flow dependency depth is defined as the definition of the variables used in node v - using the average chain length, let For the set of variables used in node v, for variables The definition-use chain length of variable x at the point of use in node v is the distance from its nearest definition point to the edge of the control flow graph. The data flow dependency depth is calculated as the arithmetic mean of the definition-use chain lengths of all variables used in node v. The data flow dependency depth is smoothed using a logarithmic function, and the control-data flow coupling strength of node v is adjusted. The calculation formula is: ; Coupling strength combines control dependency depth and data flow dependency depth, and uses a multiplicative form to represent the interaction effect of control flow and data flow. The logarithmic function is used for data flow dependency depth because the impact of data flow dependency depth on complexity increases sublinearly, and the logarithmic function can smooth out large data flow depth values. Nodes with high control dependency depth and high data flow dependency depth have the highest coupling strength, reflecting the complexity and degree of coupling of the code. The following statistical features are extracted: the average coupling strength is the arithmetic mean of the coupling strengths of all nodes; the standard deviation of the coupling strength is calculated, and the proportion of highly coupled nodes is defined as the proportion of nodes with a coupling strength greater than the average coupling strength plus one standard deviation to the total number of nodes; the coupling strength entropy is calculated as the weighted logarithmic sum of the proportions of the coupling strengths of each node that are negative, where the proportion of the coupling strength of node v is the coupling strength of node v divided by the sum of the coupling strengths of all nodes. High entropy indicates that the coupling strength is evenly distributed, and low entropy indicates that a few nodes are highly coupled. Identify branch nodes and loop nodes in the control flow graph and extract its structural features; For each branch node v, suppose it has k successor branches. weight The calculation is the number of nodes in the branch subgraph multiplied by the average execution frequency of the branch subgraph. A branch subgraph is a control flow graph subgraph extending from the branch origin to the branch junction or function exit point. The number of nodes in the branch subgraph is the total number of nodes in the subgraph. The average execution frequency of the branch subgraph is the arithmetic mean of the execution frequencies of all nodes in the branch subgraph. The balance of branch node v is also considered. Calculated using the Gini impurity formula: ; A balance degree close to 1 indicates that the weights of each branch are similar, i.e., the branches are highly balanced; a balance degree close to 0 indicates that a certain branch is dominant, i.e., the branches are unbalanced. The average balance degree of all branch nodes is calculated as the branch balance degree feature. For each loop structure L, count the number of loop exits, the nesting depth, and the number of function calls within the loop body. The number of loop exits includes normal exits, break statements, continue statements, and return statements. Calculate the loop complexity of loop structure L. The calculation formula is: ; in This represents the number of exports in the cycle. The nesting depth of the loop, The number of function calls within the loop body; multi-exit loops are more complex; the nesting depth uses the square root function to affect loop complexity because the complexity of nesting increases sublinearly; function calls increase loop complexity because of potential side effects and exceptions; the maximum and average loop complexity of all loops are extracted as loop complexity features. Define exception control flow edges as break and continue edges in loops, jump edges in goto statements, return edges in the middle of functions, and exception throwing and catching edges; calculate exception control flow density. The formula is: ; in To control the number of flow edges in abnormal situations, To control the total number of edges in the flow graph, The number of goto statements. To control the total number of nodes in the flow graph, The penalty coefficient for goto is 2.0; the goto statement is specially penalized because it violates structured programming principles and significantly reduces code readability and maintainability. The extracted keyness statistical features, coupling strength statistical features, average branch balance, loop complexity features, and abnormal control flow density are combined into control flow graph perception features. The idea behind the control-dataflow coupling strength is that traditional methods extract control flow features and data flow features independently without considering their interaction. This invention quantifies the interaction effect of control flow and data flow of a node by multiplying the control dependency depth and data flow dependency depth, and applies a logarithmic function to smooth the data flow dependency depth, which can more comprehensively reflect the complexity and coupling degree of the code. The structural features of the control flow graph include branch balance and loop complexity. Compared with traditional coarse-grained measures such as cyclomatic complexity, it can more finely characterize the structural characteristics of the control flow graph and enhance the ability of code features to express the complexity of control flow.

[0024] In one embodiment of the present invention, step S5 includes: Extract basic features from the results of lexical analysis, syntactic analysis, semantic analysis, and data flow analysis; Lexical features include the number of identifiers, keyword frequency, comment density, and lines of code; syntactic features include the total number of nodes in the abstract syntax tree, the maximum depth of the tree, the average branching factor of the tree, and the number of leaf nodes; semantic features include symbol table size, number of functions, number of classes, maximum inheritance depth, and cyclomatic complexity. Basic data flow features include the total number of definition-use chains, average definition-use chain length, average variable lifetime, number of nodes in the control flow graph, and number of edges in the control flow graph. Lexical features, syntactic features, semantic features, and basic data flow features are respectively used to construct lexical feature vectors, syntactic feature vectors, semantic feature vectors, and basic data flow feature vectors; the data flow enhancement features obtained in step S3 are merged with the basic data flow feature vectors to form a data flow feature vector; the control flow graph perception features obtained in step S4 are merged with the semantic feature vectors to form an enhanced semantic feature vector.

[0025] In one embodiment of the present invention, step S6 includes: The lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector are normalized using the L2 norm to obtain normalized feature vectors. For different detection tasks, an attention-based task-adaptive feature fusion method is adopted; The testing tasks include five aspects: code quality testing, security vulnerability detection, performance issue detection, code style inspection, and code complexity testing. For detection task t, attention scores are calculated for four feature dimensions: lexical, syntactic, enhanced semantic, and data flow. For feature dimension i, where i takes the values ​​lex, syn, sem-enh, or data, representing the feature dimensions of the lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector, respectively, the attention scores are calculated for the normalized feature vector. Through the mapping matrix After performing a linear transformation, the hyperbolic tangent activation function is applied, followed by the query vector of the detection task t. Attention score is obtained by weighting. : ; in Let be the mapping matrix for feature dimension i, which maps the feature vectors of that dimension to the hidden layer space. The hidden layer dimension is set to 64. The bias vector is used as the bias vector. The mapping matrix, bias vector, and query vector are learned by pre-training on the labeled dataset using a differentiable auxiliary classification head. After pre-training, the parameter values ​​of the mapping matrix, bias vector, and query vector are fixed for subsequent feature fusion and detection tasks. The pre-training process for the mapping matrix, bias vector, and query vector includes: constructing a differentiable auxiliary classifier head (such as a fully connected layer with a Softmax or Sigmoid output layer) corresponding to the final detection task; inputting the fused feature vector obtained in step S6 into the auxiliary classifier head; using the backpropagation algorithm to jointly optimize the mapping matrix, bias vector, query vector, and auxiliary classifier head parameters; the loss function is determined according to the specific detection task, using cross-entropy loss for classification tasks and mean squared error loss for regression tasks; the training dataset needs to contain a sufficient number of labeled samples to learn effective attention weights, typically requiring at least several thousand labeled samples per task; during training, techniques such as learning rate decay and early stopping can be used to avoid overfitting; the model performance is monitored using the validation set to select the optimal model parameters; after pre-training, the auxiliary classifier head is no longer used, the parameters of the mapping matrix, bias vector, and query vector are fixed, and its output fused feature vector is used as the input feature of the classifier model in step S7; The attention scores for the four feature dimensions are normalized using the softmax function to obtain task-specific attention weights. The attention weights for feature dimension i in detection task t are... The calculation formula is: ; For the four feature dimensions of lexical, syntactic, enhanced semantics, and data flow, a projection matrix is ​​used. Normalized eigenvectors Mapping to a unified dimensional space, with the unified dimension set to 128; then, weighting and summing the mapped feature vectors according to task-specific attention weights to obtain the fused feature vector for detection task t. : ; The four normalized feature vectors are concatenated to obtain a concatenated feature vector. This concatenated feature vector is then concatenated with the fused feature vector to obtain the final feature vector for detection task t. The final feature vector retains detailed information from the original features while also incorporating task-specific fused features, thus improving the expressive power and task adaptability of the features. The idea behind the task-adaptive feature fusion method based on attention mechanisms is that traditional methods fuse features by simply concatenating or using fixed-weight weighted summation, without considering the interaction relationships between features of different analysis dimensions and the characteristics of different detection tasks. This invention calculates task-specific attention weights for four feature dimensions—lexical, syntactic, enhanced semantic, and data flow—using a mapping matrix, hyperbolic tangent activation function, and task query vector. This allows for automatic adjustment of the importance of each feature dimension according to different detection tasks, improving the flexibility and task adaptability of feature fusion.

[0026] In one embodiment of the present invention, step S7 includes: For each detection task t, a classifier model is constructed using the gradient boosting decision tree algorithm; The final feature vector of the detection task t obtained in step S6 is used as the model input, and the detection result is used as the model output. A gradient boosting decision tree model is trained on a labeled dataset containing code samples and their corresponding detection task labels. The model hyperparameters, including tree depth, learning rate, and regularization parameters, are tuned using cross-validation. After training, for each code sample to be detected, steps S1 to S6 are executed sequentially to extract its final feature vector, which is then input into the gradient boosting decision tree model for the corresponding task for prediction, obtaining the detection result for that task. The performance of each task model is evaluated on a test dataset, with evaluation metrics including accuracy, precision, recall, F1 score, and AUC. Based on the evaluation results, the advantages and disadvantages of the model are analyzed, and the feature extraction strategy and model parameters are adjusted and optimized as needed. It should be noted that in step S3, the decay coefficient of the path reachability weight... The value ranges from 0.1 to 0.5, with smaller values ​​being preferred. A value that makes the impact of path length on weights more moderate, and a larger value... The value causes the weight of distant points to decay rapidly; the specific value can be determined through cross-validation based on the average function length and control flow complexity of the codebase; the control flow complexity weight... and The coefficients control the contribution of branches and loops to complexity, and their value ranges are set based on empirical code analysis and can be optimized through grid search during training; in step S4, the weight coefficients for keyness calculation are... and The value is based on code analysis experience and can be adjusted according to specific application scenarios; average number of iterations per loop. The value is set to 10 based on statistical results from common code patterns. For code in specific domains, this can be adjusted according to the actual situation; the goto penalty coefficient in the exception control flow density calculation... The value is 2.0 because the goto statement is quite destructive to the code structure; In the data flow analysis of step S2, the time complexity of constructing the control flow graph and iteratively calculating the data flow information is O(n). Where V is the number of nodes and E is the number of edges, incremental analysis and symbolic execution optimization techniques can be used to reduce computational overhead for large code files; in step S3, the shortest path length is calculated using a breadth-first search algorithm, with a time complexity of O(n log n). For all definitions, the total computational complexity using pairs is... ,in To define the number of chains used, redundant calculations can be reduced by caching the already calculated path lengths; in step S4, the computational time complexity of the dependencies is controlled. For large control flow graphs, the dominator tree algorithm and post-dominator tree algorithm can be used for optimization, reducing the complexity to [missing information]. ; The statistical features extracted in step S3 include four indicators: average influence, influence variance, maximum influence, and influence concentration. These indicators characterize the distribution characteristics of data flow influence from different perspectives. Average influence reflects the overall level of influence, influence variance reflects the dispersion of influence, maximum influence identifies key definition points, and influence concentration reflects whether influence is concentrated in a few definition points. The keyness statistical features extracted in step S4 include average keyness, keyness variance, maximum keyness, and key node proportion. These indicators characterize the importance distribution of nodes in the control flow graph. The coupling strength statistical features include average coupling strength, coupling strength standard deviation, high coupling node proportion, and coupling strength entropy. These indicators comprehensively reflect the interaction complexity of the code's control flow and data flow. In step S6, the mapping matrix of the attention mechanism Bias vector Projection matrix and task query vector The model is learned through end-to-end training. The training process uses the backpropagation algorithm, and the loss function is determined according to the specific detection task. Cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. The training dataset needs to contain a sufficient number of labeled samples to learn effective attention weights. Typically, each task requires at least several thousand labeled samples. During training, techniques such as learning rate decay and early stopping can be used to avoid overfitting. The model performance is monitored and the optimal model parameters are selected through the validation set. This invention relates to five detection tasks: code quality detection, security vulnerability detection, performance issue detection, code style checking, and code complexity detection. Code quality detection primarily identifies defects and non-standard coding practices in the code, such as uninitialized variables and resource leaks. Security vulnerability detection identifies security risks in the code, such as SQL injection and buffer overflows. Performance issue detection identifies code patterns that affect program performance, such as inefficient algorithms and redundant computations. Code style checking evaluates whether the code conforms to coding standards, such as naming conventions and indentation formats. Code complexity detection evaluates the complexity of the code and identifies overly complex and difficult-to-maintain code snippets. The classifier for each task is trained independently, using task-specific labeled data and attention weights. The symbol-based large-scale code data feature extraction and retrieval method proposed in this invention can effectively improve the discriminative and expressive capabilities of code features through multi-dimensional analysis and task-adaptive feature fusion, thereby enhancing the accuracy and effectiveness of static code detection and making it suitable for various code detection tasks.

[0027] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for feature extraction and retrieval of large-scale code data based on symbols, characterized in that, include: S1. Perform lexical analysis, syntax analysis and semantic analysis on the source code in sequence, and construct an abstract syntax tree, symbol table and control flow graph. The nodes of the control flow graph represent basic code blocks and the edges represent control flow transfer relationships. S2. Perform data flow analysis based on symbol tables and control flow graphs, and establish variable definitions using chains; S3. For each variable definition point in the definition-use chain, calculate the influence propagation coefficient of the definition point based on the path characteristics from the definition point to the use point, the control flow complexity, and the semantic importance of the use point; extract the statistical features of the influence propagation coefficients of all definition points as data flow enhancement features; S4. For nodes in the control flow graph, calculate the control-data flow coupling strength of the node based on the product relationship between control dependency depth and data flow dependency depth; extract the statistical features of coupling strength and the structural features of the control flow graph as control flow graph perception features; S5. Extract lexical feature vectors, syntactic feature vectors, semantic feature vectors, and basic data flow feature vectors from lexical analysis, syntax analysis, semantic analysis, and data flow analysis. Merge the data flow enhanced features with the basic data flow feature vectors to obtain the data flow feature vectors. Merge the control flow graph-aware features with the semantic feature vectors to obtain the enhanced semantic feature vectors. S6. Obtain the detection task, use the attention mechanism to calculate the task-specific attention weights of lexical feature vector, syntactic feature vector, enhanced semantic feature vector and data flow feature vector, and perform weighted fusion of the four feature vectors based on the attention weights to obtain the fused feature vector. Concatenate the fused feature vector with the four feature vectors to obtain the final feature vector. S7. Use the final feature vector to train a classifier, and use the classifier to predict the code to be detected to obtain the detection result.

2. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 1, characterized in that, The calculation of the influence propagation coefficient in step S3 includes: For each variable's definition point, obtain the set of all usage points for that definition point from the definition-usage chain. For each usage point, calculate the path reachability weight, control flow complexity weight, and usage point semantic importance weight. Multiply and sum these three weights to obtain the influence propagation coefficient of that definition point. : ； In the formula, v is a variable and d is the definition point. Let u be the set of all points that are used to define the point, and u be the point that is used. For path reachability weights, To control the flow complexity weights, To use point semantic importance weights.

3. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 2, characterized in that, The path reachability weight is calculated using a breadth-first search algorithm to determine the shortest path length from the defined point to the point of use on the control flow graph. The path length is expressed in units of the number of edges in the control flow graph, and the path reachability weight is calculated using an exponential decay function. : ； In the formula To define the shortest path length from point d to point u, The attenuation coefficient has a value range of 0.1 to 0.

5.

4. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 2, characterized in that, The control flow complexity weight is calculated based on the number of branch nodes and loop nodes along the path from the definition point to the usage point. A branch node is a node in the control flow graph with multiple successor nodes, and a loop node is the head node of a loop structure in the control flow graph. The calculation formula is: ； In the formula This represents the number of branch nodes traversed along the path. This represents the number of loop nodes traversed along the path. The value range is 0.3 to 0.

8. The value range is 1.0 to 2.

0.

5. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 2, characterized in that, The semantic importance weight of the usage point is determined based on the node type of the usage point in the abstract syntax tree. The weight of the function call node is 2.0, the weight of the return statement node is 1.8, the weight of the conditional expression node is 1.5, the weight of the loop condition node is 1.3, and the weight of other types of nodes is 1.

0.

6. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 1, characterized in that, The calculation of the control-data flow coupling strength in step S4 includes: For nodes in the control flow graph, calculate the control dependency depth and data flow dependency depth. The control dependency depth is obtained by traversing the control dependency set of the node, assigning a weight to each control node according to its type, and summing the weights. The weight of a simple branch node is 1.0, the weight of a loop condition node is 2.0, the weight of a switch statement node is 1.5, and the weight of a short-circuit logical operator node is 0.

5. The data flow dependency depth is defined by the variable used in the node - using the average chain length; The data flow dependency depth is smoothed using a logarithmic function, and the control-data flow coupling strength of node v is adjusted. The calculation formula is: ； In the formula To control dependency depth, For data flow dependency depth.

7. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 6, characterized in that, The structural features of the control flow graph in step S4 include branch balance, and the calculation of branch balance includes: For a branch node with k successor branches, a control flow subgraph is constructed for each branch, extending from the branch origin to the branch confluence point or function exit. The product of the number of nodes in the subgraph and the average execution frequency of the subgraph is used as the branch weight. The balance of the branch node is calculated using the Gini impurity formula. : ； In the formula For the m-th branch, The weight of the branch; Extract the average balance of all branch nodes as the branch balance feature.

8. The method for feature extraction and retrieval of large-scale code data based on symbols according to claim 7, characterized in that, The structural features of the control flow graph in step S4 include loop complexity, and the calculation of loop complexity includes: For loop structures, we calculate the number of loop exits, nesting depth, and number of function calls within the loop body. Loop exits include normal exits, break statements, continue statements, and return statements. The loop complexity of loop structure L is calculated. The calculation formula is: ； In the formula This represents the number of exports in the cycle. The nesting depth of the loop, The number of function calls within the loop body; Extract the maximum and average loop complexity of all loops as loop complexity features.

9. A method for feature extraction and retrieval of large-scale code data based on symbols according to claim 1, characterized in that, The calculation of the attention mechanism in step S6 includes: The lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector are all normalized using the L2 norm. The normalized feature vectors are then linearly transformed using a mapping matrix and followed by a hyperbolic tangent activation function. The attention scores are then weighted using the query vector of the detection task. The attention scores of the four feature vectors are normalized using the softmax function to obtain task-specific attention weights. For a detection task t and feature dimension i, the attention score is... and attention weight The calculation formula is: ； ； In the formula Let be the normalized feature vector of feature dimension i, where i takes the values ​​lex, syn, sem-enh, or data, representing the feature dimensions of the lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector, respectively. For the mapping matrix, For bias vectors, Let be the query vector for task t.

10. A method for feature extraction and retrieval of large-scale code data based on symbols according to claim 9, characterized in that, The calculation of the fused feature vector in step S6 includes: For the four feature dimensions—lexical feature vector, syntactic feature vector, enhanced semantic feature vector, and data flow feature vector—a projection matrix is ​​used to map the normalized feature vectors to a unified dimensional space. The mapped feature vectors are then weighted and summed according to task-specific attention weights to obtain the fused feature vector for task t. : ； In the formula The projection matrix; The four normalized feature vectors are concatenated to obtain the concatenated feature vector, which is then concatenated with the fused feature vector to obtain the final feature vector.

Images