A Sandbox-Free Evaluation Method for AI-Generated Code
By stripping redundant nodes and extracting depth, semantic category, and text fingerprint features, combined with semantically weighted longest common subsequence alignment and continuous exponential penalty function, this method solves the problem of distinguishing between critical logical mutations and legitimate refactorings in existing sandbox-free static code evaluation methods. It achieves a balance between sensitivity and tolerance, and improves the accuracy of code evaluation.
Patent Information
- Application Number
- CN202611363581.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-11-17
AI Technical Summary
Existing sandbox-free static code evaluation methods suffer from problems such as inability to effectively distinguish between critical logic mutations and legitimate equivalent refactorings, over-smoothing of fatal errors, and insufficient local resilience, leading to distorted evaluation results.
By obtaining the abstract syntax tree of the reference code and candidate code, removing redundant nodes, and extracting features such as deep labels, semantic categories and text fingerprints, a multiset of key semantic nodes is constructed using semantically weighted longest common subsequence alignment and a continuous exponential penalty function. Character-level N-tuple statistical similarity is then fused to achieve adaptive weight fusion of alignment scores.
While ensuring the rigor of micro-logical review, it provides macro-topological tolerance, significantly amplifies the impact of fatal logical errors, overcomes the defects of excessive smoothing of deep continuous space features and inability to identify discrete operator-level errors, and achieves decoupling and balance between sensitivity and tolerance.
Smart Images

Figure CN123114935A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer software quality assessment technology, and in particular relates to a sandbox-free AI-generated code assessment method. Background Technology
[0002] With the development of large language model code generation technology, sandbox-free static code evaluation technology has emerged. This technology analyzes the code text or structural features without actually executing the code to predict the functional correctness of the generated code, thereby avoiding the high computational cost and security risks brought about by sandbox execution.
[0003] Traditional techniques mainly employ three types of processing methods: The first type is based on lexical sequences and hybrid N-gram statistics, which calculates similarity by performing character-level or lexical segment matching on the code; the second type is based on deep neural network vectors, which maps the code to a continuous high-dimensional semantic space and then compares vector similarity; the third type is a structured method based on abstract syntax trees and control flow, which measures code structural differences by using tree edit distance or heuristic feature concatenation.
[0004] However, current sandbox-free static code evaluation methods have the following problems: Lexical statistical methods have weight blind spots, giving core logic operators and redundant syntax symbols equal importance, making it difficult to distinguish between critical logic mutations and legitimate equivalent refactoring; Deep neural vector-based methods suffer from over-smoothing, where high-dimensional continuous spaces can easily mask discrete but fatal operator-level errors; Tree-based methods suffer from insufficient local flexibility due to strict topological isomorphism requirements, and legitimate code format adjustments can cause a cascading drop in scores, resulting in distorted evaluation results. Summary of the Invention
[0005] Therefore, it is necessary to provide a sandbox-free AI-generated code evaluation method that can decouple logic auditing and topology tolerance, maintain high sensitivity to core logic errors, and have local flexibility for legitimate code refactoring, in order to address the above-mentioned technical problems.
[0006] Firstly, this application provides a sandbox-free method for evaluating AI-generated code, including:
[0007] S1. Obtain reference code and candidate code, perform abstract syntax tree parsing on the reference code and candidate code to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree; after performing execution path sensitivity pre-analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree, remove redundant nodes to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree with path sensitivity labels.
[0008] S2. Extract the deep labels, generalized semantic categories, text fingerprint truncation, and semantic importance weights of each retained node in the target reference abstract syntax tree and the target candidate abstract syntax tree, and concatenate them with the path sensitivity labels to obtain the reference node sequence and the candidate node sequence.
[0009] S3. Based on semantic importance weights, perform semantically weighted longest common subsequence alignment on the reference node sequence and the candidate node sequence, and calculate the basic alignment score;
[0010] S4. Extract nodes belonging to the preset key semantic categories from the reference node sequence and candidate node sequence, construct the reference key multiset and the candidate key multiset, and determine the missing nodes as those that exist in the reference key multiset but not in the candidate key multiset based on the generalized semantic category and text fingerprint truncation.
[0011] S5. Based on the non-negative truncation difference of node frequency between the reference key multiset and the candidate key multiset, the depth label and path sensitivity label of each missing node, and the number of missing nodes under each key semantic category, calculate the composite missing metric. Based on the composite missing metric and the basic alignment score, calculate the core evaluation score through the continuous exponential penalty function.
[0012] S6. Calculate the statistical similarity score based on the character-level N-tuples of the reference code and candidate code. Adaptively weight the core evaluation score and statistical similarity score according to the structural complexity of the target reference abstract syntax tree and the target candidate abstract syntax tree to obtain the code evaluation score.
[0013] The aforementioned sandbox-free AI-generated code evaluation method reduces the dimensionality of the abstract syntax tree to a path-sensitive node sequence by stripping redundant nodes and extracting multi-dimensional features such as depth, semantic category, and text fingerprint. This avoids the weight blind spot in traditional lexical statistics where key operators and ordinary symbols are weighted equally. It employs semantically weighted longest common subsequence alignment, with depth and category as necessary conditions, tolerating legitimate local code reconstructions and addressing the precipitous drop in scores caused by structural rigidity in tree isomorphic matching. By constructing a multiset of key semantic nodes and applying a composite penalty based on frequency, depth, and path sensitivity to missing nodes, and using a continuous exponential decay function to suppress basic alignment scores, it significantly amplifies the impact of fatal logical errors on the score, overcoming the shortcomings of excessive smoothing of deep continuous spatial features and the inability to identify discrete operator-level errors. Finally, it integrates character-level N-tuple statistical similarity and adaptively weights it according to structural complexity, ensuring strict micro-logical review while providing macro-topological tolerance, achieving a decoupling and balance between sensitivity and tolerance. Attached Figure Description
[0014] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 A schematic diagram illustrating the application environment of the sandbox-free AI code generation evaluation method provided by this invention;
[0016] Figure 2 A flowchart illustrating a sandbox-free AI-generated code evaluation method provided by the present invention;
[0017] Figure 3 This is a schematic diagram of the process for generating code evaluation scores in one optional embodiment of the present invention; Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0019] This application provides a sandbox-free AI-generated code evaluation method that can be applied to, for example... Figure 1 In the application environment shown, terminal 102 can communicate with server 101 via a network. A data storage system can store the data that server 101 needs to process. The data storage system can be integrated onto server 101 or located in the cloud or on other network servers. Server 101 can be used to obtain reference code and candidate code, and execute the sandbox-free AI-generated code evaluation method to generate code evaluation scores; terminal 102 can be used to upload candidate code to be evaluated to server 101, and to receive and display the code evaluation scores. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Server 101 can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0020] In one exemplary embodiment, such as Figure 2 As shown, a sandbox-free AI-generated code evaluation method is provided, which can be applied to... Figure 1 Taking server 101 as an example, the explanation includes the following steps:
[0021] S1. Obtain reference code and candidate code, perform abstract syntax tree parsing on the reference code and candidate code to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree; after performing execution path sensitivity pre-analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree, remove redundant nodes to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree with path sensitivity labels.
[0022] Specifically, reference code refers to standard functional code that serves as the evaluation benchmark. Reference code provides the correct syntactic structure, execution logic, and functional implementation basis, and is the benchmark sample for code similarity and integrity evaluation. Candidate code refers to the code to be evaluated generated by the AI model, which is the test object in this quantitative evaluation. The original reference abstract syntax tree and the original candidate abstract syntax tree refer to the initial abstract syntax tree without any optimization or preprocessing, completely preserving all the original syntactic node structures of the reference code and candidate code. Redundant nodes refer to invalid syntactic nodes in the abstract syntax tree that do not participate in the core logic execution and do not contribute to the code semantics and execution logic. Redundant nodes include blank syntactic nodes, comment-affiliated nodes, empty syntactic placeholder nodes, invalid duplicate syntactic nodes, and invalid redundant child nodes under unreachable branches. Path sensitivity labels refer to tags used to identify the execution path type to which an abstract syntax tree node belongs. Path sensitivity labels can be used to distinguish the logical priority and criticality of nodes.
[0023] Optionally, server 101 can obtain reference code and candidate code from the data code set, and perform abstract syntax tree parsing on the reference code and candidate code respectively using a preset syntax parsing tool to generate an original reference abstract syntax tree and an original candidate abstract syntax tree. Server 101 can perform control flow analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree respectively, construct a control flow graph and establish node relationships, and determine the path type of each node. The path type includes mandatory paths, conditional branches, and unreachable branches. Server 101 can mark mandatory path nodes as critical path nodes, conditional branch nodes as conditional path nodes, and unreachable branch nodes as edge nodes, and assign path sensitivity labels to each node to obtain an original reference abstract syntax tree and an original candidate abstract syntax tree with path sensitivity labels. Server 101 can identify and remove redundant syntactic structure nodes from the marked nodes according to preset redundant node filtering rules to obtain a target reference abstract syntax tree and a target candidate abstract syntax tree with path sensitivity labels.
[0024] S2. Extract the deep labels, generalized semantic categories, text fingerprint truncation, and semantic importance weights of each retained node in the target reference abstract syntax tree and the target candidate abstract syntax tree, and concatenate them with the path sensitivity labels to obtain the reference node sequence and the candidate node sequence.
[0025] Specifically, a retained node refers to a syntax node that, after redundant nodes have been removed, remains in the target reference abstract syntax tree and the target candidate abstract syntax tree, carrying the core semantics and effective execution logic of the code. A depth label refers to a quantitative marker used to characterize the nesting depth of a retained node in the abstract syntax tree; depth labels can reflect the structural hierarchy information of the node and the complexity of the code logic nesting. A generalized semantic category refers to a unified category label that categorizes various syntax nodes according to their functional semantic dimension; generalized semantic categories can be used to discard minor syntactic differences and focus on the core functional semantics of the code. Generalized semantic categories include four types: regular variables, control flow, API calls, and mathematical operations. Text fingerprint truncation refers to a lightweight text feature obtained by truncating and normalizing the original text content corresponding to the retained node to a fixed length; text fingerprint truncation can be used to retain the core text features of the node while avoiding comparison interference caused by redundant characters and differences in naming length. Semantic importance weight refers to a pre-defined quantitative weight value based on the generalized semantic category to which a node belongs. Semantic importance weight can be used to distinguish the importance of different semantic functional nodes in code logic evaluation, with core functional nodes corresponding to higher semantic importance weight.
[0026] Optionally, server 101 can perform a preorder traversal of the target reference abstract syntax tree and the target candidate abstract syntax tree respectively. For each retained node, a depth label is generated based on the nesting depth of the retained node. The syntax node type of the retained node is mapped to a preset generalized semantic category, and a semantic importance weight is determined. The original text is extracted based on the node type and truncated to a fixed length to obtain a text fingerprint. Server 101 can retrieve the path sensitivity label of the retained node and concatenate the depth label, generalized semantic category, text fingerprint truncation, semantic importance weight, and path sensitivity label according to a preset delimiter to generate node lexical units. Server 101 can summarize all node lexical units in traversal order to generate a reference node sequence and a candidate node sequence.
[0027] S3. Based on semantic importance weights, perform semantically weighted longest common subsequence alignment on the reference node sequence and candidate node sequence, and calculate the basic alignment score.
[0028] Specifically, semantically weighted longest common subsequence alignment refers to a sequence matching method that introduces semantic importance weights for nodes to differentiate scores based on the longest common subsequence matching algorithm. A matching node pair refers to a node pair formed during sequence alignment when a reference node in the reference node sequence and a candidate node in the candidate node sequence meet preset matching conditions. The optimal matching node sequence can be used to represent the maximum overlap in structure and semantics between two code segments.
[0029] Optionally, server 101 can employ a semantically weighted longest common subsequence algorithm to globally align the reference node sequence and candidate node sequence, and calculate the optimal matching node sequence. When determining whether a reference node in the reference node sequence matches a candidate node in the candidate node sequence, server 101 can use the following necessary conditions: the depth label of the reference node is the same as the depth label of the candidate node; the generalized semantic category of the reference node is the same as the generalized semantic category of the candidate node; and the text fingerprint truncation of the reference node is the same as the text fingerprint truncation of the candidate node. When all three conditions are met, server 101 can determine the reference node and the candidate node as a valid matching node pair. For each valid matching node pair, server 101 can extract the semantic importance weight corresponding to the reference node and the semantic importance weight corresponding to the candidate node, calculate the product of the two semantic importance weights, and use this as the matching score for the valid matching node pair. Server 101 can sum the matching scores of all valid matching node pairs to obtain the total matching score of the optimal matching node sequence. Server 101 can calculate the sum of the semantic importance weights of each reference node in the reference node sequence to obtain the total semantic weight of the reference node sequence. Server 101 can also calculate the sum of the semantic importance weights of each candidate node in the candidate node sequence to obtain the total semantic weight of the candidate node sequence. Server 101 can use the total matching score of the optimal matching node sequence as the numerator, and the larger of the total semantic weight of the reference node sequence and the total semantic weight of the candidate node sequence as the denominator, and then normalize the result to obtain the basic alignment score.
[0030] S4. Extract nodes belonging to the preset key semantic categories from the reference node sequence and candidate node sequence, construct the reference key multiset and the candidate key multiset, and identify the nodes that exist in the reference key multiset but not in the candidate key multiset as missing nodes.
[0031] Specifically, key semantic categories refer to the key semantic categories that are pre-selected from generalized semantic categories and carry the core functions and core execution logic of the code. Key semantic categories can be used to filter secondary nodes and focus on the evaluation of the core logic of the code. Key semantic categories include control flow nodes, API call nodes, and core mathematical operation nodes. Missing nodes can be used to represent the core functions and key logic nodes that are missing in the candidate code.
[0032] Optionally, server 101 can retrieve a preset set of key semantic categories, traverse all node words in the reference node sequence, filter out node words belonging to the set of key semantic categories, retain the actual frequency of each node word in the reference node sequence, and generate a reference key multiset according to the multiset construction rules. Server 101 can traverse all node words in the candidate node sequence, filter out node words belonging to the set of key semantic categories, count the actual frequency of each node word in the candidate node sequence, and generate a candidate key multiset according to the multiset construction rules. Server 101 can perform element-by-element comparison between the reference key multiset and the candidate key multiset, traverse all node words in the reference key multiset, and use the same generalized semantic category and the same text fingerprint truncation as the criteria for determining node identity. Nodes that exist in the reference key multiset but do not exist in the candidate key multiset with the same corresponding node word are marked as missing nodes.
[0033] S5. Based on the non-negative truncation difference of node frequency between the reference key multiset and the candidate key multiset, the depth label and path sensitivity label of each missing node, and the number of missing nodes under each key semantic category, calculate the composite missing metric. Based on the composite missing metric and the basic alignment score, calculate the core evaluation score through a continuous exponential penalty function.
[0034] Specifically, the non-negative truncation difference of node frequency can be used to quantify the degree of missing core node frequency in candidate code compared to reference code. The hierarchical depth penalty value refers to the penalty value calculated based on the depth label of the missing node and a preset depth sensitivity coefficient. This value reflects the degree of impact of the nesting depth of the missing node on the logic; deeper nesting typically results in a more localized impact, and the penalty can be appropriately reduced. The path sensitivity penalty value refers to the penalty value determined based on the path sensitivity label bound to the missing node. This value can be quantified using a preset path sensitivity coefficient to reflect the differentiated impact of missing nodes under different execution paths. The error coupling penalty coefficient refers to the coefficient set to address the coupling amplification effect caused by the number of missing nodes in the same key semantic category. This coefficient can be used to achieve a coupling penalty effect where the more similar missing nodes, the stronger the penalty. The category missing measure can characterize the overall severity of missing nodes in that key semantic category. The composite missing measure refers to a multi-dimensional comprehensive quantitative index obtained by summing the category missing measures of each key semantic category. This index comprehensively characterizes the severity of missing core semantic logic in the candidate code and serves as the core input parameter for penalty calculation.
[0035] Optionally, server 101 can obtain all missing nodes under each key semantic category. For each key semantic category, server 101 can traverse each missing node under that key semantic category, obtain the frequency of occurrence of the node terminology corresponding to the missing node in the reference key multiset, use this frequency as the node frequency difference of the missing node, and sum the node frequency differences of all missing nodes under that key semantic category to obtain the non-negative truncated node frequency difference of that key semantic category. Server 101 can extract depth labels from the node terms of each missing node, and calculate the hierarchical depth penalty value of each missing node by combining it with a preset depth sensitivity coefficient. Server 101 can match the corresponding path sensitivity coefficient according to the path sensitivity label of each missing node to determine the path sensitivity penalty value of each missing node. Server 101 can count the number of similar missing nodes under each key semantic category, add 1 to the result of multiplying the preset coupling coefficient by the number of similar missing nodes, and obtain the error coupling penalty coefficient of each key semantic category. Server 101 can sum the hierarchical depth penalty values of all missing nodes under each key semantic category to obtain the depth penalty sum, sum the path sensitivity penalty values to obtain the path penalty sum, and then sum the depth penalty sum, path penalty sum, and non-negative truncation differences of node frequencies according to preset category weights, multiplying the sum by the error coupling penalty coefficient to obtain the category missing measure for each category. The cumulative category missing measures yield the composite missing measure. Server 101 can use the ratio of the composite missing measure to the total number of nodes in the reference key multi-convergence as the penalty index, inputting it into a continuous exponential penalty function to correct the basic alignment score, thus obtaining the core evaluation score.
[0036] S6. Calculate the statistical similarity score based on the character-level N-tuples of the reference code and candidate code. Adaptively weight the core evaluation score and statistical similarity score according to the structural complexity of the target reference abstract syntax tree and the target candidate abstract syntax tree to obtain the code evaluation score.
[0037] Specifically, a character-level N-tuple refers to a set of continuous character substrings obtained by truncating a complete code text using a single character as the basic unit and a sliding window of a preset fixed length N, where N is a preset positive integer, typically between 2 and 6. Character-level N-tuples can be used to finely represent the character layout, writing style, and surface text features of code text. Statistical similarity score refers to the degree of similarity between two code segments at the surface text level. Structural complexity can be used to objectively measure the nesting depth and logical scale of the code. Overall structural complexity can be used to comprehensively reflect the overall structural complexity level of the code pair being compared. Fusion weights can be used to adjust the proportion of the core evaluation score and the statistical similarity score in the final score.
[0038] Optionally, server 101 can extract character-level N-tuple sets from the reference code and candidate code respectively, and count the number of identical character substrings in the character-level N-tuple sets of the reference code and candidate code as the character matching count. Server 101 can calculate precision and recall, and obtain the statistical similarity score through a weighted harmonic average with equal weights. Server 101 can count the total number of nodes and the maximum depth label of the target reference abstract syntax tree and the target candidate abstract syntax tree respectively, divide the product of the total number of nodes and the maximum depth label by the number of lines of code of the corresponding code to obtain the reference structure complexity and the candidate structure complexity respectively, and take the average of the reference structure complexity and the candidate structure complexity as the overall structure complexity. Server 101 can input the overall structure complexity into a preset weight mapping function to obtain the fusion weight. Server 101 can perform weighted fusion of the core evaluation score and the statistical similarity score based on the fusion weight to obtain the code evaluation score. In the batch evaluation scenario, server 101 can summarize the code evaluation scores of each code pair, sort the candidate codes according to the scores, and generate a batch evaluation report.
[0039] The aforementioned sandbox-free AI-generated code evaluation method reduces the dimensionality of the abstract syntax tree to a path-sensitive node sequence by stripping redundant nodes and extracting multi-dimensional features such as depth, semantic category, and text fingerprint. This avoids the weight blind spot in traditional lexical statistics where key operators and ordinary symbols are weighted equally. It employs semantically weighted longest common subsequence alignment, with depth and category as necessary conditions, tolerating legitimate local code reconstructions and addressing the precipitous drop in scores caused by structural rigidity in tree isomorphic matching. By constructing a multiset of key semantic nodes and applying a composite penalty based on frequency, depth, and path sensitivity to missing nodes, and using a continuous exponential decay function to suppress basic alignment scores, it significantly amplifies the impact of fatal logical errors on the score, overcoming the shortcomings of excessive smoothing of deep continuous spatial features and the inability to identify discrete operator-level errors. Finally, it integrates character-level N-tuple statistical similarity and adaptively weights it according to structural complexity, ensuring strict micro-logical review while providing macro-topological tolerance, achieving a decoupling and balance between sensitivity and tolerance.
[0040] In an exemplary embodiment, abstract syntax tree parsing is performed on the reference code and candidate code to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree; after performing execution path sensitivity pre-analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree, redundant nodes are removed to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree with path sensitivity labels, including:
[0041] S11. Use a parsing tool to perform abstract syntax tree parsing on the reference code and candidate code respectively, to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree.
[0042] Specifically, a parsing tool can refer to a dedicated program component that can perform lexical and syntactic analysis on source code and generate an abstract syntax tree according to the syntax rules of the programming language. A parsing tool can be used to automatically convert source code in text form into a structured tree data representation. This parsing tool can be a tree-sitter or other syntax parsing components with equivalent functions.
[0043] Optionally, the server can input the complete text of the reference code into a parsing tool. The parsing tool reads the entire character sequence of the reference code text, performs a lexical scan, segments the character sequence into a series of lexical units, and labels each lexical unit with a lexical type, thus obtaining the lexical unit sequence of the reference code. The server can then use the parsing tool to perform syntactic structure analysis on the lexical unit sequence of the reference code according to preset programming language syntax rules. This analysis identifies the syntactic hierarchical relationships between the lexical units, organizes them into a tree-like node structure according to the syntactic hierarchy, assigns a corresponding node type identifier to each syntactic node in the tree structure, and determines the parent-child and sibling relationships between the syntactic nodes, generating a raw reference abstract syntax tree that completely covers all syntactic information of the reference code. Alternatively, the server can input the complete text of the candidate code into a parsing tool. The parsing tool reads the entire character sequence of the candidate code text, performs a lexical scan, segments the character sequence into a series of lexical units, and labels each lexical unit with a lexical type, thus obtaining the lexical unit sequence of the candidate code. The server can use parsing tools to perform syntactic structure analysis on the lexical unit sequence of candidate code according to preset programming language syntax rules, identify the syntactic hierarchy between lexical units in the candidate code's lexical unit sequence, organize each lexical unit into a tree node structure according to the syntactic hierarchy, assign a corresponding node type identifier to each syntactic node in the tree node structure, determine the parent-child and sibling relationships between each syntactic node, and generate an original candidate abstract syntax tree that completely covers all syntactic information of the candidate code.
[0044] S12. Perform control flow analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree respectively, and construct a control flow graph; establish the node association relationship between the control flow graph and the original reference abstract syntax tree and the original candidate abstract syntax tree, and obtain the path type of each node in the original reference abstract syntax tree and the original candidate abstract syntax tree. The path type includes mandatory path, conditional branch and unreachable branch.
[0045] Specifically, a control flow graph (CFB) can refer to a directed graph structure built with basic blocks as nodes and control flow transitions as edges. The CFB can be used to visually represent the execution order and jump relationships of code blocks during program execution. Node associations refer to the mapping relationship between basic blocks in the CFB and corresponding syntax nodes in the original abstract syntax tree (API). Node associations can be used to pass the execution path information obtained from control flow analysis back to the syntax nodes in the original API. Path types refer to the execution path attribute classification of each syntax node in the original API based on the control flow analysis results. Path types include mandatory paths, conditional branches, and unreachable branches. A mandatory path is an execution path where nodes on that path will definitely be executed regardless of which conditional branch the program chooses during runtime. Mandatory paths correspond to the main logic and core functionality of the program. Conditional branches are execution paths where the program needs to decide whether to execute nodes based on conditional judgments during runtime. Conditional branches correspond to branch statements, loop bodies, and conditional judgment logic in the program. An unreachable branch can refer to an execution path that will never be executed during program runtime because the condition is always false or there is a logical contradiction. Unreachable branches correspond to dead code and invalid logical branches.
[0046] Optionally, the server can perform control flow analysis on the original reference abstract syntax tree (API), traversing all syntax nodes in the API, identifying key control flow nodes, dividing the API into multiple basic blocks based on these nodes, analyzing the control transfer relationships between these basic blocks, using each basic block as a node in the control flow graph, and using the control flow transfers between basic blocks as edges in the control flow graph, thus constructing a reference control flow graph corresponding to the API. The server can establish node associations between each basic block in the reference control flow graph and the corresponding syntax nodes in the API, passing the execution path information of the reference control flow graph to each syntax node in the API. The server can determine the path type of syntax nodes appearing on all paths from the entry node to the exit node in the reference control flow graph as mandatory paths, the path type of syntax nodes appearing only partially on paths from the entry node to the exit node as conditional branches, and the path type of syntax nodes not appearing on any path from the entry node to the exit node as unreachable branches. The server can perform control flow analysis on the original candidate abstract syntax tree, traverse all syntax nodes in the original candidate abstract syntax tree, identify key control flow nodes in the original candidate abstract syntax tree, divide into basic blocks, construct the candidate control flow graph corresponding to the original candidate abstract syntax tree, establish the node association relationship between each basic block in the candidate control flow graph and the corresponding syntax node in the original candidate abstract syntax tree, and determine the path type of each syntax node in the original candidate abstract syntax tree based on the node association relationship.
[0047] S13. Mark each node of the path type as a required path as a critical path node, mark each node of the path type as a conditional branch as a conditional path node, and mark each node of the path type as an unreachable branch as an edge node. Assign path sensitivity labels to critical path nodes, conditional path nodes, and edge nodes to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree with path sensitivity labels.
[0048] Optionally, the server can traverse all syntax nodes in the original reference abstract syntax tree and read the path type of each syntax node. For syntax nodes whose path type is a required path, the server can mark the syntax node as a critical path node and assign it a corresponding path sensitivity label; for syntax nodes whose path type is a conditional branch, the server can mark the syntax node as a conditional path node and assign it a corresponding path sensitivity label; for syntax nodes whose path type is an unreachable branch, the server can mark the syntax node as an edge node and assign it a corresponding path sensitivity label. After completing the marking and label assignment of all syntax nodes, the original reference abstract syntax tree with path sensitivity labels is obtained. The server can also traverse all syntax nodes in the original candidate abstract syntax tree, mark each syntax node as a critical path node, conditional path node, or edge node according to its path type, and assign it a corresponding path sensitivity label, thus obtaining the original candidate abstract syntax tree with path sensitivity labels.
[0049] S14. Redundant nodes are stripped from the original reference abstract syntax tree and the original candidate abstract syntax tree with path-sensitive labels. Redundant syntactic structure nodes are removed from critical path nodes, conditional path nodes and edge nodes to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree.
[0050] Specifically, redundant syntax structure nodes refer to invalid syntax nodes in the original abstract syntax tree that do not participate in the core logic execution and make no substantial contribution to the code semantics and execution logic. Redundant syntax structure nodes include blank syntax nodes, comment-affiliated nodes, empty syntax placeholder nodes, invalid and repeated syntax nodes, and invalid redundant child nodes under unreachable branches. Retained nodes refer to the syntax nodes that remain in the target reference abstract syntax tree and the target candidate abstract syntax tree after the redundant nodes have been removed. Retained nodes are all non-redundant nodes that carry the core semantics and effective execution logic of the code.
[0051] Optionally, the server can identify redundant nodes in the original reference abstract syntax tree (API) with path sensitivity labels according to preset redundant node filtering rules. It traverses all syntax nodes marked as critical path nodes, conditional path nodes, and edge nodes in the original API, identifying syntax nodes that are blank syntax nodes, comment subsidiary nodes, empty syntax placeholder nodes, and invalid duplicate syntax nodes as redundant syntax structure nodes. For syntax nodes with path sensitivity labels as edge nodes, the non-functional child nodes under the edge node's syntax node are also identified as redundant syntax structure nodes. The server can remove all the identified redundant syntax structure nodes from the original API, retaining only the non-redundant nodes among the critical path nodes, conditional path nodes, and edge nodes, obtaining a target API with all retained nodes bound to path sensitivity labels. Alternatively, the server can traverse all syntax nodes marked as critical path nodes, conditional path nodes, and edge nodes in the original candidate API, identifying and removing redundant syntax structure nodes, obtaining a target candidate API with all retained nodes bound to path sensitivity labels.
[0052] In an exemplary embodiment, the deep labels, generalized semantic categories, text fingerprint truncation, and semantic importance weights of each retained node in the target reference abstract syntax tree and the target candidate abstract syntax tree are extracted. Feature concatenation is performed on the deep labels, generalized semantic categories, text fingerprint truncation, semantic importance weights, and path sensitivity labels to obtain a reference node sequence and a candidate node sequence, including:
[0053] S21. Perform a preorder traversal on the target reference abstract syntax tree and the target candidate abstract syntax tree respectively. Based on the syntax node type of each retained node, map each retained node to a preset generalized semantic category, and determine the semantic importance weight for each retained node based on the generalized semantic category.
[0054] Specifically, the syntax node type can refer to the original node type identifier corresponding to each syntax node in the abstract syntax tree in the parsing tool. The syntax node type can be used to distinguish different syntax components such as function definitions, variable declarations, control flow statements, operators, and identifiers.
[0055] Optionally, the server can perform a preorder traversal of the target reference abstract syntax tree (API). Starting from the root node of the API, it first visits the currently reserved node, then recursively visits each child node of the currently reserved node, visiting all reserved nodes in the API in a depth-first order. For each reserved node visited, the server can read the syntax node type determined during the API parsing phase, and map the syntax node type of the reserved node to a generalized semantic category according to a preset mapping rule between syntax node types and generalized semantic categories. The server can then look up the semantic importance weight value corresponding to the generalized semantic category from a preset semantic importance weight configuration table, and determine this semantic importance weight value as the semantic importance weight of the reserved node. The server can also use the same preorder traversal method for the target candidate API, starting from the root node of the target candidate API, sequentially visiting all reserved nodes in the target candidate API, reading the syntax node type of each reserved node and mapping it to the corresponding generalized semantic category, and determining the semantic importance weight of each reserved node based on the generalized semantic category.
[0056] S22. Based on the nesting depth of each reserved node in the target reference abstract syntax tree and the target candidate abstract syntax tree, obtain the depth label of each reserved node.
[0057] Specifically, the nesting depth can refer to the number of levels of the reserved node in the target abstract syntax tree from the root node. The nesting depth can be used to quantify the syntax nesting level of the reserved node. The nesting depth of the root node is the minimum value, and the nesting depth increases with each lower level of child nodes.
[0058] Optionally, the server can perform a depth-first traversal of the target reference abstract syntax tree (API), starting from the root node and setting the nesting depth of the root node as an initial value. During the traversal, the server can determine the nesting depth of each retained node by adding one to the nesting depth of its parent node in the tree, and use this nesting depth value as the depth label of the retained node. The server can also perform a depth-first traversal of the target candidate API in the same way, calculating the nesting depth of each retained node level by level starting from the root node, and using the nesting depth value of each retained node as its depth label.
[0059] S23. Determine the original text based on the node type of each retained node, and truncate the original text to a fixed length to obtain the text fingerprint of each retained node; among them, the retained node with source code identifier uses the source code identifier as the original text, and the retained node without source code identifier uses the name of the node type as the original text.
[0060] Specifically, node type can refer to the type identifier assigned to each syntax node by the abstract syntax tree parsing tool. Node type can be used to distinguish whether the node is an identifier node, literal node, operator node, keyword node, or other specific syntax category. Source code identifier can refer to the developer-defined naming symbols that preserve the corresponding variable names, function names, class names, and method names of the node in the original code. Source code identifier can directly reflect the specific semantic meaning of the node.
[0061] Optionally, the server can traverse all reserved nodes in the target reference abstract syntax tree. For the currently accessed reserved node, the server can read the node type of the reserved node and determine whether the reserved node contains a source code identifier. If the reserved node contains a source code identifier, the server can extract the string of the source code identifier as the original text of the reserved node; if the reserved node does not contain a source code identifier, the server can extract the name string of the node type of the reserved node as the original text of the reserved node. The server can perform fixed-length truncation on the original text of the reserved node, cutting the original text to a preset fixed length. If the original text length is less than the preset fixed length, it remains unchanged, obtaining the text fingerprint truncation corresponding to the reserved node. The server can use the same method for the target candidate abstract syntax tree, traversing all reserved nodes in the target candidate abstract syntax tree, determining whether each reserved node contains a source code identifier based on its node type, extracting the corresponding original text, and then performing fixed-length truncation to obtain the text fingerprint truncation corresponding to each reserved node.
[0062] S24. Concatenate the deep labels, generalized semantic categories, text fingerprint truncation, semantic importance weights, and path sensitivity labels of each retained node according to a preset delimiter to generate node lexical units corresponding to each retained node.
[0063] Specifically, a preset separator can refer to a special character or string that is set in advance to separate different feature fields. The preset separator can be used to clearly distinguish the boundaries of each feature field in the concatenated node words, and avoid the content of different feature fields from sticking together and causing feature confusion.
[0064] Optionally, for each retained node in the target reference abstract syntax tree, the server can sequentially obtain the depth label, generalized semantic category, text fingerprint truncation, semantic importance weight, and path sensitivity label of that retained node. The server can convert these elements into strings in a preset order, inserting a preset separator between each pair of adjacent feature fields, and concatenating them to generate a unique node terminology for that retained node. The server can apply the same method to the target candidate abstract syntax tree, sequentially obtaining the depth label, generalized semantic category, text fingerprint truncation, semantic importance weight, and path sensitivity label of each retained node, concatenating them in the same preset order and using the same preset separator to generate a unique node terminology for that retained node.
[0065] Preferably, the formula for calculating node lexical units can be:
[0066]
[0067] in, To preserve nodes The corresponding node words, To preserve nodes Depth tags, To preserve nodes The generalized semantic category, To preserve nodes Text fingerprint truncation, To preserve nodes semantic importance weights, To preserve nodes Path-sensitive tags, This is a preset delimiter used to separate the feature fields. This is a string concatenation operator.
[0068] S25. Gather the node tokens corresponding to each reserved node in the target reference abstract syntax tree to obtain the reference node sequence, and gather the node tokens corresponding to each reserved node in the target candidate abstract syntax tree to obtain the candidate node sequence.
[0069] Optionally, the server can sequentially obtain the node tokens corresponding to each retained node according to the node access order of the preorder traversal of the target reference abstract syntax tree, arrange the node tokens of each retained node in the traversal order, and combine them to generate a reference node sequence. Alternatively, the server can sequentially obtain the node tokens corresponding to each retained node according to the node access order of the preorder traversal of the target candidate abstract syntax tree, arrange the node tokens of each retained node in the traversal order, and combine them to generate a candidate node sequence.
[0070] In an exemplary embodiment, based on semantic importance weights, semantically weighted longest common subsequence alignment is performed on the reference node sequence and the candidate node sequence, and a basic alignment score is calculated, including:
[0071] S31. The semantically weighted longest common subsequence algorithm is used to globally align the reference node sequence and the candidate node sequence and calculate the optimal matching node sequence. When judging whether the reference node in the reference node sequence and the candidate node in the candidate node sequence match, the same deep label, generalized semantic category and text fingerprint truncation are used as necessary conditions. When the necessary conditions are met, the product of the semantic importance weights of the reference node and the candidate node is used as the matching score.
[0072] Specifically, the semantically weighted longest common subsequence algorithm refers to a sequence alignment algorithm that, based on the traditional longest common subsequence matching algorithm, introduces semantic importance weights to differentiate the scoring of matching node pairs. It uses three dimensions—depth label, generalized semantic category, and text fingerprint truncation—to constitute the necessary conditions for node matching. The necessary conditions refer to the three conditions that must be simultaneously satisfied when determining whether a reference node and a candidate node constitute a valid match. These three conditions are: the depth label of the reference node is the same as the depth label of the candidate node; the generalized semantic category of the reference node is the same as the generalized semantic category of the candidate node; and the text fingerprint truncation of the reference node is the same as the text fingerprint truncation of the candidate node. When all three conditions are satisfied, the reference node and the candidate node are considered a valid matching node pair.
[0073] Optionally, the server can invoke the semantically weighted longest common subsequence algorithm, taking the reference node sequence and candidate node sequence as input, and globally aligning the reference node sequence and candidate node sequence. During the search for common subsequences, the server can, for each reference node in the reference node sequence and each candidate node in the candidate node sequence, sequentially determine whether the depth label of the reference node is the same as the depth label of the candidate node, whether the generalized semantic category of the reference node is the same as the generalized semantic category of the candidate node, and whether the text fingerprint truncation of the reference node is the same as the text fingerprint truncation of the candidate node. When all three conditions are met, the server can determine the reference node and the candidate node as a valid matching node pair, and use the product of the semantic importance weight of the reference node and the semantic importance weight of the candidate node as the matching score of the valid matching node pair. The server can use dynamic programming to find the matching path that maximizes the sum of the matching scores of all valid matching node pairs, while satisfying the node sequence order constraint. The server then arranges the valid matching node pairs on this matching path into an optimal matching node sequence according to the sequence order, and records the total matching score of the optimal matching node sequence.
[0074] Preferably, the calculation formula for the semantically weighted longest common subsequence algorithm can be:
[0075]
[0076] in, Align scores based on the base The optimal matching node sequence, This represents a set of valid matching node pairs in the optimal matching node sequence. For the node index in the reference node sequence, This represents the node index in the candidate node sequence. Reference node semantic importance weights, Reference node semantic importance weights, The matching score for this group of matching node pairs. The total number of nodes in the reference node sequence. The total number of nodes in the candidate node sequence. For the reference node sequence, the first The semantic importance weight of each node, The first in the candidate node sequence The semantic importance weight of each node.
[0077] S32. The basic alignment score is calculated by taking the sum of the matching scores of each matching node pair in the optimal matching node sequence as the numerator and the larger value of the sum of the semantic importance weights of the reference node sequence and the sum of the semantic importance weights of the candidate node sequence as the denominator.
[0078] Optionally, the server can iterate through all valid matching node pairs in the optimal matching node sequence, extract the matching score for each valid matching node pair, and sum the matching scores of all valid matching node pairs to obtain the total matching score of the optimal matching node sequence. The server can separately calculate the sum of the semantic importance weights corresponding to each retained node in the reference node sequence to obtain the total reference semantic importance weight, and the sum of the semantic importance weights corresponding to each retained node in the candidate node sequence to obtain the total candidate semantic importance weight. The server can compare the total reference semantic importance weight with the total candidate semantic importance weight, and take the larger value as the normalized denominator. The server can use the total matching score as the numerator, divide it by the normalized denominator, and obtain the basic alignment score through division.
[0079] In an exemplary embodiment, a composite missingness metric is calculated based on the non-negative truncation difference in node frequency between the reference key multiset and the candidate key multiset, the depth label of each missing node, the path sensitivity label of each missing node, and the number of missing nodes under each key semantic category. Based on the composite missingness metric and the base alignment score, a core evaluation score is calculated using a continuous exponential penalty function, including:
[0080] S41. Obtain the frequency of the node terminology corresponding to each missing node under each key semantic category in the reference key multiset, and obtain the node frequency difference of each missing node; sum the node frequency differences of each missing node according to the key semantic category to obtain the non-negative truncated difference of node frequency for each key semantic category.
[0081] Optionally, the server can obtain all missing nodes under each key semantic category. For each key semantic category, the server can iterate through each missing node under that key semantic category, obtain the frequency of the node terminology corresponding to the missing node in the reference key multiset, and use this frequency as the node frequency difference of the missing node. The server can sum the node frequency differences of all missing nodes under that key semantic category to obtain the non-negative truncated node frequency difference of that key semantic category. The server 101 performs the above operation on all key semantic categories to obtain the non-negative truncated node frequency difference corresponding to each key semantic category.
[0082] S42. Extract depth labels from the node words corresponding to each missing node, and calculate the hierarchical depth penalty value of each missing node based on the preset depth sensitivity coefficient and depth labels.
[0083] Specifically, the preset depth sensitivity coefficient can refer to a preset parameter used to adjust the intensity of the influence of depth on the penalty value. The preset depth sensitivity coefficient determines the decay of the hierarchical depth penalty value as the depth changes, so as to adapt to the sensitivity preferences of different evaluation scenarios for the lack of deep and shallow layers. The preset depth sensitivity coefficient can be set according to the actual evaluation needs.
[0084] Optionally, for each missing node, the server can parse the depth label field from the node tokens corresponding to the missing node. The server can read a preset depth sensitivity coefficient, and calculate the hierarchical depth penalty value of the missing node according to a preset depth penalty calculation formula based on the preset depth sensitivity coefficient and the depth label of the missing node. The server can perform the above-mentioned depth label extraction and hierarchical depth penalty value calculation operations for each missing node to obtain the hierarchical depth penalty value corresponding to each missing node.
[0085] Preferably, the formula for calculating the hierarchical depth penalty value can be:
[0086]
[0087] in, For hierarchical depth penalty values, The depth label for the missing node. This is the depth sensitivity coefficient.
[0088] S43. Determine the path sensitivity coefficient based on the path sensitivity label, and use the path sensitivity coefficient as the path sensitivity penalty value for each missing node; wherein, the path sensitivity coefficient of the critical path node in the missing node is greater than the path sensitivity coefficient of the conditional path node in the missing node, and the path sensitivity coefficient of the conditional path node is greater than the path sensitivity coefficient of the edge node in the missing node.
[0089] Optionally, the server can pre-configure a path sensitivity coefficient mapping table before the evaluation task begins. For each missing node, the server can parse the path sensitivity label from the node tokens corresponding to the missing node to determine whether the path sensitivity label of the missing node belongs to a critical path node, a conditional path node, or an edge node. The server can then look up the corresponding path sensitivity coefficient from the pre-defined path sensitivity coefficient mapping table based on the path sensitivity label of the missing node, and use this path sensitivity coefficient as the path sensitivity penalty value for the missing node. The server can perform the above path sensitivity label parsing, path sensitivity coefficient lookup, and path sensitivity penalty value assignment operations for each missing node to obtain the path sensitivity penalty value corresponding to each missing node. Specifically, critical path nodes correspond to the first path sensitivity coefficient, conditional path nodes correspond to the second path sensitivity coefficient, and edge nodes correspond to the third path sensitivity coefficient. The first path sensitivity coefficient is greater than the second path sensitivity coefficient, and the second path sensitivity coefficient is greater than the third path sensitivity coefficient.
[0090] S44. Count the number of missing nodes of the same type under each key semantic category, and add 1 to the result of multiplying the preset coupling coefficient by the number of missing nodes of the same type to obtain the error coupling penalty coefficient under each key semantic category.
[0091] Specifically, the number of missing nodes of the same type can refer to the total number of nodes identified as missing within the same key semantic category. A preset coupling coefficient can be used to control the amplification rate of the penalty due to the number of missing nodes of the same type. The coupling coefficient can determine how the penalty intensity increases rapidly with the increase of the number of missing nodes when multiple nodes of the same category are missing.
[0092] Optionally, the server can count the number of missing nodes under each key semantic category to obtain the number of similar missing nodes for each key semantic category. The server can read a preset coupling coefficient. For each key semantic category, the server can multiply the read preset coupling coefficient by the number of similar missing nodes for that key semantic category to obtain the coupling increment. The server can add a constant 1 to the coupling increment to obtain the error coupling penalty coefficient for that key semantic category.
[0093] S45. Summing the hierarchical depth penalty values of each missing node under each key semantic category yields the depth penalty sum; summing the path sensitivity penalty values of each missing node under each key semantic category yields the path penalty sum; weighting the depth penalty sum, path penalty sum, and the non-negative truncation difference of the node frequency of the key semantic category according to the preset category weights, and then multiplying them by the error coupling penalty coefficient to obtain the category missing measure of each key semantic category; summing the category missing measures yields the composite missing measure.
[0094] Specifically, the preset category weights can refer to the preset weight coefficients for the three dimensions of depth penalty sum, path penalty sum, and non-negative truncation difference of node frequency, respectively. The preset category weights can be used to adjust the relative importance of the three dimensions in the basic category missing measure.
[0095] Optionally, for each key semantic category, the server can iterate through all missing nodes under the current key semantic category, obtain the hierarchical depth penalty value for each missing node, and sum the hierarchical depth penalty values of all missing nodes under the current key semantic category to obtain the depth penalty sum for the current key semantic category. The server can also iterate through all missing nodes under the current key semantic category, obtain the path sensitivity penalty value for each missing node, and sum the path sensitivity penalty values of all missing nodes under the current key semantic category to obtain the path penalty sum for the current key semantic category. The server can obtain the non-negative truncation difference of node frequency for the current key semantic category. The server can read preset category weights, multiply the depth penalty sum by the depth dimension weight, multiply the path penalty sum by the path dimension weight, multiply the non-negative truncation difference of node frequency by the frequency dimension weight, and add these three products to obtain the basic category missing metric for the current key semantic category. The server can obtain the error coupling penalty coefficient for the current key semantic category, multiply the basic category missing metric for the current key semantic category by the error coupling penalty coefficient to obtain the category missing metric for the current key semantic category. The server can sum the category missing metrics of all key semantic categories to obtain a composite missing metric.
[0096] S46. The ratio of the composite missing measure to the total number of reference key multi-centralized nodes is used as the penalty index. Based on the basic alignment score and the penalty index, the core evaluation score is calculated through a continuous exponential penalty function.
[0097] Optionally, the server can calculate the ratio of the composite missing metric to the total number of reference key multi-set nodes, using this ratio as a penalty index. The server can obtain the base alignment score and read a preset smoothing decay coefficient. The server can then input the base alignment score and penalty index into a continuous exponential penalty function to calculate the core evaluation score.
[0098] In an exemplary embodiment, the formula for calculating the continuous exponential penalty function is as follows:
[0099] ;
[0100] in, As the core assessment score, Align scores based on the base For composite missing data measurement, To reference the total number of nodes in the key multiset, The preset smooth attenuation coefficient, It is an exponential function.
[0101] In one exemplary embodiment, such as Figure 3 As shown, statistical similarity scores are calculated based on character-level N-tuples of the reference code and candidate code. Adaptive weighting of the core evaluation score and statistical similarity score is then applied to the target reference abstract syntax tree and target candidate abstract syntax tree based on their structural complexity, resulting in a code evaluation score, including:
[0102] S51. Extract character-level N-tuples from the reference code and candidate code respectively. Calculate the number of character matches between the character-level N-tuples of the reference code and the character-level N-tuples of the candidate code. Calculate the precision based on the ratio of the number of character matches to the total number of character-level N-tuples in the candidate code. Calculate the recall based on the ratio of the number of character matches to the total number of character-level N-tuples in the reference code. Calculate the statistical similarity score based on the weighted harmonic mean of precision and recall.
[0103] Optionally, the server can perform character-level N-tuple extraction on both the complete text of the reference code and the complete text of the candidate code. The server can use a preset fixed window size N and a single character step size to sequentially extract consecutive character substrings of length N from the first character of the reference code text, and collect all extracted substrings into a character-level N-tuple set for the reference code. The server can perform the same operation on the candidate code text, using the same window size N and single character step size, sliding and extracting substrings from the first character of the candidate code text, and collecting all extracted substrings into a character-level N-tuple set for the candidate code. The server can count the number of character substrings that appear simultaneously in both the reference code and candidate code character-level N-tuple sets to obtain the character matching count. The server can also count the total number of character substrings in the candidate code character-level N-tuple set, divide the character matching count by the total number of candidate code character-level N-tuple sets, and calculate the precision. The server can count the total number of character substrings in the reference code character-level N-tuple set, divide the number of character matches by the total number of reference code character-level N-tuple sets, and calculate the recall rate. Based on the calculated precision and recall, the server can use a weighted harmonic average to calculate the statistical similarity score. When equal weights are used, the statistical similarity score is equal to twice the product of precision and recall divided by the sum of precision and recall.
[0104] S52. Based on the total number of nodes in the target reference abstract syntax tree and the target candidate abstract syntax tree, the maximum value of the depth label of each retained node, and the number of lines of code in the reference code and the candidate code, calculate the reference structure complexity and the candidate structure complexity respectively, and take the average of the reference structure complexity and the candidate structure complexity as the overall structure complexity.
[0105] Specifically, the total number of nodes can refer to the total number of nodes retained in the target reference abstract syntax tree or target candidate abstract syntax tree after removing redundant nodes. The total number of nodes can be used to reflect the structural scale of the code after simplification. The maximum depth label can refer to the maximum value of the depth labels of all retained nodes in the target reference abstract syntax tree or target candidate abstract syntax tree, that is, the maximum nesting level of the code. The maximum depth label can be used to reflect the nesting complexity of the code. The number of lines of code can refer to the effective number of lines of code in the original text of the reference code or candidate code. The number of lines of code can be used to measure the physical length of the code.
[0106] Optionally, the server can separately count the total number of retained nodes in the target reference abstract syntax tree and the total number of retained nodes in the target candidate abstract syntax trees. The server can iterate through the depth labels of all retained nodes in the target reference abstract syntax tree, selecting the largest depth label value as the reference maximum depth. Similarly, the server can iterate through the depth labels of all retained nodes in the target candidate abstract syntax trees, selecting the largest depth label value as the candidate maximum depth. The server can obtain the effective lines of code in the reference code and the candidate code, multiply the total number of retained nodes in the target reference abstract syntax tree by the reference maximum depth, and then divide by the number of lines of code in the reference code to obtain the reference structural complexity. The server can use the same calculation method, multiplying the total number of retained nodes in the target candidate abstract syntax tree by the candidate maximum depth and then dividing by the number of lines of code in the candidate code to obtain the candidate structural complexity. The server can calculate the arithmetic mean of the reference structural complexity and the candidate structural complexity, and use this average as the overall structural complexity.
[0107] S53. Calculate the fusion weights using a weight mapping function based on the overall structural complexity; then, weightedly fuse the core evaluation score and the statistical similarity score based on the fusion weights to obtain the code evaluation score.
[0108] Optionally, server 101 inputs the overall structural complexity into a preset weight mapping function. The weight mapping function, with the overall structural complexity as the independent variable, outputs a fusion weight value that monotonically decreases as the overall structural complexity increases. The server can multiply the fusion weight by the core evaluation score to obtain a weighted term for the core evaluation score. The difference between 1 and the fusion weight is used as the weight for the statistical similarity score. This weight is then multiplied by the statistical similarity score to obtain a weighted term for the statistical similarity score. The server can then add the weighted term for the core evaluation score to the weighted term for the statistical similarity score to obtain the final code evaluation score.
[0109] Preferably, the formula for calculating the weight mapping function can be:
[0110]
[0111] in, To integrate weights, For the overall structural complexity, The preset positive scale parameter.
[0112] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0113] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.
Claims
1. A sandbox-free method for evaluating AI-generated code, characterized in that, The method includes: S1. Obtain reference code and candidate code, perform abstract syntax tree parsing on the reference code and candidate code to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree; perform execution path sensitivity pre-analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree and remove redundant nodes to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree with path sensitivity labels. S2. Extract the deep label, generalized semantic category, text fingerprint truncation, and semantic importance weight of each retained node in the target reference abstract syntax tree and the target candidate abstract syntax tree, and concatenate them with the path sensitivity label to obtain the reference node sequence and the candidate node sequence. S3. Based on the semantic importance weight, perform semantically weighted longest common subsequence alignment on the reference node sequence and the candidate node sequence, and calculate the basic alignment score; S4. Extract nodes belonging to a preset key semantic category from the reference node sequence and the candidate node sequence, construct a reference key multiset and a candidate key multiset, and determine the nodes that exist in the reference key multiset but not in the candidate key multiset as missing nodes based on the generalized semantic category and the text fingerprint truncation. S5. Based on the non-negative truncation difference of node frequency between the reference key multiset and the candidate key multiset, the depth label and path sensitivity label of each missing node, and the number of missing nodes under each key semantic category, calculate the composite missing metric. Based on the composite missing metric and the basic alignment score, calculate the core evaluation score through a continuous exponential penalty function. S6. Calculate the statistical similarity score based on the character-level N-tuples of the reference code and the candidate code, and perform adaptive weight fusion of the core evaluation score and the statistical similarity score according to the structural complexity of the target reference abstract syntax tree and the target candidate abstract syntax tree to obtain the code evaluation score.
2. The method according to claim 1, characterized in that, The reference code and the candidate code are parsed using an abstract syntax tree to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree. After performing execution path sensitivity pre-analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree, redundant nodes are removed to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree with path sensitivity labels, including: S11. The reference code and the candidate code are parsed using a parsing tool to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree; S12. Perform control flow analysis on the original reference abstract syntax tree and the original candidate abstract syntax tree respectively, and construct a control flow graph; establish the node association relationship between the control flow graph and the original reference abstract syntax tree and the original candidate abstract syntax tree, and obtain the path type of each node in the original reference abstract syntax tree and the original candidate abstract syntax tree. The path type includes mandatory path, conditional branch and unreachable branch. S13. Mark each node whose path type is the required path as a critical path node, mark each node whose path type is the conditional branch as a conditional path node, mark each node whose path type is the unreachable branch as an edge node, and assign the path sensitivity label to the critical path node, the conditional path node and the edge node to obtain the original reference abstract syntax tree and the original candidate abstract syntax tree with the path sensitivity label; S14. Redundant nodes are stripped from the original reference abstract syntax tree and the original candidate abstract syntax tree with the path sensitivity label. Redundant syntax structure nodes are removed from the critical path nodes, the conditional path nodes and the edge nodes to obtain the target reference abstract syntax tree and the target candidate abstract syntax tree.
3. The method according to claim 1, characterized in that, The process involves extracting the deep labels, generalized semantic categories, text fingerprint truncation, and semantic importance weights of each retained node in the target reference abstract syntax tree and the target candidate abstract syntax tree, and concatenating these features with the path sensitivity labels to obtain a reference node sequence and a candidate node sequence, including: S21. Perform a preorder traversal on the target reference abstract syntax tree and the target candidate abstract syntax tree respectively. Based on the syntax node type of each reserved node, map each reserved node to a preset generalized semantic category, and determine the semantic importance weight for each reserved node based on the generalized semantic category. S22. Based on the nesting depth of each reserved node in the target reference abstract syntax tree and the target candidate abstract syntax tree, obtain the depth label of each reserved node; S23. Determine the original text according to the node type of each of the reserved nodes, and truncate the original text to a fixed length to obtain the text fingerprint of each of the reserved nodes; wherein, the reserved node with source code identifier uses the source code identifier as the original text, and the reserved node without source code identifier uses the name of the node type as the original text. S24. The depth label, generalized semantic category, text fingerprint truncation, semantic importance weight and path sensitivity label of each of the retained nodes are concatenated into strings according to a preset separator to generate node lexical units corresponding to each of the retained nodes; S25. Gather the node lexical units corresponding to each of the reserved nodes in the target reference abstract syntax tree to obtain the reference node sequence, and gather the node lexical units corresponding to each of the reserved nodes in the target candidate abstract syntax tree to obtain the candidate node sequence.
4. The method according to claim 3, characterized in that, The step of aligning the reference node sequence and the candidate node sequence using a semantically weighted longest common subsequence based on the semantic importance weight, and calculating the basic alignment score, includes: S31. Using the semantically weighted longest common subsequence algorithm, the reference node sequence and the candidate node sequence are globally aligned to calculate the optimal matching node sequence; wherein, when determining whether the reference node in the reference node sequence and the candidate node in the candidate node sequence match, the same deep label, the generalized semantic category and the text fingerprint truncation are taken as necessary conditions, and when the necessary conditions are met, the product of the semantic importance weights of the reference node and the candidate node is taken as the matching score; S32. The basic alignment score is calculated by taking the sum of the matching scores of each matching node pair in the optimal matching node sequence as the numerator and the larger value of the sum of the semantic importance weights of the reference node sequence and the sum of the semantic importance weights of the candidate node sequence as the denominator.
5. The method according to claim 1, characterized in that, The method involves calculating a composite missing metric based on the non-negative truncation difference of node frequencies between the reference key multiset and the candidate key multiset, the depth label and path sensitivity label of each missing node, and the number of missing nodes under each key semantic category. Based on the composite missing metric and the base alignment score, a core evaluation score is calculated using a continuous exponential penalty function, including: S41. Obtain the frequency of the node terminology corresponding to each missing node under each of the key semantic categories in the reference key multiset, and obtain the node frequency difference of each missing node; sum the node frequency difference of each missing node according to the key semantic category to obtain the non-negative truncated difference of the node frequency of each of the key semantic categories. S42. Extract the depth label from the node lexical corresponding to each missing node, and calculate the hierarchical depth penalty value of each missing node based on the preset depth sensitivity coefficient and the depth label. S43. Determine the path sensitivity coefficient based on the path sensitivity label, and use the path sensitivity coefficient as the path sensitivity penalty value for each missing node; wherein, the path sensitivity coefficient of the critical path node in the missing node is greater than the path sensitivity coefficient of the conditional path node in the missing node, and the path sensitivity coefficient of the conditional path node is greater than the path sensitivity coefficient of the edge node in the missing node. S44. Count the number of missing nodes of the same type under each of the key semantic categories, and add 1 to the result of multiplying the preset coupling coefficient by the number of missing nodes of the same type to obtain the error coupling penalty coefficient under each of the key semantic categories. S45. Summing the hierarchical depth penalty values of the missing nodes under each of the key semantic categories yields a depth penalty sum; summing the path sensitivity penalty values of the missing nodes under each of the key semantic categories yields a path penalty sum; weighting the depth penalty sum, the path penalty sum, and the non-negative truncation difference of the node frequency of the key semantic category according to a preset category weight, and multiplying the sum by the error coupling penalty coefficient, yields the category missing metric for each of the key semantic categories; summing the category missing metrics yields the composite missing metric. S46. Using the ratio of the composite missing metric to the total number of reference key multi-centralized nodes as a penalty index, the core evaluation score is calculated based on the base alignment score and the penalty index using the continuous exponential penalty function.
6. The method according to claim 5, characterized in that, The formula for calculating the continuous exponential penalty function is as follows: ; in, The core assessment score is... The basic alignment score, For the composite missing measure, The total number of nodes in the reference key multiset. The preset smooth attenuation coefficient, It is an exponential function.
7. The method according to claim 1, characterized in that, The statistical similarity score is calculated based on the character-level N-tuples of the reference code and the candidate code. An adaptive weighted fusion of the core evaluation score and the statistical similarity score is performed according to the structural complexity of the target reference abstract syntax tree and the target candidate abstract syntax tree to obtain the code evaluation score, including: S51. Extract the character-level N-tuples from the reference code and the candidate code respectively; calculate the number of character matches between the character-level N-tuples of the reference code and the character-level N-tuples of the candidate code; calculate the precision based on the ratio of the number of character matches to the total number of character-level N-tuples in the candidate code; calculate the recall based on the ratio of the number of character matches to the total number of character-level N-tuples in the reference code; calculate the statistical similarity score based on the weighted harmonic average of the precision and the recall. S52. Based on the total number of nodes in the target reference abstract syntax tree and the target candidate abstract syntax tree, the maximum value of the depth label of each of the retained nodes, and the number of lines of code in the reference code and the candidate code, calculate the reference structure complexity and the candidate structure complexity respectively, and take the average of the reference structure complexity and the candidate structure complexity as the overall structure complexity. S53. Calculate the fusion weight using a weight mapping function based on the overall structural complexity; and perform a weighted fusion of the core evaluation score and the statistical similarity score based on the fusion weight to obtain the code evaluation score.
