Code evaluation method and device, electronic equipment, medium and program product

Abstract

Embodiments of the present application disclose a code evaluation method and device, electronic equipment, medium and program product. The method comprises: obtaining target code features corresponding to a code to be evaluated, and obtaining target specification rule information; performing multi-dimensional consistency verification on the target code features based on the target specification rule information to obtain verification results corresponding to a plurality of verification dimensions respectively; performing specification compliance degree evaluation on the code to be evaluated based on the verification results corresponding to the plurality of verification dimensions respectively to obtain a specification evaluation result corresponding to the code to be evaluated, wherein the specification evaluation result comprises a specification compliance level and / or defect repair information; and generating a specification compliance degree evaluation report corresponding to the code to be evaluated based on the specification evaluation result.

Description

Technical Field

[0001] This application relates to the field of code evaluation technology, and in particular to a code evaluation method, apparatus, electronic device, medium, and program product. Background Technology

[0002] As artificial intelligence technology is increasingly applied in software development, automatically generating code using AI models or development tools has become a common development method. To ensure the generated code meets pre-defined development requirements, it is typically necessary to evaluate it.

[0003] In related technologies, code evaluation methods mostly rely on manual review, static code analysis, or functional testing. Static code analysis mainly targets syntax, encoding format, etc., while functional testing mainly focuses on whether the code can achieve the expected function. It is difficult to conduct more efficient and reliable evaluation of the generated code. Summary of the Invention

[0004] This application provides a code evaluation method, apparatus, electronic device, medium, and program product, which can improve the efficiency and reliability of code evaluation. The above technical solution is as follows: In a first aspect, embodiments of this application provide a code evaluation method, including: Obtain the target code features corresponding to the code to be evaluated, and obtain the target specification rule information. The target specification rule information is used to characterize the specification requirements that the code to be evaluated should meet. Based on the above target specification rules information, multi-dimensional consistency verification is performed on the above target code features to obtain the verification results corresponding to multiple verification dimensions. Based on the verification results corresponding to the above multiple verification dimensions, the code to be evaluated is evaluated for compliance with the standards, and the standard evaluation result corresponding to the code to be evaluated is obtained. The standard evaluation result includes the standard compliance level and / or defect repair information. Based on the above standard evaluation results, a standard compliance evaluation report is generated for the above-mentioned code to be evaluated.

[0005] In one possible implementation, obtaining the target code features corresponding to the code to be evaluated includes: Obtain the code to be evaluated; Code cleaning and abstract syntax tree parsing are performed on the code to be evaluated to determine multiple code objects in the code to be evaluated and the code relationships between the multiple code objects. Based on multiple code objects and the code relationships described above, code behavior paths and corresponding execution conditions and behaviors are determined. The code behavior paths are used to characterize the associated sequence formed by at least two code objects according to their code execution relationships. Based on multiple code objects, code relationships, code behavior paths, execution conditions, and execution behaviors, feature extraction is performed to obtain target code features.

[0006] In one possible implementation, obtaining the target specification rule information includes: Obtain preset standard data; The above-mentioned preset standard data is structured and parsed to determine multiple standard objects and the constraint information corresponding to each standard object. The constraint information includes at least one of constraint conditions and expected execution behavior. Based on the above-mentioned normative objects and the constraint information corresponding to each of the above-mentioned normative objects, the normative association relationship between the above-mentioned normative objects is determined. Based on multiple normative objects, the constraint information corresponding to each normative object, and the normative relationship, a normative constraint path is determined. The normative constraint path is used to characterize the constraint relationship formed by at least two normative objects according to the normative relationship. Based on the above-mentioned normative constraint path, target normative rule information is generated.

[0007] In one possible implementation, based on the aforementioned target specification rule information, a multi-dimensional consistency check is performed on the aforementioned target code features to obtain check results corresponding to multiple check dimensions, including: Based on the normative constraint paths in the above target normative rule information, candidate code behavior paths corresponding to the above normative constraint paths are determined from the above code behavior paths. The consistency comparison between the normative objects, constraints, expected execution behaviors, and normative relationships in the above-mentioned normative constraint paths and the code objects, execution conditions, execution behaviors, and code relationships in the above-mentioned candidate code behavior paths was performed, resulting in multiple consistency comparison results. Based on the above consistency comparison results, the results are classified according to the corresponding specification types to obtain the verification results corresponding to multiple verification dimensions.

[0008] In one possible implementation, the code to be evaluated is assessed for compliance with specifications based on the verification results corresponding to the multiple verification dimensions, resulting in a specification evaluation result for the code to be evaluated, including: Based on the verification results corresponding to the above multiple verification dimensions, the standard compliance sub-scores corresponding to the above multiple verification dimensions are determined. Based on at least one of the rule priority, rule quantity, and preset violation severity level of the normative rules corresponding to the above-mentioned verification dimensions in the above-mentioned target normative rule information, the evaluation weights corresponding to the above-mentioned verification dimensions are determined respectively. The compliance score is obtained by weighting multiple compliance scores and their corresponding evaluation weights. Based on the above compliance score, a compliance level is determined. Based on the above compliance levels, the compliance assessment results are obtained.

[0009] In one possible implementation, the code to be evaluated is assessed for compliance with specifications based on the verification results corresponding to the multiple verification dimensions, resulting in a specification evaluation result for the code to be evaluated, including: If at least one of the verification results corresponding to the above-mentioned verification dimensions has a specification inconsistency, the target verification result with specification inconsistency shall be determined from the above-mentioned verification results. Based on the above target verification results, the corresponding target specification constraint path and the target code behavior path corresponding to the above target specification constraint path are determined. Identify the inconsistencies between the target specification constraint path and the target code behavior path, and locate the defects in the code to be evaluated based on the inconsistencies to obtain defect location information. Based on the above defect location information and the constraint information in the above target specification constraint path, a repair suggestion corresponding to the above defect location information is generated. Based on the above defect location information and the above repair suggestions, defect repair information is obtained, and based on the above defect repair information, the standard assessment result is obtained.

[0010] In one possible implementation, based on the above-mentioned specification evaluation results, a specification compliance evaluation report corresponding to the code to be evaluated is generated, including: The above-mentioned standard assessment results are classified and processed to obtain assessment content data, which includes grade assessment data and / or defect assessment data. When the aforementioned assessment content data includes the aforementioned defect assessment data, based on the aforementioned target specification rule information, the specification basis information corresponding to the aforementioned defect assessment data is determined, and a mapping relationship is established between the aforementioned defect assessment data and the aforementioned specification basis information. Based on the assessment content data, the data is organized according to a preset report structure. When the assessment content data includes the defect assessment data, the defect assessment data and the corresponding standard basis information are organized based on the mapping relationship to generate a standard compliance assessment report.

[0011] In one possible implementation, the determination of the specification constraint path based on multiple specification objects, the constraint information corresponding to each specification object, and the specification relationships includes: From the constraint information corresponding to each of the above-mentioned standard objects, identify multiple conflicting constraint information that act on the same standard object and conflict with each other; Based on at least one of the specification priority, version information and scope of application corresponding to the above-mentioned conflicting constraint information, conflict resolution is performed on the above-mentioned conflicting constraint information to determine the effective constraint information. Based on the above effective constraint information and other constraint information besides the multiple conflicting constraint information mentioned above, the target constraint information corresponding to each of the above-mentioned specification objects is determined. Based on the above-mentioned normative objects, the target constraint information corresponding to each of the above-mentioned normative objects, and the above-mentioned normative relationships, the normative constraint path is determined.

[0012] In one possible implementation, the above-mentioned specification objects, constraints, expected execution behaviors, and specification relationships in the specification constraint paths are compared with the code objects, execution conditions, execution behaviors, and code relationships in the candidate code behavior paths to obtain multiple consistency comparison results, including: Determine the target verification dimension corresponding to the above-mentioned specification constraint path. The target verification dimension is one of the following: interface contract dimension, data model dimension, service rule dimension, architecture constraint dimension, security rule dimension, and coding specification dimension. Based on the above target verification dimensions, a target consistency verification strategy corresponding to the above target verification dimensions is determined from a number of preset consistency verification strategies; Based on the above-mentioned target consistency verification strategy, the consistency comparison of the above-mentioned specification constraint path and the above-mentioned candidate code behavior path is performed to obtain multiple consistency comparison results.

[0013] In one possible implementation, the above method also includes: Obtain the admission criteria corresponding to the target code generation tool that generated the above-mentioned code to be evaluated; Obtain the compliance level and / or defect assessment data corresponding to the code to be evaluated from the above compliance assessment report; Based on the above-mentioned compliance level and / or the above-mentioned defect assessment data, and matched with the above-mentioned admission criteria, the admission criteria result corresponding to the above-mentioned target code generation tool is obtained.

[0014] Secondly, embodiments of this application provide a code evaluation apparatus, including: The first acquisition module is used to acquire the target code features corresponding to the code to be evaluated, and to acquire the target specification rule information. The target specification rule information is used to characterize the specification requirements that the code to be evaluated should meet. The verification module is used to perform multi-dimensional consistency verification on the target code features based on the target specification rules information mentioned above, and obtain the verification results corresponding to multiple verification dimensions respectively. The evaluation module is used to evaluate the compliance of the code to be evaluated based on the verification results corresponding to the above multiple verification dimensions, and to obtain the standard evaluation result corresponding to the code to be evaluated. The standard evaluation result includes the standard compliance level and / or defect repair information. The generation module is used to generate a compliance assessment report for the code to be assessed based on the above-mentioned standard assessment results.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps provided in the first aspect of embodiments of this application.

[0016] Fourthly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and executing the method steps provided in the first aspect of embodiments of this application by a processor.

[0017] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to execute the method provided in the first aspect of embodiments of this application.

[0018] This application embodiment obtains the target code features corresponding to the code to be evaluated, as well as the target specification rule information; then, based on the target specification rule information, it performs multi-dimensional consistency verification on the target code features, obtaining verification results corresponding to multiple verification dimensions; and based on the verification results corresponding to the multiple verification dimensions, it performs specification compliance assessment on the code to be evaluated, obtaining the specification assessment result corresponding to the code to be evaluated, which includes specification compliance level and / or defect repair information; and then, based on the specification assessment result, it generates a specification compliance assessment report corresponding to the code to be evaluated. Therefore, it can use the target specification rule information as a unified assessment basis to automatically verify the specification compliance of the code to be evaluated from multiple verification dimensions, and further convert the verification results of different dimensions into specification compliance level and / or defect repair information, thereby achieving a quantitative representation of the specification compliance degree of the code to be evaluated and providing guidance for locating and repairing specification inconsistencies; at the same time, by generating a specification compliance assessment report, it can uniformly record the assessment process and assessment results, facilitating subsequent code quality comparison, problem tracing, and specification governance, improving the objectivity, reliability, and processing efficiency of code assessment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 An exemplary system architecture diagram of a code evaluation method provided in this application embodiment; Figure 2 A flowchart illustrating a code evaluation method provided in an embodiment of this application; Figure 3 A flowchart illustrating a method for obtaining target code features provided in an embodiment of this application; Figure 4 A flowchart illustrating a method for obtaining target specification rule information provided in an embodiment of this application; Figure 5 A flowchart illustrating a method for determining verification results provided in an embodiment of this application; Figure 6 A flowchart illustrating a method for determining standardized evaluation results provided in an embodiment of this application; Figure 7 A flowchart illustrating another method for determining the standard evaluation results provided in this application embodiment; Figure 8This is a schematic diagram of the structure of a code evaluation device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0022] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims. Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the association relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0023] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0024] As artificial intelligence technology is increasingly applied in software development, automatically generating code using AI models or development tools has become a common development method. To ensure the generated code meets pre-defined development requirements, it is typically necessary to evaluate it.

[0025] In related technologies, code evaluation methods mostly rely on manual review, static code inspection, or functional testing. Among them, manual review is greatly affected by the experience of the reviewers, static code inspection mainly focuses on syntax, coding format, etc., and functional testing mainly focuses on whether the code can achieve the expected function. However, the ability to quantify the evaluation results and locate defects is limited, which leads to a long code evaluation process and poor stability of the evaluation results, resulting in low efficiency and reliability of code evaluation.

[0026] Therefore, this application provides a code evaluation method to solve the technical problems of low efficiency and reliability in code evaluation.

[0027] Please see Figure 1 , Figure 1 An exemplary system architecture diagram of a code evaluation method provided in this application embodiment.

[0028] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0029] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to tablet computers, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0030] In this embodiment, terminal 101 first sends the code to be evaluated to server 103 via network 102. Server 103 obtains the target code features corresponding to the code to be evaluated, as well as the target specification rule information. Then, based on the target specification rule information, it performs multi-dimensional consistency verification on the target code features to obtain verification results corresponding to multiple verification dimensions. Furthermore, based on the verification results corresponding to the multiple verification dimensions, it performs specification compliance evaluation on the code to be evaluated to obtain the specification evaluation result corresponding to the code to be evaluated. The specification evaluation result includes specification compliance level and / or defect repair information. Based on the specification evaluation result, it generates a specification compliance evaluation report corresponding to the code to be evaluated.

[0031] Server 103 can be a server that provides various services. It should be noted that server 103 can be hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0032] Alternatively, the system architecture may not include server 103. In other words, server 103 can be an optional device in this application embodiment. That is, the method provided in this application embodiment can be applied to a system structure that only includes terminal 101. That is, terminal 101 obtains the target code features corresponding to the code to be evaluated, and obtains the target specification rule information; then, based on the target specification rule information, it performs multi-dimensional consistency verification on the target code features to obtain verification results corresponding to multiple verification dimensions; and based on the verification results corresponding to the multiple verification dimensions, it performs specification compliance evaluation on the code to be evaluated to obtain the specification evaluation result corresponding to the code to be evaluated. The specification evaluation result includes specification compliance level and / or defect repair information; and then, based on the specification evaluation result, it generates a specification compliance evaluation report corresponding to the code to be evaluated.

[0033] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating a code evaluation method provided in an embodiment of this application. Figure 2 As shown, code evaluation methods can include at least: S202: Obtain the target code characteristics corresponding to the code to be evaluated, and obtain the target specification rule information.

[0035] The code to be evaluated can be code data that requires assessment of its compliance with specifications. This code can be automatically generated by artificial intelligence models, AI code generation tools, or AI coding agents according to preset development specifications, or it can be code obtained by these tools after completing or refactoring existing code. The code to be evaluated can include one or more code files, and can also include classes, functions, interfaces, code modules, or other code units within those files.

[0036] Understandably, target code features can be characteristic information extracted from the code to be evaluated, capable of characterizing the structure, data, and execution logic of the code. Target code features can include at least one of the following: interface features, data structure features, logical relationship features, code syntax structure, function call relationships, module dependencies, parameter definitions, return value definitions, data types, conditional relationships, etc. For example, for an interface in the code to be evaluated, the interface name, request parameters, parameter types, return parameters, and interface call relationships can be extracted as corresponding target code features.

[0037] It should be noted that the target specification rule information can be rule information used for automatic verification of the code to be evaluated. The aforementioned target specification rule information can be generated based on the Spec-Driven Development (SDD) specification, where the SDD specification can include interface contracts, data models, service rules, architectural constraints, security rules, and coding standards. Accordingly, the target specification rule information can include at least one of the following: interface contract rules, data model rules, service rules, architectural constraint rules, security rules, and coding standard rules. For example, interface contract rules can specify the input parameters, output parameters, and their data types of the interface; service rules can specify the operations allowed or prohibited under different service states; architectural constraint rules can specify the allowed calling relationships between different code modules.

[0038] In one implementation, within the SDD (Software-Defined Code) scenario, the code to be evaluated, generated based on the SDD specification, can be acquired first. Redundant code within this code is cleaned, and an abstract syntax tree (AST) is parsed. Based on the parsing results, code features such as interfaces, data structures, and logical relationships are extracted to obtain the target code features. Conversely, pre-configured structured SDD specification data can be acquired. This data undergoes redundancy cleansing, rule conflict resolution, and structured parsing to extract specification rules suitable for code verification. A specification rule base is then constructed based on these extracted rules, and target specification rule information is retrieved from this base. Thus, target code features describing the actual code implementation can be obtained from the code side, and target specification rule information describing the SDD specification requirements can be obtained from the specification side, providing a data foundation for subsequent consistency verification of the specification compliance of the code to be evaluated.

[0039] S204: Based on the above target specification rules information, perform multi-dimensional consistency verification on the above target code features to obtain the verification results corresponding to multiple verification dimensions.

[0040] The multiple verification dimensions can be multiple evaluation directions used when verifying the consistency between the code to be evaluated and the target specification rules information, based on different types of development requirements. These multiple verification dimensions can include at least two of the following: syntax structure dimension, semantic logic dimension, interface contract dimension, data model dimension, service rule dimension, architectural constraint dimension, security rule dimension, and coding standard dimension.

[0041] For example, at the interface contract level, it is possible to verify whether the actual interface name, number of parameters, parameter types, and return values ​​defined in the code to be evaluated conform to the corresponding interface specifications; at the data model level, it is possible to verify whether the data fields, field types, and data structures in the code conform to the specification requirements; at the service rules level, it is possible to verify whether the condition judgments, service state transitions, and corresponding processing logic in the code conform to the pre-defined service rules; at the architectural constraints level, it is possible to verify whether the dependency relationships or calling relationships between different modules conform to the corresponding architectural specifications; at the security rules level, it is possible to verify whether the code sets up the corresponding permission verification, security verification, or exception handling logic as required by the specifications.

[0042] Understandably, the verification result can be the data obtained by comparing the characteristics of the target code with the corresponding target specification rules for a single verification dimension. The verification result can be used to characterize whether the code to be evaluated conforms to or does not meet the specification requirements under that verification dimension, and may also include information such as inconsistent code content, corresponding specification rules, and differences.

[0043] In one implementation, target code features can be matched against corresponding specification rules according to the rule type corresponding to the target specification rule information. For example, interface-related code features can be compared with interface contract rules, data structure-related code features with data model rules, code logic relationships with service rules, and module call relationships with architectural constraint rules. For each verification dimension, the consistency between the actual code implementation and the specification requirements can be determined, thus obtaining verification results for multiple verification dimensions. For rules that can be directly compared in a structured way, automatic comparison can be performed based on the abstract syntax tree parsing results and rule data; for rules involving code semantics or service logic, consistency judgment can be made by combining the logical relationships of the code and the semantic constraints in the specification, thereby improving the coverage of different types of specification requirements.

[0044] S206: Based on the verification results corresponding to the above multiple verification dimensions, the code to be evaluated is evaluated for compliance with the standards, and the standard evaluation result corresponding to the code to be evaluated is obtained. The standard evaluation result includes the standard compliance level and / or defect repair information.

[0045] Optionally, the evaluation results may also include information such as a compliance score to reflect the compliance status of the code being evaluated.

[0046] The compliance level can be a graded information based on the degree of compliance of the code under evaluation. It is used to represent the overall compliance of the code with the target specification rules in a discrete grade manner. For example, the compliance sub-scores can be determined first based on the verification results of multiple verification dimensions, and then weighted according to the evaluation weights corresponding to different verification dimensions to obtain a compliance score in the range of 0 to 100. The compliance score can then be divided into different compliance levels according to a preset scoring interval. Different compliance levels can represent high, average, or low compliance, or can be represented by grades such as A, B, C, and D.

[0047] It is understandable that defect repair information can be generated to assist in defect handling for code content in the code to be evaluated that does not meet the target specification rules. Defect repair information may include at least one of the following: defect location, defect type, corresponding specification basis information, and repair suggestions. For example, when the verification result shows that an interface in the code to be evaluated lacks the parameter verification logic specified in the specification, the code file and code location of the interface can be located, the corresponding parameter verification specification can be identified as the specification basis information, and repair suggestions to add corresponding parameter verification processing can be generated.

[0048] In one implementation, a specification compliance sub-score can be calculated for each of the multiple verification dimensions based on the verification results. This sub-scores are then weighted using a pre-defined multi-dimensional weighted quantification scoring model to obtain the specification compliance score for the code to be evaluated. The specification compliance level is then determined based on the correspondence between the score and pre-defined level classification conditions. Simultaneously, for any content in the verification results indicating specification inconsistencies, the corresponding code location can be determined based on the verification result. This information is then linked to the target specification rule information that led to the verification result, thereby determining the specification basis information and remediation suggestions for the defect, and generating defect remediation information. Finally, the obtained specification compliance level and / or defect remediation information can be used as the specification evaluation result for the code to be evaluated.

[0049] S208: Based on the above standard evaluation results, generate a standard compliance evaluation report for the above-mentioned code to be evaluated.

[0050] The compliance assessment report refers to a structured data set or report file that records and displays the compliance status of the code to be assessed. It can be used to reflect the overall assessment status of the code relative to the target specification rules. The compliance assessment report may include at least one of the following: compliance score, compliance level, verification status for each verification dimension, defect information, defect location, corresponding specification basis information, and remediation suggestions.

[0051] For example, for a code to be evaluated, the compliance assessment report can record the overall compliance level of the code and show the verification status of verification dimensions such as interface contract, data model, service rules, architectural constraints and security rules. For code content with inconsistent specifications, the report can further record the corresponding code location, the violated specification rules and corresponding remediation suggestions, so that the code evaluation results can be correlated with the specific specification information.

[0052] In one implementation, the compliance assessment results can be categorized and organized. Compliance level and score information are organized into level assessment data, while defect location, defect type, compliance basis information, and remediation suggestions are organized into defect assessment data. Furthermore, the level assessment data and / or defect assessment data are organized according to a pre-defined report structure to generate a compliance assessment report for the code to be assessed. The generated compliance assessment report can be further used to compare compliance capabilities among different AI code generation tools, determine the admission criteria for AI code generation tools, and optimize subsequent iterations of the code generation model, thus providing a unified data basis for the standardized evaluation of AI-generated code.

[0053] In this embodiment, the target code features corresponding to the code to be evaluated and the target specification rule information are obtained. Then, based on the target specification rule information, multi-dimensional consistency verification is performed on the target code features to obtain verification results corresponding to multiple verification dimensions. Based on the verification results corresponding to the multiple verification dimensions, the code to be evaluated is assessed for compliance with specifications to obtain the specification assessment results corresponding to the code to be evaluated. These specification assessment results include specification compliance level and / or defect repair information. Finally, based on the specification assessment results, a specification compliance assessment report corresponding to the code to be evaluated is generated. Therefore, the target specification rule information can be used as a unified evaluation basis to automatically verify the specification compliance of the code to be evaluated from multiple verification dimensions. The verification results from different dimensions are further transformed into specification compliance level and / or defect repair information, thereby achieving a quantitative representation of the specification compliance degree of the code to be evaluated and providing guidance for locating and repairing specification inconsistencies. Simultaneously, by generating a specification compliance assessment report, the evaluation process and results can be uniformly recorded, facilitating subsequent code quality comparison, problem tracing, and specification governance, thus improving the objectivity, reliability, and processing efficiency of code evaluation.

[0054] In one embodiment, see Figure 3 In S202, obtaining the target code features corresponding to the code to be evaluated may include the following steps: S302: Obtain the code to be evaluated.

[0055] Optionally, the code to be evaluated can be code generated by an artificial intelligence model or code generation tool according to preset development requirements, or it can be code that has been completed, modified, or restructured by an artificial intelligence model.

[0056] In some embodiments, the corresponding code to be evaluated can be obtained based on the project identifier, code repository identifier, or code version information corresponding to the task to be evaluated, so that subsequent code analysis can be performed on the current evaluation object.

[0057] S304: Perform code cleaning and abstract syntax tree parsing on the above-mentioned code to be evaluated to determine multiple code objects in the above-mentioned code to be evaluated and the code association relationships between the multiple code objects.

[0058] In this context, a code object can be a program element identified from the code to be evaluated that can serve as the basic unit for code structure analysis or execution logic analysis. Code objects can include one or more program elements such as classes, functions, methods, interfaces, variables, data fields, conditional branches, loop structures, call statements, return statements, and code modules. For example, for order cancellation-related code, the order cancellation interface, order status variables, permission verification functions, and refund processing functions can be identified as different code objects.

[0059] It is understandable that code relationships refer to the connections between multiple code objects based on code structure or program execution. Code relationships can include at least one of the following: call relationships, control dependencies, data dependencies, data transfer relationships, state transition relationships, module dependencies, and execution order relationships. For example, when a first function calls a second function, it can be determined that there is a call relationship between the first and second functions; when the execution of a conditional branch depends on a certain state variable, it can be determined that there is a control dependency between the conditional branch and the state variable.

[0060] Optionally, code cleaning and abstract syntax tree (AST) parsing can be performed on the code to be evaluated. Code cleaning may include removing redundant content that does not affect program logic analysis, standardizing code format, or filtering auxiliary content that cannot participate in specification verification. Subsequently, a parser matching the programming language of the code to be evaluated can be used to convert the code into an AST, and multiple code objects can be identified based on the node types, hierarchical relationships, and reference relationships in the AST. Further, the code association relationships between multiple code objects can be determined based on function call information, variable reference information, conditional branch relationships, data read / write relationships, and module dependencies. Thus, the code to be evaluated can be converted from text form into structured code data that reflects the program structure and object relationships.

[0061] S306: Based on multiple code objects and the code association relationships described above, determine the code behavior path and the execution conditions and execution behaviors corresponding to the code behavior path. The code behavior path is used to characterize the association sequence formed by at least two code objects according to the code execution relationship.

[0062] The code behavior path can be used to characterize the code execution process or processing logic. A code behavior path can include multiple code objects arranged according to a predetermined execution relationship, as well as the code associations between adjacent code objects. For example, a code behavior path can be represented as: order cancellation interface—order status judgment—permission verification—refund processing function. The code behavior path can be used to characterize the processing procedures that the code to be evaluated may execute in the corresponding service scenario.

[0063] Understandably, execution conditions can be conditions that one or more code objects in a code behavior path must meet when executed. Execution conditions can include variable value conditions, state conditions, permission conditions, parameter conditions, branch judgment conditions, and other conditions that can affect the selection of the code execution path. For example, if there is logic in the code that "calls the refund function when the order status is paid," then "order status is paid" can be used as the execution condition for the corresponding code behavior path.

[0064] In addition, execution behavior can be the actual program operation executed in the code behavior path when the corresponding execution conditions are met. Execution behavior can include one or more behaviors such as function calls, interface calls, data reading, data writing, status updates, permission checks, result returns, exception handling, and execution termination. For example, calling the refund processing function after the order payment condition is met can be identified as the execution behavior of the corresponding code behavior path.

[0065] Specifically, an interface, function, or other code object that can trigger code execution can be used as the starting object. Subsequent code objects are determined sequentially according to call relationships, control dependencies, data dependencies, or execution order, thus forming a code behavior path. During the formation of this path, conditional statements, status checks, or parameter checks can be extracted and identified as corresponding execution conditions. Simultaneously, the functions actually called, data processing operations performed, status update operations, or return operations under the corresponding execution conditions can be extracted and identified as corresponding execution behaviors. For example, when the code contains logic for receiving an order cancellation request, checking the order status, and calling the refund interface when the order has been paid, a corresponding code behavior path can be formed, with "order status is paid" identified as the execution condition and "calling the refund interface" identified as the execution behavior.

[0066] S308: Based on multiple code objects, code relationships, code behavior paths, execution conditions, and execution behaviors, feature extraction is performed to obtain target code features.

[0067] Optionally, the object types, object attributes, association types between objects, object arrangement relationships in the code behavior path, and corresponding execution conditions and behaviors of multiple code objects can be organized according to a preset feature structure to form target code features that describe the actual implementation of the code to be evaluated. Target code features may include one or more features such as interface features, data structure features, logical relationship features, call relationship features, and behavioral features. This allows for a unified representation of the static structural information and execution logic information of the code to be evaluated, enabling subsequent multi-dimensional consistency verification of the target code features based on target specification rules.

[0068] In this embodiment, by performing code cleaning and abstract syntax tree parsing on the code to be evaluated, multiple code objects and their code relationships can be identified from the code. Furthermore, based on these code objects and relationships, code behavior paths, execution conditions, and execution behaviors are determined, thereby converting the original code text into structured information that simultaneously represents the code structure and execution logic. On this basis, by extracting features from the above information, target code features describing the actual implementation of the code to be evaluated can be obtained. This helps reduce the interference of redundant code and irrelevant information on the subsequent evaluation process, and enables subsequent consistency checks to be performed not only based on the static structure of the code but also by analyzing the actual execution relationships and behavioral logic of the code, thereby improving the completeness of code feature representation and the accuracy and reliability of subsequent specification consistency checks.

[0069] In one embodiment, see Figure 4 In S202, obtaining the target specification rule information may include the following steps: S402: Obtain preset specification data.

[0070] The preset specification data can be data used to constrain the code development process or the code implementation results. Preset specification data can include one or more types of data such as interface specifications, data model specifications, service rules, architectural constraints, security rules, and coding standards.

[0071] Optionally, the preset specification data can be stored in structured files, semi-structured files, database records, or other data formats that can characterize code development requirements. For example, the preset specification data can record the input parameters, output parameters, and parameter types of a certain interface, and can also record the processing operations that are allowed or prohibited under a specific service state.

[0072] Specifically, preset specification data can be obtained from preset storage locations. These preset storage locations may include local configuration files, specification management platforms, databases, or other data sources used to store specification data. In some embodiments, preset specification data corresponding to the code to be evaluated can also be obtained from multiple specification data sets based on the project, code module, or service scenario to which the code to be evaluated belongs. For example, if the code to be evaluated belongs to an order processing module, preset specification data related to the order interface, order data model, and order service rules can be obtained.

[0073] S404: Perform structured parsing on the above-mentioned preset specification data to determine multiple specification objects and the constraint information corresponding to each of the above-mentioned specification objects. The constraint information includes at least one of constraint conditions and expected execution behavior.

[0074] The specification object can be a data entity identified from predefined specification data. Specification objects can include interfaces, data fields, data models, service states, service operations, code modules, services, functions, and other objects that can be constrained by specifications. For example, when a piece of specification data is used to limit the calling conditions of an order cancellation interface, that order cancellation interface can be identified as a specification object; when a piece of specification data is used to limit the transformation relationship between order states, the corresponding order state or state transformation operation can be identified as a specification object.

[0075] It should be noted that constraint information can be used to describe the restrictions or execution requirements that a specification object should meet. Constraint information can include at least one of the constraint condition and the expected execution behavior, and can also include information such as the value range of the constraint object, invocation requirements, dependency requirements, and access requirements. For example, for an order cancellation interface, its constraint information can include the constraint condition "order status is paid" and the expected execution behavior "entering the refund processing flow".

[0076] Constraints can be conditions that must be met for a specification to take effect or for a corresponding processing action to be allowed to execute. Constraints can include data value conditions, service status conditions, identity and permission conditions, call order conditions, and module dependency conditions. For example, constraints could be "order status is paid," "the calling user has order operation permissions," or "the target module can only be called by the specified service." Expected execution behavior can be the processing action that should or should not be executed according to the specification requirements, provided the corresponding constraints are met. Expected execution behavior can include calling the target interface, executing the target function, updating target data, performing permission checks, returning a preset result, or prohibiting the execution of the target operation. For example, if the order status is paid, the expected execution behavior could be calling the refund processing interface instead of directly executing the order cancellation operation.

[0077] Specifically, pre-defined specification data can be parsed based on its data structure, field identifiers, object identifiers, and rule descriptions. For example, information such as interface name, input parameters, and output parameters can be extracted from interface definitions to identify the corresponding interface as a specification object; information such as data fields and field types can be extracted from data model definitions to identify the corresponding data model or data field as a specification object; and constrained interfaces, service states, code modules, or functions can be identified from service rules, architectural constraints, or security rules and identified as specification objects. Furthermore, the condition and behavior descriptions corresponding to each specification object can be parsed to identify constraints such as those limiting the specification's effectiveness, data value range, or calling conditions, and to identify expected execution behaviors such as interface calls, function executions, data updates, and state transitions that the specification requires or prohibits, thereby obtaining the constraint information corresponding to each specification object.

[0078] S406: Based on the above-mentioned standard objects and the constraint information corresponding to each of the above-mentioned standard objects, determine the standard association relationship between the above-mentioned standard objects.

[0079] In this context, normative relationships can be associations formed between different normative objects due to service processing, data transfer, call dependencies, or constraint transitivity. Normative relationships can include at least one of the following: call relationships, dependency relationships, data relationships, pre- and post-requirement relationships, state transition relationships, and constraint transitivity relationships. For example, for the norm that "the order cancellation interface should call the refund interface when the order has been paid," normative relationships can be established between the order cancellation interface, the order status, and the refund interface.

[0080] Specifically, corresponding associations can be established based on the calling relationships, dependencies, pre- and post-requirements, state transition relationships, or common constraint information between specification objects. For example, when the expected execution behavior corresponding to the first specification object points to the second specification object, it can be determined that there is an execution association relationship between the first and second specification objects; when the constraint condition of one specification object depends on the state or data of another specification object, it can be determined that there is a conditional dependency relationship between the two specification objects. Thus, originally independent specification objects can be organized into a specification relationship structure with associations.

[0081] S408: Based on multiple of the above-mentioned standard objects, the constraint information corresponding to each of the above-mentioned standard objects, and the above-mentioned standard association relationships, determine the standard constraint path. The above-mentioned standard constraint path is used to characterize the constraint relationship formed by at least two standard objects according to the above-mentioned standard association relationships.

[0082] The specification constraint path can be used to describe the constraint relationships between multiple specification objects. A specification constraint path can include multiple specification objects, as well as the specification associations between adjacent specification objects, and can further associate the constraints and expected execution behaviors corresponding to each specification object.

[0083] Understandably, by defining constraint paths, scattered specification requirements can be organized into an associative structure that reflects complete constraint logic. For example, specification constraint paths can be formed based on the relationships between service interfaces, constraints, and target processing behaviors, enabling subsequent determination of whether the actual code implementation conforms to the corresponding specification requirements.

[0084] In one embodiment, determining the specification constraint path based on multiple specification objects, the constraint information corresponding to each specification object, and the specification association may include the following steps: identifying multiple conflicting constraint information that acts on the same specification object and conflicts with it from the constraint information corresponding to each specification object; resolving the conflicts of the multiple conflicting constraint information based on at least one of specification priority, version information, and scope of application corresponding to each of the multiple conflicting constraint information to determine the effective constraint information; determining the target constraint information corresponding to each specification object based on the effective constraint information and other constraint information besides the multiple conflicting constraint information; and determining the specification constraint path based on the multiple specification objects, the target constraint information corresponding to each specification object, and the specification association.

[0085] Among them, conflict constraint information can be multiple constraints acting on the same specification object, with inconsistent corresponding constraints and / or expected execution behaviors, and can be used to identify specification requirements that cannot be directly effective simultaneously. Specification priority can be information representing the execution priority order of different constraint information, and can be used to determine the preferred constraint information when multiple constraint information conflicts. Scope of application can be information that limits the effective object or effective scenario of the constraint information, including projects, modules, interface types, service scenarios, etc., and can be used to determine whether the corresponding constraint information is applicable to the current specification object. Effective constraint information can be the actually effective constraint information determined after conflict resolution of multiple conflicting constraint information, and can be used to participate in the construction of subsequent specification constraint paths. Target constraint information can be the constraint information actually adopted by each specification object after conflict resolution, including effective constraint information and other constraint information that has not conflicted. Version information can be information representing the specification version to which the constraint information belongs, and can be used to determine the currently effective constraint information when there are conflicts between constraint information of different versions.

[0086] In one specific embodiment, multiple constraint information can first be categorized according to the specification object, and the constraint conditions and expected execution behaviors of multiple constraint information acting on the same specification object can be compared. When multiple constraint information corresponds to different execution requirements under the same or overlapping conditions, they can be identified as conflicting constraint information. Further, conflict resolution can be performed based on at least one of the specification priority, version information, and scope of application of the multiple conflicting constraint information. For example, constraint information with higher priority can be used first, or constraint information corresponding to the current effective version and the current application scenario can be used to determine the effective constraint information. Subsequently, the effective constraint information corresponding to the conflicting specification object, as well as other constraint information that has not conflicted, can be determined as the target constraint information corresponding to each specification object. Finally, multiple specification objects and their corresponding target constraint information can be organized according to the specification association relationship between multiple specification objects to form a specification constraint path, so that the subsequent code consistency verification based on the specification constraint path uses the effective specification requirements after conflict resolution.

[0087] This application's embodiments identify multiple conflicting constraint information acting on the same specification object, and resolve these conflicts based on at least one of specification priority, version information, and scope of application to determine the actually effective constraint information. Subsequently, by combining other non-conflicting constraint information, the target constraint information corresponding to each specification object is determined, and a specification constraint path is constructed based on the target constraint information and specification relationships. This reduces the likelihood of contradictory specification requirements simultaneously participating in subsequent code consistency checks, ensuring that the specification constraint path reflects the valid specification requirements in the current scenario. This improves the consistency and usability of specification rule information and enhances the accuracy and reliability of subsequent code specification consistency checks.

[0088] S410: Based on the above-mentioned normative constraint path, generate target normative rule information.

[0089] Specifically, the specification objects, specification relationships, constraints, and expected execution behaviors in the specification constraint path can be organized according to a preset data structure to form target specification rule information that can be used for subsequent consistency verification. Different specification constraint paths can generate corresponding specification rules separately, or they can be classified according to categories such as interface, data, service logic, architecture, security, or coding standards. Thus, the preset specification data can be converted into target specification rule information that can be compared with the characteristics of the target code corresponding to the code to be evaluated, thereby providing specification-side data support for subsequent multi-dimensional consistency verification.

[0090] In this embodiment, by performing structured parsing on preset specification data, multiple specification objects and their corresponding constraint information are determined, and the specification relationships and constraint paths between these objects are further identified. Furthermore, in cases of constraint conflicts, conflict resolution is performed on the conflicting constraint information to determine the valid constraint content for each specification object. Then, target specification rule information is generated based on the processed specification constraint paths. This transforms scattered and potentially conflicting specification data into structured, clearly defined, and applicable specification rule information for the current scenario. This reduces the interference of invalid or contradictory specification requirements on subsequent verification processes and provides a unified and reliable specification-side data foundation for subsequent multi-dimensional consistency verification based on target specification rule information and target code features, thereby improving the accuracy and reliability of specification parsing and code evaluation.

[0091] In one embodiment, see Figure 5 In S204 above, the multi-dimensional consistency verification of the target code features based on the target specification rule information to obtain verification results corresponding to multiple verification dimensions may include the following steps: S502: Based on the normative constraint paths in the above target normative rule information, determine the candidate code behavior paths corresponding to the above normative constraint paths from the above code behavior paths.

[0092] Among them, the candidate code behavior path can be a code behavior path that corresponds to the current specification constraint path in terms of the object of action, the content of processing, or the execution logic, selected from one or more code behavior paths corresponding to the code to be evaluated.

[0093] Specifically, we can first identify the normative objects in the normative constraint path, and then search for the corresponding code objects in multiple code behavior paths based on information such as the object type, object name, module, service semantics, or associated objects of the normative objects. Furthermore, we can filter the relevant code behavior paths based on the code association, execution conditions, and execution behavior associated with the code object, and determine the code behavior paths that correspond to the normative constraint paths in terms of objects and processing logic as candidate code behavior paths.

[0094] S504: Compare the consistency of the specification objects, constraints, expected execution behaviors, and specification relationships in the above specification constraint paths with the code objects, execution conditions, execution behaviors, and code relationships in the above candidate code behavior paths to obtain multiple consistency comparison results.

[0095] Among them, each consistency comparison result can be the result information obtained by comparing the specification content in the specification constraint path with the actual code implementation content in the candidate code behavior path, which can be used to characterize the consistency between the two in the corresponding comparison item.

[0096] Optionally, the consistency comparison result may include states such as consistent, inconsistent, or partially consistent, or it may include specific differences. For example, it may compare whether the specification object corresponds to the code object, whether the constraints and execution conditions are consistent, whether the expected execution behavior is consistent with the actual execution behavior, and whether the specification association relationship is consistent with the code association relationship, and obtain a corresponding consistency comparison result for each comparison item. In some embodiments, the consistency comparison result may also include information such as the location of the difference, the type of the difference, or the degree of the difference.

[0097] Optionally, the above-mentioned consistency comparison of the specification objects, constraints, expected execution behaviors, and specification relationships in the specification constraint path with the code objects, execution conditions, execution behaviors, and code relationships in the candidate code behavior path, to obtain multiple consistency comparison results, may include the following steps: determining the target verification dimension corresponding to the specification constraint path, wherein the target verification dimension is one of the following: interface contract dimension, data model dimension, service rule dimension, architecture constraint dimension, security rule dimension, and coding specification dimension; based on the target verification dimension, determining the target consistency verification strategy corresponding to the target verification dimension from a set of preset consistency verification strategies; and based on the target consistency verification strategy, performing a consistency comparison between the specification constraint path and the candidate code behavior path to obtain multiple consistency comparison results.

[0098] The target validation dimension can be the validation category to which the current specification constraint path belongs, and can be used to determine the corresponding consistency comparison method. The target validation dimension can include one of the following: interface contract dimension, data model dimension, service rule dimension, architectural constraint dimension, security rule dimension, and coding specification dimension. For example, when the specification constraint path is mainly used to limit interface parameters and return values, the interface contract dimension can be determined as the target validation dimension.

[0099] Optionally, the consistency verification strategy can be a pre-set comparison rule or processing method for different verification dimensions, which can be used to compare the consistency between the specification constraint path and the candidate code behavior path. Different consistency verification strategies can correspond to different comparison contents. For example, the interface contract dimension can compare interface names, parameters, data types, and return values; the service rule dimension can compare constraints, execution behaviors, and processing order; and the architecture constraint dimension can compare the call relationships or dependencies between modules. The target consistency verification strategy can be a consistency verification strategy determined from multiple pre-set consistency verification strategies that corresponds to the current target verification dimension, and can be used to compare the consistency between the current specification constraint path and the candidate code behavior path.

[0100] In one implementation, the target validation dimension for a specification constraint path can be determined based on the specification objects, constraint information, or specification type contained within it. For example, when a specification constraint path describes data field and data type requirements, the target validation dimension can be determined as the data model dimension; when it describes permission validation requirements, the target validation dimension can be determined as the security rule dimension. Furthermore, a pre-established correspondence between different validation dimensions and consistency validation strategies can be established, and a target consistency validation strategy can be selected from multiple consistency validation strategies based on the target validation dimension. Subsequently, the corresponding content in the specification constraint path and the candidate code behavior path can be compared according to the target consistency validation strategy. For example, under the service rule dimension, the consistency between constraint conditions and execution conditions, expected execution behavior and actual execution behavior, and the corresponding before-and-after relationships can be compared to obtain multiple consistency comparison results. Thus, appropriate validation methods can be adopted for different types of specifications, improving the adaptability and accuracy of multi-dimensional consistency validation.

[0101] In this embodiment, the target verification dimension corresponding to the specification constraint path is determined, and a corresponding target consistency verification strategy is determined from multiple consistency verification strategies based on the aforementioned target verification dimension. Then, the specification constraint path and candidate code behavior path are compared for consistency according to the aforementioned target consistency verification strategy. Therefore, a verification method adapted to different types of specification requirements can be adopted, avoiding the use of a single verification method to uniformly process different types of specifications. This allows specifications of different dimensions, such as interface contracts, data models, service rules, architectural constraints, and security rules, to be verified specifically, thereby improving the adaptability, accuracy, and reliability of multi-dimensional consistency verification.

[0102] S506: Based on the above consistency comparison results, classify them according to the corresponding specification type to obtain the verification results corresponding to multiple verification dimensions.

[0103] The specification type can be a type of information obtained by classifying target specification rule information according to the code attributes or development requirements constrained by the specification content. Specification types can include at least one of the following: interface contract type, data model type, service rule type, architectural constraint type, security rule type, and coding specification type. For example, a specification used to constrain interface input and return parameters can be identified as an interface contract type; a specification used to limit service state transitions or service processing logic can be identified as a service rule type; and a specification used to limit the calling relationships between modules can be identified as an architectural constraint type.

[0104] It is understandable that the verification results corresponding to multiple verification dimensions can serve as data reflecting the compliance of the code under evaluation with specifications under different verification dimensions. Multiple verification dimensions can correspond to different specification types, such as at least two of the following: interface contract dimension, data model dimension, service rule dimension, architectural constraint dimension, security rule dimension, and coding standard dimension. The verification results for each verification dimension can include information such as consistent items, inconsistent items, the number of consistent items, and the content of the differences. For example, the verification results corresponding to the interface contract dimension can include the consistency comparison results corresponding to multiple interface rules, and the verification results corresponding to the service rule dimension can include the consistency comparison results corresponding to multiple service rules.

[0105] Specifically, the specification type to which each specification constraint path belongs can be determined, and the consistency comparison results generated by specification constraint paths corresponding to the same specification type can be classified into the same verification dimension.

[0106] For example, consistency comparison results corresponding to interface contract types can be categorized into the interface contract validation dimension, consistency comparison results corresponding to data model types into the data model validation dimension, consistency comparison results corresponding to service rule types into the service rule validation dimension, and consistency comparison results corresponding to architectural constraint types into the architectural constraint validation dimension. For each validation dimension, multiple consistency comparison results under that dimension can be summarized to obtain the validation result corresponding to that dimension. The above validation results can record the consistency status and existing differences of each specification rule under that validation dimension, and serve as the data basis for subsequently determining specification compliance scores, specification compliance levels, or defect repair information.

[0107] In this embodiment, by first filtering the corresponding candidate code behavior paths according to the specification constraint paths, and then performing consistency comparisons from multiple aspects such as objects, conditions, behaviors and relationships, and classifying the comparison results according to the specification type, the actual compliance of the code to be evaluated with respect to the target specification rule information can be reflected from multiple verification dimensions, thereby improving the completeness and accuracy of specification consistency verification.

[0108] In one embodiment, see Figure 6 In S206, the above-mentioned code to be evaluated is evaluated for compliance with specifications based on the verification results corresponding to the above-mentioned multiple verification dimensions, and the specification evaluation result corresponding to the code to be evaluated is obtained. This may include the following steps: S602: Based on the verification results corresponding to the above multiple verification dimensions, determine the specification compliance sub-scores corresponding to the above multiple verification dimensions.

[0109] The specification compliance sub-score is a quantitative result determined for the verification result corresponding to a verification dimension. It can be used to characterize the degree to which the code under evaluation conforms to the relevant specification rules under the corresponding verification dimension. The specification compliance sub-score can be represented by a percentage scale, a ten-point scale, or a numerical value between 0 and 1.

[0110] Specifically, for each validation dimension, multiple consistency comparison results corresponding to that dimension can be statistically analyzed, and the specification compliance sub-score for that validation dimension can be determined based on these results. For example, the number of consistency comparison results representing consistency and the total number of consistency comparison results participating in the comparison can be determined, and the specification compliance sub-score for that dimension can be determined based on the ratio between the two. If the consistency comparison results also include partial consistency, corresponding score values ​​can be set for consistent, partially consistent, and inconsistent consistency comparison results, and the specification compliance sub-score can be determined based on the score values ​​corresponding to multiple consistency comparison results. Taking the service rule dimension as an example, if 18 out of the 20 service rules participating in the validation have consistent consistency comparison results, the specification compliance sub-score for the service rule dimension can be determined based on the ratio between the number of service rules that meet the specification requirements and the total number of service rules participating in the validation.

[0111] S604: Based on at least one of the rule priority, rule quantity, and preset violation severity level of the normative rules corresponding to the above-mentioned verification dimensions in the above-mentioned target normative rule information, determine the evaluation weights corresponding to the above-mentioned verification dimensions respectively.

[0112] Rule priority can be used to characterize the importance or execution priority of different specification rules in the code specification constraint process. Different specification rules can be assigned different priority levels based on their specification type, service impact, security requirements, or pre-configured rule levels. For example, specification rules involving permission verification, security control, or interface compatibility can have higher rule priority, while some encoding format specification rules can have relatively lower rule priority.

[0113] Understandably, the number of rules can be the number of specification rules corresponding to a single validation dimension. The number of specification rules corresponding to different validation dimensions can be the same or different. The number of rules can be used to reflect the rule coverage or rule proportion of the corresponding validation dimension in the current target specification rule information.

[0114] It should be noted that the preset violation severity level represents the potential impact of a violation when the code to be evaluated fails to meet the specified rules. The preset violation severity level can be pre-set based on factors such as violation type, scope of impact, service impact, and security risks, or it can be determined based on rule attributes in the target specification rule information. For example, violations that cause interface unavailability, missing permission checks, or service process errors can be assigned a higher violation severity level, while violations that do not conform to naming formats or code formatting standards can be assigned a lower violation severity level.

[0115] Specifically, the standard rules corresponding to each verification dimension can be obtained separately. A priority evaluation value can be determined based on the rule priority of each standard rule under that verification dimension, a rule quantity evaluation value can be determined based on the number of standard rules, and a severity evaluation value can be determined based on the preset violation severity level corresponding to each standard rule under that verification dimension. Further, the initial weights of the corresponding verification dimensions can be determined based on one or more of the above evaluation values, and the initial weights corresponding to multiple verification dimensions can be normalized to obtain the evaluation weights corresponding to each verification dimension. For example, when the standard rules in the security rule dimension have high rule priority and the corresponding preset violation severity level is generally high, the evaluation weight corresponding to the security rule dimension can be increased.

[0116] S606: The compliance score is obtained by weighting multiple compliance scores and their corresponding evaluation weights.

[0117] The evaluation weight can be used to characterize the degree of influence of each verification dimension on the overall evaluation result. The evaluation weight can be determined based on at least one of the rule priority, number of rules, and preset violation severity level of the standard rules under the corresponding verification dimension. For example, verification dimensions with higher rule priority, more rules, or higher violation severity can be assigned a larger evaluation weight to increase the degree of influence of that verification dimension on the overall evaluation result.

[0118] Understandably, compliance scores can be used to quantify the overall compliance level of the code being evaluated. Compliance scores can be represented in a standardized format from 0 to 100, where a higher score indicates a greater degree of consistency between the code being evaluated and the target specification rules.

[0119] Specifically, the specification compliance sub-score for each validation dimension can be multiplied by the evaluation weight corresponding to that validation dimension, and the products of multiple validation dimensions can be summed to obtain the specification compliance score for the code to be evaluated. In one example, if the specification compliance sub-score for the i-th validation dimension is S... i The corresponding evaluation weight is W. i Then it can be based on multiple W i ×S i The summation result determines the compliance score. Therefore, while comprehensively evaluating the results of multiple verification dimensions, different levels of importance of verification dimensions can have varying degrees of impact on the final score.

[0120] S608: Based on the above compliance score, the compliance level is determined to obtain the compliance level.

[0121] The compliance level can be used to represent the overall compliance level of the code being evaluated in a discrete form. The compliance level can be divided according to multiple pre-set scoring intervals. For example, a first scoring interval can correspond to a first compliance level, a second scoring interval to a second compliance level, and different levels can be represented using labels such as A, B, C, D, etc., or using information such as excellent, good, satisfactory, unsatisfactory, etc.

[0122] Specifically, multiple scoring intervals and corresponding compliance levels for each interval can be pre-defined, and a target scoring interval for the compliance score can be determined. The compliance level corresponding to this target scoring interval is then set as the compliance level for the code to be evaluated. For example, 90 points and above can be assigned to Level 1, 80 points and above but below 90 points to Level 2, 60 points and above but below 80 points to Level 3, and below 60 points to Level 4. It should be noted that the above scoring intervals and level classifications are merely examples and can be adjusted according to different projects, code types, or compliance requirements in practical applications.

[0123] S610: Based on the above compliance levels, the standard evaluation results are obtained.

[0124] Specifically, the compliance level can be used as the evaluation result of the overall compliance of the code to be evaluated. The compliance level, the corresponding compliance score, and the compliance sub-scores for each verification dimension can be correlated to form the compliance evaluation result. In some embodiments, the compliance evaluation result can be further correlated with the identification information, code version information, or evaluation time information of the code to be evaluated, so as to generate a compliance evaluation report and use it to compare the compliance of different codes to be evaluated or different code generation tools.

[0125] In this embodiment, a compliance sub-score is determined for each verification dimension based on the verification results corresponding to multiple verification dimensions. An evaluation weight is then determined by combining at least one of the rule priority, number of rules, and preset violation severity level of the corresponding regulatory rules for each verification dimension. Subsequently, a weighted average is performed based on the multiple compliance sub-scores and their corresponding evaluation weights to obtain a compliance score. The compliance level is then further determined based on this compliance score. This allows for the unified transformation of compliance status across multiple verification dimensions into a quantifiable and comparable evaluation result, and enables regulatory requirements of varying importance to influence the overall evaluation result, thereby improving the objectivity, discriminativeness, and reliability of the compliance evaluation.

[0126] In one embodiment, see Figure 7 In S206 above, the above-mentioned code to be evaluated is evaluated for compliance with specifications based on the verification results corresponding to the above-mentioned multiple verification dimensions, and the specification evaluation result corresponding to the code to be evaluated is obtained. This may include the following steps: S702: If at least one of the verification results corresponding to the above-mentioned verification dimensions indicates that there is a specification inconsistency, determine the target verification result with specification inconsistency from the above-mentioned verification results.

[0127] Inconsistency in specification can refer to a discrepancy between the actual code implementation represented by the code to be evaluated and the expected specification requirements represented by the target specification rules. Inconsistency can manifest as the code lacking objects required by the specification, inconsistencies between code execution conditions and specification constraints, inconsistencies between code execution behavior and the expected execution behavior of the specification, or inconsistencies between the relationships between code objects and the specification relationships between specification objects.

[0128] It is understandable that the target validation result can be a validation result selected from the validation results corresponding to multiple validation dimensions. A target validation result can correspond to one validation dimension, or it can correspond to one or more inconsistent items under that validation dimension.

[0129] Specifically, the validation results corresponding to multiple validation dimensions can be traversed to determine whether there are any inconsistent consistency comparison results among them, and the validation result containing one or more inconsistent items can be determined as the target validation result. If there are specification inconsistencies in multiple validation dimensions, multiple target validation results can be determined separately. For example, when there are inconsistencies in both the interface contract dimension and the service rule dimension, the corresponding interface contract target validation result and service rule target validation result can be determined separately.

[0130] S704: Based on the above target verification results, determine the corresponding target specification constraint path and the target code behavior path corresponding to the above target specification constraint path.

[0131] The target specification constraint path can be the specification constraint path corresponding to the target validation result, that is, the specification constraint path corresponding to the specification rule that causes the target validation result to represent a specification inconsistency. For example, when the target validation result represents "paid orders have not entered the refund processing process", the specification constraint path corresponding to "order cancellation interface - order status is paid - refund processing interface" can be determined as the target specification constraint path.

[0132] Accordingly, the target code behavior path can be the code behavior path that is determined to differ from the target specification constraint path after a consistency comparison. For example, the code behavior path in the code to be evaluated that actually executes "order cancellation interface - order status is paid - delete order record" can be determined as the target code behavior path corresponding to the aforementioned target specification constraint path.

[0133] Specifically, based on the specification rule identifier, specification type, or consistency comparison result recorded in the target validation result, the specification constraint path that generated the target validation result can be traced back and identified as the target specification constraint path. Furthermore, based on the candidate code behavior path or code object information recorded in the target validation result, the code behavior path that differs from the target specification constraint path in the consistency comparison can be identified and identified as the target code behavior path. Thus, abstract specification inconsistency results can be restored to specific specification constraint relationships and code execution relationships.

[0134] S706: Determine the inconsistency information between the above target specification constraint path and the above target code behavior path, and locate the defects in the above code to be evaluated based on the above inconsistency information to obtain defect location information.

[0135] Inconsistency information can be used to characterize the differences between the target specification constraint path and the aforementioned target code behavior path. Inconsistency information may include at least one of the following: object inconsistency information, condition inconsistency information, behavior inconsistency information, and association relationship inconsistency information. It may also include information such as the path node where the difference occurs, the type of difference, and the corresponding code location.

[0136] It is understood that defect location information may include at least one of the following: code file, code module, class, function, interface, line of code, code object identifier, and defect type.

[0137] Specifically, the specification objects, constraints, expected execution behaviors, and specification relationships in the target specification constraint path can be compared with the code objects, execution conditions, execution behaviors, and code relationships in the target code behavior path to identify discrepancies and generate corresponding inconsistency information. Furthermore, based on the location information of the code object corresponding to the inconsistency information in the abstract syntax tree or source code, the corresponding code file, function, code node, or line of code can be determined, thereby obtaining defect location information. For example, when the inconsistency information indicates that a certain expected interface call was not actually executed, the code location missing that interface call can be determined based on the conditional branch node in the corresponding code behavior path.

[0138] S708: Based on the above defect location information and the constraint information in the above target specification constraint path, generate repair suggestions corresponding to the above defect location information.

[0139] The remediation suggestions can be used to indicate how to adjust the code under evaluation to meet the relevant specification requirements. Remediation suggestions can include adding code processing logic, modifying execution conditions, replacing execution behavior, adjusting call relationships, adding permission checks, or adjusting data structures. For example, if the code lacks refund processing logic, a remediation suggestion could be generated to "call the refund interface when the order status is 'paid' and prevent direct order deletion."

[0140] Specifically, the code objects requiring adjustment can be identified based on defect location information, and the target implementation requirements that the code to be evaluated should meet can be determined based on the constraints and / or expected execution behaviors in the target specification constraint path. Then, remediation suggestions can be generated based on the differences between the actual code implementation and the target implementation requirements. For example, when defect location information indicates that a conditional branch lacks permission checks, while the target specification constraint path stipulates that permission checks should be performed before executing the target operation, a remediation suggestion can be generated to add permission check logic to the corresponding code location.

[0141] S710: Based on the above defect location information and the above repair suggestions, defect repair information is obtained, and based on the above defect repair information, the standard assessment result is obtained.

[0142] Defect remediation information can be used to describe inconsistencies in specifications and their handling methods. This information may include at least one of the following: defect location, defect type, corresponding specification information, inconsistent content, and remediation recommendations. Through this information, a correspondence can be established between defects in the code to be evaluated and the corresponding specification requirements and remediation methods.

[0143] Specifically, defect location information, corresponding inconsistency information, specification basis information in the target specification constraint path, and repair suggestions can be linked and organized to form one or more defect repair information entries. When multiple specification inconsistency issues exist, corresponding defect repair information can be generated for each target verification result, and the multiple defect repair information entries can be summarized to form the specification evaluation result corresponding to the code to be evaluated.

[0144] In this embodiment, when at least one verification result indicates a specification inconsistency, the corresponding target specification constraint path and target code behavior path are determined, and the inconsistency information between them is further identified. Subsequently, based on the aforementioned inconsistency information, defect localization is performed on the code to be evaluated, and corresponding repair suggestions are generated by combining the constraint information in the target specification constraint path, thereby forming defect repair information. Thus, specification inconsistency issues can be further located from the verification result to the specific code location, and associated with the corresponding specification requirements and repair methods, thereby improving the accuracy and traceability of defect localization and providing a clear basis for subsequent code repair.

[0145] In one embodiment, in S208 above, generating a compliance assessment report for the code to be assessed based on the above-mentioned standard assessment results may include the following steps: classifying the above-mentioned standard assessment results to obtain assessment content data, the assessment content data including grade assessment data and / or defect assessment data; if the above-mentioned assessment content data includes the above-mentioned defect assessment data, determining the standard basis information corresponding to the above-mentioned defect assessment data based on the above-mentioned target standard rule information, and establishing a mapping relationship between the above-mentioned defect assessment data and the above-mentioned standard basis information; organizing the data according to a preset report structure based on the assessment content data, and if the assessment content data includes the above-mentioned defect assessment data, organizing the defect assessment data and the corresponding standard basis information based on the mapping relationship to generate a compliance assessment report.

[0146] Optionally, the evaluation content data may include at least one of grade evaluation data and defect evaluation data, and may also include auxiliary information such as code identifier, code version, and evaluation time corresponding to the code to be evaluated. The grade evaluation data can be used to characterize the degree of compliance with specifications of the code as a whole or for each verification dimension. Grade evaluation data may include a specification compliance level, and in some embodiments, may also include specification compliance scores, specification compliance sub-scores for each verification dimension, etc. Defect evaluation data can be data used to describe specification inconsistencies existing in the code to be evaluated. Defect evaluation data may include at least one of defect location information, defect type, inconsistency information, and remediation suggestions.

[0147] Optionally, the standard basis information can be used to explain the standard requirements corresponding to the defect assessment data. The standard basis information can originate from the target standard rule information and may include the corresponding standard rules, standard objects, constraints, expected execution behaviors, etc. For example, when the defect assessment data indicates that the code to be assessed lacks permission verification, the standard rule specifying the corresponding permission verification requirements in the target standard rule information can be determined as the standard basis information corresponding to that defect.

[0148] It is understandable that the above mapping relationship can be used to characterize the specific specification requirements violated or not met by a code defect. The mapping relationship can be recorded through specification rule identifiers, defect identifiers, or other related identifiers.

[0149] It should be noted that the preset report structure can be a pre-set data structure or report template used to organize and display the assessment content data. The preset report structure may include one or more content areas such as overall assessment information, level assessment information, multi-dimensional assessment information, defect information, standard basis information, and remediation suggestions. Different report structures can be set according to the report usage requirements in different application scenarios. Furthermore, the standard compliance assessment report can be used to reflect the standard compliance status of the code being assessed. The standard compliance assessment report can be output in the form of structured data, tables, electronic documents, or visual pages, and may include information such as standard compliance level, standard compliance score, defect location, standard basis information, and remediation suggestions.

[0150] In one specific embodiment, the level assessment data can be filled into the overall assessment area of ​​a preset report structure to display the standard compliance level and standard compliance score of the code to be assessed. If the assessment content data includes defect assessment data, the defect assessment data can be filled into the defect information area to display the defect location, defect type, and remediation suggestions. Based on the aforementioned mapping relationship, the standard basis information corresponding to each defect is filled into the standard basis area of ​​the corresponding defect. For example, the standard compliance assessment report can first display the overall standard compliance level of the code to be assessed, and then further display the assessment status of each verification dimension. For code with standard inconsistencies, the report can also display the defect code location, actual code behavior, corresponding standard requirements, and remediation suggestions. Thus, the standard compliance assessment report can simultaneously reflect the overall assessment status of the code to be assessed, and, in the case of standard inconsistencies, reflect the specific defect situation and the corresponding standard basis.

[0151] In this embodiment, the evaluation results are categorized to obtain grade evaluation data and / or defect evaluation data. If defect evaluation data exists, the corresponding standard basis information is determined based on the target standard rule information, establishing a mapping relationship between the defect evaluation data and the standard basis information. Subsequently, the evaluation content data is organized according to a preset report structure. If defect evaluation data exists, the defect evaluation data and its corresponding standard basis information are organized based on the above mapping relationship to generate a standard compliance evaluation report. This allows evaluation information such as scores, grades, defects, and standard basis to be organized according to a unified structure. Furthermore, in cases of standard inconsistencies, specific defects can be mapped to corresponding standard requirements, thereby improving the readability, traceability, and utilization efficiency of the evaluation results, and facilitating subsequent problem localization, code repair, and evaluation result management.

[0152] In one embodiment, the method further includes: obtaining the admission criteria corresponding to the target code generation tool that generates the code to be evaluated; obtaining the standard compliance level and / or defect assessment data corresponding to the code to be evaluated from the standard compliance assessment report; and matching the standard compliance level and / or defect assessment data with the admission criteria to obtain the admission result corresponding to the target code generation tool.

[0153] The target code generation tool can be an AI code generation model, code generation platform, code generation plugin, code generation agent, or other tool that can automatically generate, complete, or reconstruct code based on input information.

[0154] Optionally, admission criteria may include at least one of the following: compliance level threshold, allowed number of defects, allowed types of defects, and defect severity requirements. It is understood that admission results may include admission, disapproval, or pending further evaluation.

[0155] In one implementation, appropriate admission criteria can be pre-configured for different code generation tools, application scenarios, or code types. After completing the compliance assessment of the code to be evaluated, the corresponding compliance level and / or defect assessment data can be obtained from the compliance assessment report, and the obtained data can be matched with the admission criteria. Specifically, it can be determined whether the compliance level reaches the preset level, and further, whether the number, type, or severity of defects in the defect assessment data meets the corresponding restrictions.

[0156] For example, when the admission criteria stipulate that the compliance level is not lower than level A and high-severity defects are not allowed, if the compliance level of the code to be evaluated is level A and there are no high-severity defects in the defect evaluation data, the admission result of the target code generation tool can be determined as admission; otherwise, it can be determined as non-admission.

[0157] In this embodiment, the access criteria corresponding to the target code generation tool are obtained, and the compliance level and / or defect assessment data of the code to be evaluated are extracted from the compliance assessment report. Then, the compliance level and / or defect assessment data are matched with the access criteria to obtain the access judgment result for the target code generation tool. Therefore, the code compliance assessment result can be further used for the access judgment of the code generation tool, providing a unified and quantifiable evaluation basis for the access process and reducing the subjectivity of relying solely on human experience for tool selection or access judgment. This improves the objectivity, reliability, and processing efficiency of the code generation tool access assessment.

[0158] In one implementation of S204, for certain verification dimensions, the corresponding verification results can be determined by combining the results of static code analysis and manual review. Specifically, the structured SDD specification and the code to be evaluated can be obtained, and a static code analysis tool can be used to perform a preliminary check on the code to be evaluated, obtaining static analysis results. The static analysis content can include code syntax, encoding format, naming conventions, and preset static rules. Further, the evaluation content requiring manual review can be determined based on the SDD specification, and technical personnel can review the interface contract, semantic logic, service rules, and architectural constraints, obtaining manual review results. Subsequently, the static analysis results and manual review results can be quantified and combined according to preset weights to obtain the specification compliance score and / or specification compliance level corresponding to the code to be evaluated. Thus, code analysis tools can be used to reduce the manual workload of basic checks, and manual review can supplement semantic and service-level verifications that are difficult for static analysis tools to complete, improving the completeness of code specification compliance evaluation.

[0159] In one embodiment, semantic consistency evaluation of the target specification rule information and the code to be evaluated can also be performed based on a pre-trained model. Specifically, the target specification rule information and the code to be evaluated can be preprocessed separately and input into a pre-trained code model to obtain the specification feature vector corresponding to the target specification rule information and the code feature vector corresponding to the code to be evaluated. The pre-trained code model can be a model capable of feature encoding code and natural language specifications in related technologies. Further, the similarity between the specification feature vector and the code feature vector can be calculated, and the semantic consistency between the corresponding specification rule and the code to be evaluated can be determined based on the similarity. For example, semantic similarity can be determined by cosine similarity, vector distance, or the matching probability output by the pre-trained model, and the corresponding verification result can be generated based on a preset similarity threshold. For verification content such as interface contracts, data models, or semantic logic, the semantic similarity between the corresponding specification content and the code content can be calculated separately to help determine the specification compliance of the code to be evaluated under the corresponding verification dimension.

[0160] This application uses the Structured Design Code (SDD) specification as the basis for code evaluation, transforming enterprise R&D specifications into automatically verifiable rules. It then employs a multi-dimensional weighted quantification method to generate standardized specification compliance scores and levels, thereby reducing biases caused by inconsistent understandings of the specifications among different R&D personnel and AI models, and achieving unified implementation of enterprise R&D specifications. Simultaneously, this application can quantitatively compare the code specification compliance of different AI code generation tools, model versions, and R&D teams based on the unified specification. This provides an objective basis for the selection, procurement, access, and iterative optimization of AI R&D agents, and further accumulates specification data, defect data, and evaluation result data. By combining evaluation results and defect information, it can also pinpoint deficiencies in specification compliance in code generation models, which can be used for model fine-tuning, prompt word optimization, and subsequent retesting, thus adapting to the continuous development needs of AICoding and specification-driven development.

[0161] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0162] Based on the above-mentioned code evaluation method, such as Figure 8 As shown in the embodiments of this application, a code evaluation apparatus 800 for implementing the code evaluation method described above is also provided. The code evaluation apparatus 800 includes: The first acquisition module 810 is used to acquire the target code features corresponding to the code to be evaluated, and to acquire the target specification rule information; The verification module 820 is used to perform multi-dimensional consistency verification on the target code features based on the target specification rule information mentioned above, and obtain the verification results corresponding to multiple verification dimensions respectively. The evaluation module 830 is used to evaluate the compliance of the code to be evaluated based on the verification results corresponding to the above multiple verification dimensions, and to obtain the standard evaluation result corresponding to the code to be evaluated. The standard evaluation result includes the standard compliance level and / or defect repair information. The generation module 840 is used to generate a standard compliance assessment report for the code to be assessed based on the above standard assessment results.

[0163] In one possible implementation, the first acquisition module 810 is specifically used for: Obtain the code to be evaluated; Code cleaning and abstract syntax tree parsing are performed on the code to be evaluated to determine multiple code objects in the code to be evaluated and the code relationships between the multiple code objects. Based on multiple code objects and the code relationships described above, code behavior paths and corresponding execution conditions and behaviors are determined. The code behavior paths are used to characterize the associated sequence formed by at least two code objects according to their code execution relationships. Based on multiple code objects, code relationships, code behavior paths, execution conditions, and execution behaviors, feature extraction is performed to obtain target code features.

[0164] In one possible implementation, the first acquisition module 810 is specifically used for: Obtain preset standard data; The above-mentioned preset standard data is structured and parsed to determine multiple standard objects and the constraint information corresponding to each standard object. The constraint information includes at least one of constraint conditions and expected execution behavior. Based on the above-mentioned normative objects and the constraint information corresponding to each of the above-mentioned normative objects, the normative association relationship between the above-mentioned normative objects is determined. Based on multiple normative objects, the constraint information corresponding to each normative object, and the normative relationship, a normative constraint path is determined. The normative constraint path is used to characterize the constraint relationship formed by at least two normative objects according to the normative relationship. Based on the above-mentioned normative constraint path, target normative rule information is generated.

[0165] In one possible implementation, the verification module 820 is specifically used for: Based on the normative constraint paths in the above target normative rule information, candidate code behavior paths corresponding to the above normative constraint paths are determined from the above code behavior paths. The consistency comparison between the normative objects, constraints, expected execution behaviors, and normative relationships in the above-mentioned normative constraint paths and the code objects, execution conditions, execution behaviors, and code relationships in the above-mentioned candidate code behavior paths was performed, resulting in multiple consistency comparison results. Based on the above consistency comparison results, the results are classified according to the corresponding specification types to obtain the verification results corresponding to multiple verification dimensions.

[0166] In one possible implementation, the evaluation module 830 is specifically used for: Based on the verification results corresponding to the above multiple verification dimensions, the standard compliance sub-scores corresponding to the above multiple verification dimensions are determined. Based on at least one of the rule priority, rule quantity, and preset violation severity level of the normative rules corresponding to the above-mentioned verification dimensions in the above-mentioned target normative rule information, the evaluation weights corresponding to the above-mentioned verification dimensions are determined respectively. The compliance score is obtained by weighting multiple compliance scores and their corresponding evaluation weights. Based on the above compliance score, a compliance level is determined. Based on the above compliance levels, the compliance assessment results are obtained.

[0167] In one possible implementation, the evaluation module 830 is specifically used for: If at least one of the verification results corresponding to the above-mentioned verification dimensions has a specification inconsistency, the target verification result with specification inconsistency shall be determined from the above-mentioned verification results. Based on the above target verification results, the corresponding target specification constraint path and the target code behavior path corresponding to the above target specification constraint path are determined. Identify the inconsistencies between the target specification constraint path and the target code behavior path, and locate the defects in the code to be evaluated based on the inconsistencies to obtain defect location information. Based on the above defect location information and the constraint information in the above target specification constraint path, a repair suggestion corresponding to the above defect location information is generated. Based on the above defect location information and the above repair suggestions, defect repair information is obtained, and based on the above defect repair information, the standard assessment result is obtained.

[0168] In one possible implementation, the generation module 840 is specifically used for: The above-mentioned standard assessment results are classified and processed to obtain assessment content data, which includes grade assessment data and / or defect assessment data. When the aforementioned assessment content data includes the aforementioned defect assessment data, based on the aforementioned target specification rule information, the specification basis information corresponding to the aforementioned defect assessment data is determined, and a mapping relationship is established between the aforementioned defect assessment data and the aforementioned specification basis information. Based on the assessment content data, the data is organized according to a preset report structure. When the assessment content data includes the defect assessment data, the defect assessment data and the corresponding standard basis information are organized based on the mapping relationship to generate a standard compliance assessment report.

[0169] In one possible implementation, the first acquisition module 810 is specifically used for: From the constraint information corresponding to each of the above-mentioned standard objects, identify multiple conflicting constraint information that act on the same standard object and conflict with each other; Based on at least one of the specification priority, version information and scope of application corresponding to the above-mentioned conflicting constraint information, conflict resolution is performed on the above-mentioned conflicting constraint information to determine the effective constraint information. Based on the above effective constraint information and other constraint information besides the multiple conflicting constraint information mentioned above, the target constraint information corresponding to each of the above-mentioned specification objects is determined. Based on the above-mentioned normative objects, the target constraint information corresponding to each of the above-mentioned normative objects, and the above-mentioned normative relationships, the normative constraint path is determined.

[0170] In one possible implementation, the verification module 820 is specifically used for: Determine the target verification dimension corresponding to the above-mentioned specification constraint path. The target verification dimension is one of the following: interface contract dimension, data model dimension, service rule dimension, architecture constraint dimension, security rule dimension, and coding specification dimension. Based on the above target verification dimensions, a target consistency verification strategy corresponding to the above target verification dimensions is determined from a number of preset consistency verification strategies; Based on the above-mentioned target consistency verification strategy, the consistency comparison of the above-mentioned specification constraint path and the above-mentioned candidate code behavior path is performed to obtain multiple consistency comparison results.

[0171] In one possible implementation, the above-mentioned device further includes: The second acquisition module is used to acquire the admission criteria corresponding to the target code generation tool that generates the above-mentioned code to be evaluated. The third acquisition module is used to obtain the standard compliance level and / or defect assessment data corresponding to the code to be evaluated from the above standard compliance assessment report. The determination module is used to match the above-mentioned standard compliance level and / or the above-mentioned defect assessment data with the above-mentioned admission judgment conditions to obtain the admission judgment result corresponding to the above-mentioned target code generation tool.

[0172] The division of modules in the above-described code evaluation device is for illustrative purposes only. In other embodiments, the code evaluation device can be divided into different modules as needed to complete all or part of the functions of the above-described code evaluation device. The implementation of each module in the code evaluation device provided in this application embodiment can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the code evaluation method described in the embodiments of this application.

[0173] This application also provides an electronic device, which can be a server, and its internal structure diagram can be as follows: Figure 9 As shown, this electronic device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The processor executes computer programs to implement a code evaluation method.

[0174] Those skilled in the art will understand that Figure 9The structures shown are merely block diagrams of some structures related to the embodiments of this application and do not constitute a limitation on the electronic devices on which the embodiments of this application are applied. Specific electronic devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0175] In one possible implementation, a computer storage medium is provided that stores instructions, which, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0176] In one possible implementation, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0177] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer storage medium or transmitted through the computer storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0178] It should be noted that the information (including but not limited to target specification rules information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the code to be evaluated and target code features involved in this application were obtained under full authorization.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0180] The above-described embodiments are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Various modifications and improvements made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application should fall within the protection scope defined by the claims.

[0181] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A code evaluation method, comprising: Obtain the target code features corresponding to the code to be evaluated, and obtain the target specification rule information, wherein the target specification rule information is used to characterize the specification requirements that the code to be evaluated should meet; Based on the target specification rule information, a multi-dimensional consistency check is performed on the target code features to obtain the check results corresponding to multiple check dimensions. Based on the verification results corresponding to the multiple verification dimensions, the code to be evaluated is evaluated for compliance with the standards, and the standard evaluation result corresponding to the code to be evaluated is obtained. The standard evaluation result includes the standard compliance level and / or defect repair information. Based on the standard evaluation results, a standard compliance evaluation report is generated for the code to be evaluated.

2. The method as described in claim 1, wherein obtaining the target code features corresponding to the code to be evaluated includes: Obtain the code to be evaluated; The code to be evaluated is cleaned and an abstract syntax tree is parsed to determine multiple code objects in the code to be evaluated and the code relationships between the multiple code objects. Based on multiple code objects and the code association relationships, a code behavior path and the execution conditions and execution behaviors corresponding to the code behavior path are determined. The code behavior path is used to characterize the association sequence formed by at least two code objects according to the code execution relationship. Feature extraction is performed based on multiple code objects, code relationships, code behavior paths, execution conditions, and execution behaviors to obtain target code features.

3. The method as described in claim 2, wherein obtaining the target specification rule information includes: Obtain preset standard data; The preset specification data is structured and parsed to determine multiple specification objects and the constraint information corresponding to each specification object. The constraint information includes at least one of constraint conditions and expected execution behavior. Based on the multiple standard objects and the constraint information corresponding to each standard object, the standard association relationship between the multiple standard objects is determined; Based on multiple standard objects, the constraint information corresponding to each standard object, and the standard association relationship, a standard constraint path is determined. The standard constraint path is used to characterize the constraint relationship formed by at least two standard objects according to the standard association relationship. Based on the specified constraint path, target specification rule information is generated.

4. The method as described in claim 3, wherein performing multi-dimensional consistency verification on the target code features based on the target specification rule information to obtain verification results corresponding to multiple verification dimensions includes: Based on the specification constraint paths in the target specification rule information, candidate code behavior paths corresponding to the specification constraint paths are determined from the code behavior paths; The consistency comparison between the specification objects, constraints, expected execution behaviors, and specification relationships in the specification constraint path and the code objects, execution conditions, execution behaviors, and code relationships in the candidate code behavior path is performed to obtain multiple consistency comparison results; Based on the multiple consistency comparison results, they are classified according to the corresponding specification types to obtain the verification results corresponding to multiple verification dimensions.

5. The method as described in claim 1, wherein the step of evaluating the code to be evaluated based on the validation results corresponding to the plurality of validation dimensions to obtain the specification evaluation result corresponding to the code to be evaluated includes: Based on the verification results corresponding to the multiple verification dimensions, determine the specification compliance sub-scores corresponding to the multiple verification dimensions. Based on at least one of the rule priority, rule quantity, and preset violation severity level of the normative rules corresponding to the multiple verification dimensions in the target normative rule information, the evaluation weights corresponding to the multiple verification dimensions are determined respectively. The compliance score is obtained by weighting multiple compliance scores and their corresponding evaluation weights. The compliance level is determined based on the compliance score. Based on the compliance level, the standard evaluation results are obtained.

6. The method as described in claim 4, wherein the step of evaluating the code to be evaluated based on the validation results corresponding to the plurality of validation dimensions to obtain the specification evaluation result corresponding to the code to be evaluated includes: If at least one of the verification results corresponding to the multiple verification dimensions indicates a specification inconsistency, the target verification result with specification inconsistency shall be determined from the multiple verification results. Based on the target verification results, the corresponding target specification constraint path and the target code behavior path corresponding to the target specification constraint path are determined. Inconsistency information between the target specification constraint path and the target code behavior path is determined, and the defect location of the code to be evaluated is performed based on the inconsistency information to obtain defect location information; Based on the defect location information and the constraint information in the target specification constraint path, a repair suggestion corresponding to the defect location information is generated. Based on the defect location information and the repair suggestions, defect repair information is obtained, and based on the defect repair information, a standard evaluation result is obtained.

7. The method as described in claim 1, wherein generating a compliance assessment report for the code to be assessed based on the specification assessment results includes: The evaluation results are classified and processed to obtain evaluation content data, which includes grade evaluation data and / or defect evaluation data. When the evaluation content data includes the defect evaluation data, based on the target specification rule information, the specification basis information corresponding to the defect evaluation data is determined, and a mapping relationship between the defect evaluation data and the specification basis information is established; Based on the assessment content data, the data is organized according to a preset report structure. When the assessment content data includes the defect assessment data, the defect assessment data and the corresponding standard basis information are organized based on the mapping relationship to generate a standard compliance assessment report.

8. The method as described in claim 3, wherein determining the specification constraint path based on multiple specification objects, constraint information corresponding to each specification object, and specification association relationships includes: From the constraint information corresponding to each of the aforementioned specification objects, determine multiple conflicting constraint information that act on the same specification object and have conflicts; Based on at least one of the specification priority, version information and scope of application corresponding to the multiple conflict constraint information, conflict resolution is performed on the multiple conflict constraint information to determine the effective constraint information; Based on the effective constraint information and other constraint information besides the multiple conflict constraint information, the target constraint information corresponding to each of the specification objects is determined. Based on multiple specification objects, the target constraint information corresponding to each specification object, and the specification association relationship, the specification constraint path is determined.

9. The method as described in claim 4, wherein the consistency comparison of the specification objects, constraints, expected execution behaviors, and specification relationships in the specification constraint path with the code objects, execution conditions, execution behaviors, and code relationships in the candidate code behavior path, to obtain multiple consistency comparison results, includes: Determine the target verification dimension corresponding to the specified constraint path. The target verification dimension is one of the following: interface contract dimension, data model dimension, service rule dimension, architecture constraint dimension, security rule dimension, and coding specification dimension. Based on the target verification dimension, a target consistency verification strategy corresponding to the target verification dimension is determined from a plurality of preset consistency verification strategies; Based on the target consistency verification strategy, the consistency comparison between the specification constraint path and the candidate code behavior path is performed to obtain multiple consistency comparison results.

10. The method of claim 7, further comprising: Obtain the admission criteria corresponding to the target code generation tool that generates the code to be evaluated; Obtain the compliance level and / or defect assessment data corresponding to the code to be evaluated from the compliance assessment report; Based on the standard compliance level and / or the defect assessment data, the admission judgment results corresponding to the target code generation tool are matched with the admission judgment conditions to obtain the admission judgment results.

11. A code evaluation apparatus, comprising: The first acquisition module is used to acquire the target code features corresponding to the code to be evaluated, and to acquire the target specification rule information, wherein the target specification rule information is used to characterize the specification requirements that the code to be evaluated should meet; The verification module is used to perform multi-dimensional consistency verification on the target code features based on the target specification rule information, and obtain the verification results corresponding to multiple verification dimensions respectively; The evaluation module is used to evaluate the code to be evaluated based on the verification results corresponding to the multiple verification dimensions, and to obtain the standard evaluation result corresponding to the code to be evaluated. The standard evaluation result includes the standard compliance level and / or defect repair information. The generation module is used to generate a standard compliance assessment report corresponding to the code to be assessed based on the standard assessment results.

12. An electronic device, comprising: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-10.

13. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1-10.

14. A computer program product comprising instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of the method as claimed in any one of claims 1-10.

Images