A software vulnerability benchmark construction method, device and equipment and storage medium

Abstract

The method comprises the following steps: obtaining a submission record of an artificial intelligence infrastructure code warehouse; performing security semantic feature mining on the submission record to generate a security semantic feature set, calculating a security correlation score of each submission record relative to the security semantic feature set, and generating a candidate queue; performing code review model pre-screening and consistency review on the submission record in the candidate queue to generate a vulnerability event record; based on the vulnerability event record, locating a pre-repair version and a post-repair version, obtaining a code difference between the versions, determining pre-repair and post-repair codes from the pre-repair version and the post-repair version based on the code difference; performing sample alignment on the pre-repair and post-repair codes to generate a patch sample pair; performing quality checking on the vulnerability event record and the patch sample pair, and outputting a software vulnerability benchmark; and effectively realizing the construction of the software vulnerability benchmark.

Description

Technical Field

[0001] This disclosure relates to the field of software vulnerability detection technology, and in particular to a method, apparatus, device and storage medium for constructing a software vulnerability benchmark. Background Technology

[0002] Currently, building software vulnerability benchmarks mainly relies on manual review of each submission record and keyword matching based on a fixed set of security keywords, combined with manual verification to construct a vulnerability sample set. However, AI infrastructure involves various heterogeneous code repositories, and the submission specifications of different projects vary significantly. Fixed keyword sets are insufficient to cover diverse code, easily leading to a large number of missed and false positives. Secondly, manual review of each submission record is inefficient and difficult to scale up for building vulnerability benchmark datasets. Finally, the lack of an effective correlation mechanism between vulnerability events and specific code samples makes it difficult to trace the final output samples back to specific vulnerability events and fix code submissions. Summary of the Invention

[0003] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a method, apparatus, device, and storage medium for constructing a software vulnerability benchmark.

[0004] An exemplary embodiment of this disclosure provides a method for constructing a software vulnerability benchmark, the method comprising: Retrieve the commit history of the AI ​​infrastructure code repository; Security semantic feature mining is performed on the submission records to generate a security semantic feature set. The security relevance score of each submission record relative to the security semantic feature set is calculated to generate a candidate queue. The submission records in the candidate queue are pre-screened by a code review model and undergo consistency verification to generate vulnerability event records; Based on the vulnerability event records, by locating the pre-fix version and the post-fix version, the code differences between the versions are obtained, and based on the code differences, the code before and after the fix is ​​determined from the pre-fix version and the post-fix version; The code before and after the repair is aligned to generate patch sample pairs; The vulnerability event records and patch samples are subjected to quality verification, and a software vulnerability benchmark is output.

[0005] In some exemplary embodiments, the code review model includes a first code review model and a second code review model; The code review model pre-screening includes an initial review by a first code review model and a review by a second code review model. The step of performing code review model pre-screening and consistency verification on the submission records in the candidate queue, and generating vulnerability event records, includes: Based on the local code repository mirror, obtain the actual code differences of the commit records in the candidate queue; The actual code differences are input into the first code review model to perform a preliminary review by the first code review model, and a first judgment result is output. The first judgment result includes the model being determined to be a vulnerability, pending judgment, and the model being determined to be a non-vulnerable. If the first judgment result is that the model is determined to be a vulnerability or to be judged, then the corresponding real code difference is input into the second code review model for second code review model verification, and the second judgment result is output. The second judgment result includes the model being determined to be a vulnerability, to be judged, and the model being determined to be a non-vulnerable. If the first judgment result indicates that the model is not a vulnerability, then the corresponding submission record will be removed; Based on the first judgment result and the second judgment result, a model pre-screening result is generated, which includes model identification vulnerabilities, model pending judgment, blank or model rejection; If the model pre-screening result is a vulnerability identified by the model, the corresponding submission record is taken as a high-confidence vulnerability candidate. The high-confidence vulnerability candidate is subjected to consistency review based on preset judgment parameters to obtain the first review result. The judgment parameters include title, submission information, source code file, actual code difference, candidate general defect enumeration output by the code review model, vulnerability type and judgment reason. If the first review result is a confirmed vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is that the model needs to be judged, then based on the candidate general defect enumeration, vulnerability type, judgment reason and disagreement content output by the code review model, a consistency review is performed to obtain a second review result; If the second review result confirms a vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is blank or the model is rejected, no vulnerability event record will be generated.

[0006] In some exemplary embodiments, the step of generating a model pre-screening result based on the first judgment result and the second judgment result includes model identification vulnerabilities, model pending judgment, blank, or model rejection, including: If the output of the first code review model indicates that the model is not a vulnerability, then the model pre-screening result is blank or the model is rejected. If the output of the first code review model is that the model has identified a vulnerability, the output of the second code review model is that the model has identified a vulnerability, and the candidate general defect enumerations output by the two models are consistent, then the model pre-screening result is that the model has identified a vulnerability. If the first code review model outputs a vulnerability, the second code review model outputs a pending judgment or a non-vulnerability, or the candidate general defect enumerations output by the two models are inconsistent, then the model pre-screening result is pending judgment. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model determines that it is not a vulnerability", then the model pre-screening result is blank or the model rejects it. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model is determined to be a vulnerability or to be judged", then the model pre-screening result is "the model is to be judged".

[0007] In some exemplary embodiments, the code before and after the repair includes code before the repair and code after the repair; The process of obtaining code differences between versions based on the vulnerability event records by locating the pre-fix and post-fix versions, and determining the code before and after the fix based on the code differences, includes: Based on the vulnerability event records, the pre-fix version and the post-fix version are located through a preset code location strategy. The code location strategy includes syntax tree parsing, function boundary identification based on abstract syntax tree, function signature matching based on language-specific rules, and function boundary expansion based on difference block context. Obtain the modified files, line numbers, difference blocks, change types, and file extensions between the pre-fix version and the post-fix version, and filter out non-source code files while retaining source code files; Based on the source code file, the modified line numbers, and the difference blocks, determine the affected functions and their location information; Based on the location information, the code of the affected function is extracted from the pre-fix version and the post-fix version, respectively, and used as the pre-fix code and the post-fix code.

[0008] In some exemplary embodiments, sample alignment is performed on the code before and after the repair to generate patch sample pairs, including: Based on code name, code signature, file path, calling context, and code similarity, sample alignment is performed between the code before and the code after repair. Based on the sample alignment results, the internal vulnerability numbers of the vulnerability events are associated to generate patch sample pairs. The patch sample pairs include internal vulnerability number, project group, repository name, commit after fix, commit before fix, source code file path, function code before fix, function code after fix, function name, function signature, change line number, and confirmed vulnerability tag.

[0009] In some exemplary embodiments, the commit record includes a commit title, commit information, file path, code differences, function name, class name, method name, variable name, import dependencies, API call, and configuration item name; The security semantic feature set includes a strong security semantic feature set and a weak security semantic feature set; The step of performing security semantic feature mining on the submitted records to generate a security semantic feature set includes: The submission records are normalized to obtain candidate feature units; Security semantic feature mining is performed on the candidate feature units, and a security semantic feature score is calculated. The security semantic feature score includes semantic association score, repair intention score, code evidence score, statistical differentiation score, and stability score. The security semantic feature scores are weighted to obtain the comprehensive score of the candidate feature units, and then sorted based on the comprehensive score. In response to the stability score being not lower than a preset score threshold and the sorting being within a preset first interval, a set of strong security semantic features is generated. If the stability score is not lower than a preset score threshold and the sorting is in a preset second interval, then a set of weak security semantic features is generated.

[0010] In some exemplary embodiments, calculating the security relevance score of each submitted record relative to the security semantic feature set and generating a candidate queue includes: Based on the security semantic feature set, a security relevance score is calculated for each submission record. The security relevance score includes semantic security feature hit score, code evidence location weight, repair intent weight, artificial intelligence infrastructure risk weight, and noise penalty item. Based on the security relevance score, the submission records are divided into strong security category, weak security category, and rejection category; Based on the security relevance scores in descending order, the submission records corresponding to the strong security category and the weak security category are sorted to generate a candidate queue, which includes a full candidate queue, a model pre-screening queue, and a review queue.

[0011] An exemplary embodiment of this disclosure also provides a software vulnerability benchmark construction apparatus, the apparatus comprising: The acquisition module is used to retrieve the commit history of the AI ​​infrastructure code repository; The first generation module is used to perform security semantic feature mining on the submission records, generate a security semantic feature set, calculate the security relevance score of each submission record relative to the security semantic feature set, and generate a candidate queue. The second generation module is used to perform code review model pre-screening and consistency verification on the submission records in the candidate queue, and generate vulnerability event records. The determination module is used to obtain the code differences between the versions based on the vulnerability event record by locating the version before and after the fix, and to determine the code before and after the fix from the version before and after the fix based on the code differences; The third generation module is used to perform sample alignment on the code before and after the repair to generate patch sample pairs; The output module is used to perform quality verification on the vulnerability event records and the patch sample pairs, and output a software vulnerability benchmark.

[0012] In some exemplary embodiments, the commit record includes a commit title, commit information, file path, code differences, function name, class name, method name, variable name, import dependencies, API call, and configuration item name; The security semantic feature set includes a strong security semantic feature set and a weak security semantic feature set; The first generation module is specifically used for: The submission records are normalized to obtain candidate feature units; Security semantic feature mining is performed on the candidate feature units, and a security semantic feature score is calculated. The security semantic feature score includes semantic association score, repair intention score, code evidence score, statistical differentiation score, and stability score. The security semantic feature scores are weighted to obtain the comprehensive score of the candidate feature units, and then sorted based on the comprehensive score. In response to the stability score being not lower than a preset score threshold and the sorting being within a preset first interval, a set of strong security semantic features is generated. If the stability score is not lower than a preset score threshold and the sorting is in a preset second interval, then a set of weak security semantic features is generated.

[0013] In some exemplary embodiments, the first generation module is specifically used for: Based on the security semantic feature set, a security relevance score is calculated for each submission record. The security relevance score includes semantic security feature hit score, code evidence location weight, repair intent weight, artificial intelligence infrastructure risk weight, and noise penalty item. Based on the security relevance score, the submission records are divided into strong security category, weak security category, and rejection category; Based on the security relevance scores in descending order, the submission records corresponding to the strong security category and the weak security category are sorted to generate a candidate queue, which includes a full candidate queue, a model pre-screening queue, and a review queue.

[0014] In some exemplary embodiments, the code review model includes a first code review model and a second code review model; The code review model pre-screening includes an initial review by a first code review model and a review by a second code review model. The second generation module is specifically used for: Based on the local code repository mirror, obtain the actual code differences of the commit records in the candidate queue; The actual code differences are input into the first code review model to perform a preliminary review by the first code review model, and a first judgment result is output. The first judgment result includes the model being determined to be a vulnerability, pending judgment, and the model being determined to be a non-vulnerable. If the first judgment result is that the model is determined to be a vulnerability or to be judged, then the corresponding real code difference is input into the second code review model for second code review model verification, and the second judgment result is output. The second judgment result includes the model being determined to be a vulnerability, to be judged, and the model being determined to be a non-vulnerable. If the first judgment result indicates that the model is not a vulnerability, then the corresponding submission record will be removed; Based on the first judgment result and the second judgment result, a model pre-screening result is generated, which includes model identification vulnerabilities, model pending judgment, blank or model rejection; If the model pre-screening result is a vulnerability identified by the model, the corresponding submission record is taken as a high-confidence vulnerability candidate. The high-confidence vulnerability candidate is subjected to consistency review based on preset judgment parameters to obtain the first review result. The judgment parameters include title, submission information, source code file, actual code difference, candidate general defect enumeration output by the code review model, vulnerability type and judgment reason. If the first review result is a confirmed vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is that the model needs to be judged, then based on the candidate general defect enumeration, vulnerability type, judgment reason and disagreement content output by the code review model, a consistency review is performed to obtain a second review result; If the second review result confirms a vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is blank or the model is rejected, no vulnerability event record will be generated.

[0015] In some exemplary embodiments, the second generation module includes: If the output of the first code review model indicates that the model is not a vulnerability, then the model pre-screening result is blank or the model is rejected. If the output of the first code review model is that the model has identified a vulnerability, the output of the second code review model is that the model has identified a vulnerability, and the candidate general defect enumerations output by the two models are consistent, then the model pre-screening result is that the model has identified a vulnerability. If the first code review model outputs a vulnerability, the second code review model outputs a pending judgment or a non-vulnerability, or the candidate general defect enumerations output by the two models are inconsistent, then the model pre-screening result is pending judgment. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model determines that it is not a vulnerability", then the model pre-screening result is blank or the model rejects it. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model is determined to be a vulnerability or to be judged", then the model pre-screening result is "the model is to be judged".

[0016] In some exemplary embodiments, the code before and after the repair includes code before the repair and code after the repair; The determining module is specifically used for: Based on the vulnerability event records, the pre-fix version and the post-fix version are located through a preset code location strategy. The code location strategy includes syntax tree parsing, function boundary identification based on abstract syntax tree, function signature matching based on language-specific rules, and function boundary expansion based on difference block context. Obtain the modified files, line numbers, difference blocks, change types, and file extensions between the pre-fix version and the post-fix version, and filter out non-source code files while retaining source code files; Based on the source code file, the modified line numbers, and the difference blocks, determine the affected functions and their location information; Based on the location information, the code of the affected function is extracted from the pre-fix version and the post-fix version, respectively, and used as the pre-fix code and the post-fix code.

[0017] In some exemplary embodiments, the third generation module is specifically used for: Based on code name, code signature, file path, calling context, and code similarity, sample alignment is performed between the code before and the code after repair. Based on the sample alignment results, the internal vulnerability numbers of the vulnerability events are associated to generate patch sample pairs. The patch sample pairs include internal vulnerability number, project group, repository name, commit after fix, commit before fix, source code file path, function code before fix, function code after fix, function name, function signature, change line number, and confirmed vulnerability tag.

[0018] An exemplary embodiment of this disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the software vulnerability benchmarking method provided in the exemplary embodiment of this disclosure.

[0019] This disclosure also provides a computer-readable storage medium storing a computer program for performing a software vulnerability benchmarking method as provided in this disclosure.

[0020] The technical solution provided by the exemplary embodiments of this disclosure has the following advantages compared with the prior art: The software vulnerability benchmark construction method provided in this exemplary embodiment obtains the commit records of an artificial intelligence infrastructure code repository; performs security semantic feature mining on the commit records to generate a security semantic feature set; calculates the security relevance score of each commit record relative to the security semantic feature set to generate a candidate queue; performs code review model pre-screening and consistency verification on the commit records in the candidate queue to generate vulnerability event records; based on the vulnerability event records, obtains the code differences between the versions by locating the pre-fix and post-fix versions; determines the code before and after the fix from the pre-fix and post-fix versions based on the code differences; performs sample alignment on the code before and after the fix to generate patch sample pairs; performs quality verification on the vulnerability event records and the patch sample pairs, and outputs a software vulnerability benchmark; thus effectively realizing the construction of a software vulnerability benchmark. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the exemplary embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A flowchart illustrating a software vulnerability benchmark construction method provided for exemplary embodiments of this disclosure; Figure 2 A schematic diagram of the structure of a software vulnerability benchmark construction apparatus provided as an exemplary embodiment of this disclosure; Figure 3 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. Detailed Implementation

[0023] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While some exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and exemplary embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one exemplary embodiment" means "at least one exemplary embodiment"; the term "another exemplary embodiment" means "at least one additional exemplary embodiment"; the term "some exemplary embodiments" means "at least some exemplary embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] To address the aforementioned issues, this disclosure provides an exemplary embodiment of a method for constructing a software vulnerability benchmark, which will be described below with reference to specific exemplary embodiments.

[0030] Figure 1This is a flowchart illustrating a software vulnerability benchmark construction method provided as an exemplary embodiment of the present disclosure. The method can be executed by a software vulnerability benchmark construction apparatus, wherein the apparatus can be implemented in software and / or hardware, and is generally integrated into an electronic device.

[0031] An exemplary embodiment of this disclosure provides a method for constructing a software vulnerability benchmark, comprising: Step S101: Obtain the commit history of the AI ​​infrastructure code repository.

[0032] In some exemplary embodiments, the AI ​​infrastructure code repository image is read, and the project, repository, branch, reference, commit history, and file change information are parsed to generate the commit history of the AI ​​infrastructure code repository. The code repository image includes GitLab images, raw repositories, ordinary Git repositories, or code repositories exported from other version control systems. The AI ​​infrastructure code repository includes software repositories such as model service platforms, AI-Coding platforms, inference services, training task management systems, data processing pipelines, plug-in toolchains, permission systems, interface services, and operation and maintenance scripts.

[0033] As an example, generate or extract the following basic fields to obtain the commit history of the AI ​​infrastructure code repository: project: Project group, used to identify which business system or infrastructure component the sample comes from; repository: repository name, used to view code differences later; fix_commit: Candidate fix commit, corresponding to the version after the fix, i.e., after commit; parent_commit: Commit before the fix, corresponding to the version before the fix, i.e., before commit; date: Submission time, used to record the sample time, which can be used for time segmentation later; author: The submitting author, used to help determine the source of the submission and whether automated bots were involved; subject: The submission title is one of the most important fields in the consistency review; message: Submit complete information. If it is more detailed than subject, it can assist in consistency verification.

[0034] Step S102: Perform security semantic feature mining on the submission records to generate a security semantic feature set, calculate the security relevance score of each submission record relative to the security semantic feature set, and generate a candidate queue.

[0035] In some exemplary embodiments, the commit record includes a commit title, commit information, file path, code differences, function name, class name, method name, variable name, import dependencies, API call, and configuration item name; The security semantic feature set includes a strong security semantic feature set and a weak security semantic feature set; The step of performing security semantic feature mining on the submitted records to generate a security semantic feature set includes: The submission records are normalized to obtain candidate feature units; Security semantic feature mining is performed on the candidate feature units, and a security semantic feature score is calculated. The security semantic feature score includes semantic association score, repair intention score, code evidence score, statistical differentiation score, and stability score. The security semantic feature scores are weighted to obtain the comprehensive score of the candidate feature units, and then sorted based on the comprehensive score. In response to the stability score being not lower than a preset score threshold and the sorting being within a preset first interval, a set of strong security semantic features is generated. If the stability score is not lower than a preset score threshold and the sorting is in a preset second interval, then a set of weak security semantic features is generated.

[0036] In some exemplary embodiments, calculating the security relevance score of each submitted record relative to the security semantic feature set and generating a candidate queue includes: Based on the security semantic feature set, a security relevance score is calculated for each submission record. The security relevance score includes semantic security feature hit score, code evidence location weight, repair intent weight, artificial intelligence infrastructure risk weight, and noise penalty item. Based on the security relevance score, the submission records are divided into strong security category, weak security category, and rejection category; Based on the security relevance scores in descending order, the submission records corresponding to the strong security category and the weak security category are sorted to generate a candidate queue, which includes a full candidate queue, a model pre-screening queue, and a review queue.

[0037] In some exemplary embodiments, the submission records are normalized to obtain candidate feature units. The normalization process includes case normalization, camelCase naming splitting, underscore naming splitting, path segmentation, code comment removal, general stop word filtering, version number filtering, and pure format change filtering.

[0038] In some exemplary embodiments, the candidate feature unit is the basic text unit used to characterize the minimal semantics of the submission content, obtained after normalizing the submission records of the acquired artificial intelligence infrastructure code repository. It is the basic input data for subsequent security semantic feature mining algorithms. The candidate feature unit includes submission title, submission information, file path, code differences, function name, class name, method name, variable name, import dependencies, calling interface, and configuration item name, etc.

[0039] In some exemplary embodiments, security semantic feature mining is performed on the candidate feature units to calculate a security semantic feature score. The security semantic feature score includes a semantic association score, a repair intent score, a code evidence score, a statistical differentiation score, and a stability score. For any candidate feature unit t, the security semantic feature score is calculated as follows: Semantic association score SemScore(t): The candidate feature unit and its context are encoded into a vector, and the vector similarity is calculated with the security semantic reference text composed of vulnerability type description, general defect enumeration (CWE) description, security remediation intention description, and artificial intelligence infrastructure risk description to obtain the degree of association between the feature and the security semantics.

[0040] PatchScore(t): Based on whether there are any remedial code changes in the difference block where the feature is located, such as enhanced permission verification, enhanced input verification, path normalization, replacement of sensitive interface calls, tightening of exception handling, supplementation of authentication logic, convergence of configuration exposure, enhancement of sandbox boundaries, enhancement of file access restrictions, and enhancement of command execution constraints, the correlation between the feature and security remediation behavior is calculated.

[0041] CodeScore(t): The strength of the evidence at the code level is calculated based on whether the feature appears in the source code file, function signature, call statement, condition judgment, exception branch, permission judgment, path processing, serialization / deserialization, external input processing, model service interface, plugin execution entry point, task scheduling entry point, data upload and download entry point, etc.

[0042] The statistical discrimination score StatScore(t) compares the set of submissions containing the intent to fix the problem with the set of ordinary submissions, calculates the logarithmic advantage ratio of candidate feature units in the two types of submissions, and combines the inverse submission frequency to suppress high-frequency common words, so that features that appear more concentrated in the context of security fixes receive higher weights.

[0043] Stability score StableScore(t): The submitted set is resampled multiple times, and the ranking of candidate feature units is recalculated in each round of resampling. If the feature is consistently in the high score range in multiple rounds of resampling (e.g., the top 30% of candidate feature units), its stability score is increased; otherwise, its stability score is decreased.

[0044] In some exemplary embodiments, the security semantic feature scores are weighted to obtain a comprehensive score for the candidate feature units: ; in, Represents the candidate feature unit The overall score; Indicates the semantic association score; The score indicates the intention to repair; This indicates the score for the code evidence. Indicates the statistical distinction score; Indicates the stability score; The weights representing the semantic relevance scores; The weights represent the scores for the intention to repair. The weights representing the code evidence scores; Indicates the weight of the statistically distinguishing score; The weights represent the stability scores.

[0045] In some exemplary embodiments, the first interval and the second interval do not overlap and are both at the top of the sorting list. The first interval is at the very top of the sorting list, and the second interval follows the first interval. This is used to divide the set of strong security semantic features into the set of weak security semantic features. If the sorting list is not in the first interval or the second interval, it will not be included in the set of security semantic features.

[0046] In some exemplary embodiments, a security relevance score is calculated for each submission record based on the security semantic feature set. The security relevance score includes a semantic security feature hit score, code evidence location weight, repair intent weight, AI infrastructure risk weight, and noise penalty item. Candidate feature units appearing in real code differences, function signatures, permission judgments, input validation, file access, command execution, plugin calls, model service interfaces, and task scheduling entry points receive high location weights; candidate feature units appearing only in the submission title or submission description receive low location weights; submission records that only modify documents, comments, formatting, version numbers, test snapshots, or dependency lock files and have no source code repair evidence are subject to noise penalties. The semantic security feature hit score is obtained by weighted summation of the scores of the hit security semantic features. The code evidence location weight (high location weight, low location weight), repair intent weight, AI infrastructure risk weight, and noise penalty item can all be obtained through a preset configuration mapping table. The configuration mapping table includes different repair intents, AI infrastructure risk types, code evidence, and noise penalty items, as well as their corresponding weight coefficients.

[0047] In some exemplary embodiments, the following autofilter fields are generated in this step: auto_mined_strong_security_features: Strong security semantic features generated by the security semantic feature automatic mining algorithm and found in this submission; auto_mined_weak_security_features: Weak security semantic features generated by the security semantic feature automatic mining algorithm and found in this submission; security_feature_score: Security relevance score of candidate submission records; security_feature_evidence_locations: The location of evidence of security semantic features in commit titles, commit messages, file paths, difference blocks, function signatures, call statements, conditional statements, or configuration items; patch_intent_evidence: Code evidence in the difference block that demonstrates the intent to fix security issues; ai_infra_risk_evidence: Evidence of risks to artificial intelligence infrastructure related to model interfaces, inference services, training tasks, plugin execution, sandbox isolation, token authentication, access control, upload and download, data paths, configuration keys, remote command execution, or code execution; source_code_files: The modified source code files, used to determine whether the submission is suitable for further extraction of before-and-after sample pairs.

[0048] In some exemplary embodiments, the model pre-screening and review queues select the top K based on the security relevance scores in descending order to focus on submission records with high security relevance scores and improve resource utilization; the full candidate queue is used to ensure the complete traceability of submission records.

[0049] In some exemplary embodiments, the following sorting and review auxiliary fields are retained in the candidate queue: rank: The ranking of candidate samples. The higher the ranking, the more priority to view them, but it does not mean that the probability of vulnerability is higher. num_source_code_files: Modify the number of source code files. The fewer the number, the easier it is to verify consistency. total_changed_lines: The total number of lines changed. The smaller the number of lines changed, the easier it is to locate the repair point. cwe_candidates: A candidate general defect enumeration generated based on security semantic features, actual code differences, change locations, and vulnerability type descriptions, i.e., candidate CWEs; this field is only used as a reference for subsequent code review model pre-screening and consistency verification, and is not used as the final vulnerability label. The final CWE is based on the manual_cwe_ids confirmed after consistency verification.

[0050] As an example, candidate feature units with a stability score of not less than 0.6 (preset score threshold) and a comprehensive score ranking in the top 10% (first interval) are classified into a set of strong security semantic features; candidate feature units with a stability score of not less than 0.6 (preset score threshold) and a comprehensive score ranking between the top 10% and the top 30% (second interval) are classified into a set of weak security semantic features.

[0051] As an example, if a candidate feature unit appears in the actual code differences, function signatures, permission checks, input validation, file access, command execution, plugin calls, model service interfaces, or task scheduling entry points, a high position weight of 0.6 is added; if the candidate feature unit only appears in the commit title or commit description, a low position weight of 0.3 is obtained; if the commit record only modifies the documentation, comments, format, version number, test snapshot, or dependency lock file and there is no evidence of source code repair, a noise penalty is applied with a coefficient of -0.2.

[0052] Step S103: Perform code review model pre-screening and consistency verification on the submission records in the candidate queue to generate vulnerability event records.

[0053] In some exemplary embodiments, the code review model includes a first code review model and a second code review model; The code review model pre-screening includes an initial review by a first code review model and a review by a second code review model. The step of performing code review model pre-screening and consistency verification on the submission records in the candidate queue, and generating vulnerability event records, includes: Based on the local code repository mirror, obtain the actual code differences of the commit records in the candidate queue; The actual code differences are input into the first code review model to perform a preliminary review by the first code review model, and a first judgment result is output. The first judgment result includes the model being determined to be a vulnerability, pending judgment, and the model being determined to be a non-vulnerable. If the first judgment result is that the model is determined to be a vulnerability or to be judged, then the corresponding real code difference is input into the second code review model for second code review model verification, and the second judgment result is output. The second judgment result includes the model being determined to be a vulnerability, to be judged, and the model being determined to be a non-vulnerable. If the first judgment result indicates that the model is not a vulnerability, then the corresponding submission record will be removed; Based on the first judgment result and the second judgment result, a model pre-screening result is generated, which includes model identification vulnerabilities, model pending judgment, blank or model rejection; If the model pre-screening result is a vulnerability identified by the model, the corresponding submission record is taken as a high-confidence vulnerability candidate. The high-confidence vulnerability candidate is subjected to consistency review based on preset judgment parameters to obtain the first review result. The judgment parameters include title, submission information, source code file, actual code difference, candidate general defect enumeration output by the code review model, vulnerability type and judgment reason. If the first review result is a confirmed vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is that the model needs to be judged, then based on the candidate general defect enumeration, vulnerability type, judgment reason and disagreement content output by the code review model, a consistency review is performed to obtain a second review result; If the second review result confirms a vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is blank or the model is rejected, no vulnerability event record will be generated.

[0054] In some exemplary embodiments, the step of generating a model pre-screening result based on the first judgment result and the second judgment result includes model identification vulnerabilities, model pending judgment, blank, or model rejection, including: If the output of the first code review model indicates that the model is not a vulnerability, then the model pre-screening result is blank or the model is rejected. If the output of the first code review model is that the model has identified a vulnerability, the output of the second code review model is that the model has identified a vulnerability, and the candidate general defect enumerations output by the two models are consistent, then the model pre-screening result is that the model has identified a vulnerability. If the first code review model outputs a vulnerability, the second code review model outputs a pending judgment or a non-vulnerability, or the candidate general defect enumerations output by the two models are inconsistent, then the model pre-screening result is pending judgment. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model determines that it is not a vulnerability", then the model pre-screening result is blank or the model rejects it. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model is determined to be a vulnerability or to be judged", then the model pre-screening result is "the model is to be judged".

[0055] In some exemplary embodiments, if the model is determined to be non-vulnerable in the first judgment result, the corresponding submission record is excluded and no subsequent second code review model verification is performed.

[0056] In some exemplary embodiments, the second code review model only receives real code differences and commit context, and does not receive candidate general defect enumerations, so as to avoid the model being anchored by a small number of candidate general defect enumerations; during the review process of the second code review model, rank, repository, fix_commit before fix and parent_commit after fix are bound to prevent the model output from being misaligned with the original commit records.

[0057] In some exemplary embodiments, the candidate general defect enumeration is a general defect enumeration predicted by the code review model.

[0058] In some exemplary embodiments, leaving blank means leaving the corresponding commit record blank as a whole.

[0059] In some exemplary embodiments, consistency review is an automatic review operation based on rules. For example, if the title, submission information, etc. of a high-confidence vulnerability candidate include any of the parameters in the judgment parameters, then the results are combined according to the preset weight and preset score of the corresponding parameter. If the combined score is not lower than the preset review score threshold, then the first review result is a confirmed vulnerability; otherwise, the corresponding submission record or process is skipped.

[0060] In some exemplary embodiments, the model pre-screening result is the submission record of the model_needs to be judged. The system retains the candidate CWE, vulnerability type, judgment reason, and disagreement information of the two models and enters them into the consistency review queue. The system focuses on checking submission records with inconsistent candidate CWEs, mismatch between model reasoning and code evidence, submissions involving authorization authentication or external input entry points, submissions involving plugin execution or sandbox boundaries, and submissions with small number of lines of modification (below a preset threshold) but significant security impact (security impact not lower than a preset security impact threshold). After consistency review, if it is confirmed as a vulnerability, the vulnerability event record, vulnerability number, and vulnerability type are filled in; otherwise, the corresponding submission record is skipped or left blank.

[0061] In some exemplary embodiments, this step generates the following fields: first_model_label: The judgment result output by the first code review model; first_model_cwe_ids: Candidate CWEs output by the first code review model; first_model_vul_type: The type of candidate vulnerability output by the first code review model; first_model_reason: The reason for the first code review of the model; second_model_label: The judgment result output by the second code review model; second_model_cwe_ids: Candidate CWEs output by the second code review model; second_model_vul_type: The type of candidate vulnerability output by the second code review model; second_model_reason: The reason for the second code review model's judgment; model_review_status: Pre-screening results of dual models (first code review model and second code review model), including model confirmed vulnerabilities model_confirmed_vul, model pending judgment model_needs, blank or model rejected model_rejected; model_disagreement_reason: The reason for the disagreement when two models disagree; manual_label: Whether the consistency review result confirms the vulnerability; if the submission is confirmed as a vulnerability, fill in the vulnerability event record confirmed_vul; if it is not a vulnerability or the evidence is insufficient, fill in skip or leave blank; manual_cwe_ids: CWE numbers determined by the consistency review, such as CWE-89, CWE-863, CWE-434, etc.; manual_vul_type: The vulnerability type determined by the consistency review, such as SQL Injection, IncorrectAuthorization, Unrestricted File Upload, Sandbox Escape, Command Injection, etc. Review notes: Consistency review notes, model divergence notes, and code evidence notes.

[0062] Step S104: Based on the vulnerability event record, by locating the version before and after the fix, obtain the code differences between the versions, and based on the code differences, determine the code before and after the fix from the version before and after the fix.

[0063] In some exemplary embodiments, the code before and after the repair includes code before the repair and code after the repair; The process of obtaining code differences between versions based on the vulnerability event records by locating the pre-fix and post-fix versions, and determining the code before and after the fix based on the code differences, includes: Based on the vulnerability event records, the pre-fix version and the post-fix version are located through a preset code location strategy. The code location strategy includes syntax tree parsing, function boundary identification based on abstract syntax tree, function signature matching based on language-specific rules, and function boundary expansion based on difference block context. Obtain the modified files, line numbers, difference blocks, change types, and file extensions between the pre-fix version and the post-fix version, and filter out non-source code files while retaining source code files; Based on the source code file, the modified line numbers, and the difference blocks, determine the affected functions and their location information; Based on the location information, the code of the affected function is extracted from the pre-fix version and the post-fix version, respectively, and used as the pre-fix code and the post-fix code.

[0064] In some exemplary embodiments, the code differences include the modified file, the modified line number, the difference block, the change type, and the file extension.

[0065] In some exemplary embodiments, when locating the pre-fix version and the post-fix version, the security fixes that are directly related to the differences in the actual code are retained, and it is determined whether the commit records corresponding to the located pre-fix version and post-fix version have fixed high-risk issues in the artificial intelligence infrastructure, such as unauthorized access to model interfaces, plugin execution risks, command execution risks, arbitrary file access, upload path bypass, sandbox isolation defects, token leakage, missing permission verification, or external input processing defects.

[0066] Step S105: Align the code before and after the repair to generate patch sample pairs.

[0067] In some exemplary embodiments, the step of aligning the code before and after the repair to generate patch sample pairs includes: Based on code name, code signature, file path, calling context, and code similarity, sample alignment is performed between the code before and the code after repair. Based on the sample alignment results, the internal vulnerability numbers of the vulnerability events are associated to generate patch sample pairs. The patch sample pairs include internal vulnerability number, project group, repository name, commit after fix, commit before fix, source code file path, function code before fix, function code after fix, function name, function signature, change line number, and confirmed vulnerability tag.

[0068] In some exemplary embodiments, the internal vulnerability number is used to associate vulnerability events; events with the same number can be associated.

[0069] In some exemplary embodiments, code similarity is determined based on an Abstract Syntax Tree (AST).

[0070] In some exemplary embodiments, a patch sample pair is only included in the formal software vulnerability benchmark as a confirmed vulnerability sample when the commit record has been pre-screened by the code review model and confirmed as a vulnerability fix by the consistency review.

[0071] Step S106: Perform quality verification on the vulnerability event record and the patch sample pair, and output the software vulnerability benchmark.

[0072] In some exemplary embodiments, the quality verification includes at least one of the following verifications, which are only partial verifications: If the vulnerability event is correctly associated with the vulnerability event, the quality verification passes. If both the pre-fix and post-fix commits are complete and traceable, the quality check passes. If both the function code before and after the repair are not empty, the quality check passes. If the function signature is not empty, the quality check passes. If the tags of the function code before and after the repair are the same, the quality check passes. If a consistency check is performed, the quality verification passes. If the candidate CWE conforms to the numbering format and comes from the consistency review, the quality check passes; If the candidate CWE output by the first code review model and the second code review model are consistent, then the quality check passes. If no duplicate samples are found, the quality check passes.

[0073] In some exemplary embodiments, after all quality checks are passed, a software vulnerability benchmark for artificial intelligence infrastructure is output. The software vulnerability benchmark includes submitted traceable information, before-and-after sample pairs, security semantic feature mining, code review model pre-screening, and consistency verification. It can be used for AI-Coding security capability assessment, artificial intelligence infrastructure vulnerability detection model testing, code audit model evaluation, and software security dataset construction.

[0074] To implement the exemplary embodiments described above, this disclosure also proposes a software vulnerability benchmark construction apparatus.

[0075] Figure 2 This is a schematic diagram of a software vulnerability benchmark building apparatus provided as an exemplary embodiment of the present disclosure. The apparatus 200 can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 2 As shown, the device 200 includes: an acquisition module 201, a first generation module 202, a second generation module 203, a determination module 204, a third generation module 205, and an output module 206, wherein, Module 201 is used to retrieve the commit history of the AI ​​infrastructure code repository; The first generation module 202 is used to perform security semantic feature mining on the submission records, generate a security semantic feature set, calculate the security relevance score of each submission record relative to the security semantic feature set, and generate a candidate queue. The second generation module 203 is used to perform code review model pre-screening and consistency verification on the submission records in the candidate queue, and generate vulnerability event records. The determination module 204 is used to obtain the code differences between the versions based on the vulnerability event record by locating the version before and after the fix, and to determine the code before and after the fix from the version before and after the fix based on the code differences; The third generation module 205 is used to perform sample alignment on the code before and after the repair to generate patch sample pairs; The output module 206 is used to perform quality verification on the vulnerability event record and the patch sample pair, and output the software vulnerability benchmark.

[0076] In some exemplary embodiments, the commit record includes a commit title, commit information, file path, code differences, function name, class name, method name, variable name, import dependencies, API call, and configuration item name; The security semantic feature set includes a strong security semantic feature set and a weak security semantic feature set; The first generation module 202 is specifically used for: The submission records are normalized to obtain candidate feature units; Security semantic feature mining is performed on the candidate feature units, and a security semantic feature score is calculated. The security semantic feature score includes semantic association score, repair intention score, code evidence score, statistical differentiation score, and stability score. The security semantic feature scores are weighted to obtain the comprehensive score of the candidate feature units, and then sorted based on the comprehensive score. In response to the stability score being not lower than a preset score threshold and the sorting being within a preset first interval, a set of strong security semantic features is generated. If the stability score is not lower than a preset score threshold and the sorting is in a preset second interval, then a set of weak security semantic features is generated.

[0077] In some exemplary embodiments, the first generation module 202 is specifically used for: Based on the security semantic feature set, a security relevance score is calculated for each submission record. The security relevance score includes semantic security feature hit score, code evidence location weight, repair intent weight, artificial intelligence infrastructure risk weight, and noise penalty item. Based on the security relevance score, the submission records are divided into strong security category, weak security category, and rejection category; Based on the security relevance scores in descending order, the submission records corresponding to the strong security category and the weak security category are sorted to generate a candidate queue, which includes a full candidate queue, a model pre-screening queue, and a review queue.

[0078] In some exemplary embodiments, the code review model includes a first code review model and a second code review model; The code review model pre-screening includes an initial review by a first code review model and a review by a second code review model. The second generation module 203 is specifically used for: Based on the local code repository mirror, obtain the actual code differences of the commit records in the candidate queue; The actual code differences are input into the first code review model to perform a preliminary review by the first code review model, and a first judgment result is output. The first judgment result includes the model being determined to be a vulnerability, pending judgment, and the model being determined to be a non-vulnerable. If the first judgment result is that the model is determined to be a vulnerability or to be judged, then the corresponding real code difference is input into the second code review model for second code review model verification, and the second judgment result is output. The second judgment result includes the model being determined to be a vulnerability, to be judged, and the model being determined to be a non-vulnerable. If the first judgment result indicates that the model is not a vulnerability, then the corresponding submission record will be removed; Based on the first judgment result and the second judgment result, a model pre-screening result is generated, which includes model identification vulnerabilities, model pending judgment, blank or model rejection; If the model pre-screening result is a vulnerability identified by the model, the corresponding submission record is taken as a high-confidence vulnerability candidate. The high-confidence vulnerability candidate is subjected to consistency review based on preset judgment parameters to obtain the first review result. The judgment parameters include title, submission information, source code file, actual code difference, candidate general defect enumeration output by the code review model, vulnerability type and judgment reason. If the first review result is a confirmed vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is that the model needs to be judged, then based on the candidate general defect enumeration, vulnerability type, judgment reason and disagreement content output by the code review model, a consistency review is performed to obtain a second review result; If the second review result confirms a vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is blank or the model is rejected, no vulnerability event record will be generated.

[0079] In some exemplary embodiments, the second generation module 203 includes: If the output of the first code review model indicates that the model is not a vulnerability, then the model pre-screening result is blank or the model is rejected. If the output of the first code review model is that the model has identified a vulnerability, the output of the second code review model is that the model has identified a vulnerability, and the candidate general defect enumerations output by the two models are consistent, then the model pre-screening result is that the model has identified a vulnerability. If the first code review model outputs a vulnerability, the second code review model outputs a pending judgment or a non-vulnerability, or the candidate general defect enumerations output by the two models are inconsistent, then the model pre-screening result is pending judgment. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model determines that it is not a vulnerability", then the model pre-screening result is blank or the model rejects it. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model is determined to be a vulnerability or to be judged", then the model pre-screening result is "the model is to be judged".

[0080] In some exemplary embodiments, the code before and after the repair includes code before the repair and code after the repair; The determining module 204 is specifically used for: Based on the vulnerability event records, the pre-fix version and the post-fix version are located through a preset code location strategy. The code location strategy includes syntax tree parsing, function boundary identification based on abstract syntax tree, function signature matching based on language-specific rules, and function boundary expansion based on difference block context. Obtain the modified files, line numbers, difference blocks, change types, and file extensions between the pre-fix version and the post-fix version, and filter out non-source code files while retaining source code files; Based on the source code file, the modified line numbers, and the difference blocks, determine the affected functions and their location information; Based on the location information, the code of the affected function is extracted from the pre-fix version and the post-fix version, respectively, and used as the pre-fix code and the post-fix code.

[0081] In some exemplary embodiments, the third generation module 205 is specifically used for: Based on code name, code signature, file path, calling context, and code similarity, sample alignment is performed between the code before and the code after repair. Based on the sample alignment results, the internal vulnerability numbers of the vulnerability events are associated to generate patch sample pairs. The patch sample pairs include internal vulnerability number, project group, repository name, commit after fix, commit before fix, source code file path, function code before fix, function code after fix, function name, function signature, change line number, and confirmed vulnerability tag.

[0082] The software vulnerability benchmark construction apparatus provided in the exemplary embodiments of this disclosure can execute the software vulnerability benchmark construction method provided in any exemplary embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] To implement the exemplary embodiments described above, this disclosure also proposes a computer program product, including a computer program / instruction that, when executed by a processor, implements the software vulnerability benchmark construction method in the exemplary embodiments described above.

[0084] Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure.

[0085] The following is a detailed reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device 300 in the exemplary embodiments of this disclosure. The electronic device 300 in the exemplary embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the exemplary embodiments disclosed herein.

[0086] like Figure 3 As shown, the electronic device 300 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a memory 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0087] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0088] In particular, according to exemplary embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, exemplary embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such exemplary embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from memory 308, or installed from ROM 302. When the computer program is executed by processor 301, it performs the functions defined in the software vulnerability benchmark construction method of exemplary embodiments of the present disclosure.

[0089] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0090] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0091] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0092] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned software vulnerability benchmark construction method.

[0093] Electronic devices can be programmed with computer program code in one or more programming languages ​​or combinations thereof to perform the operations of this disclosure. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The units described in the exemplary embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0096] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] The above description is merely a preferred exemplary embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0099] Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual exemplary embodiments may also be implemented in combination in a single exemplary embodiment. Conversely, various features described in the context of a single exemplary embodiment may also be implemented individually or in any suitable sub-combination in multiple exemplary embodiments.

[0100] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for constructing a software vulnerability benchmark, characterized in that, include: Retrieve the commit history of the AI ​​infrastructure code repository; Security semantic feature mining is performed on the submission records to generate a security semantic feature set. The security relevance score of each submission record relative to the security semantic feature set is calculated to generate a candidate queue. The submission records in the candidate queue are pre-screened by a code review model and undergo consistency verification to generate vulnerability event records; Based on the vulnerability event records, by locating the pre-fix version and the post-fix version, the code differences between the versions are obtained, and based on the code differences, the code before and after the fix is ​​determined from the pre-fix version and the post-fix version; The code before and after the repair is aligned to generate patch sample pairs; The vulnerability event records and patch samples are subjected to quality verification, and a software vulnerability benchmark is output.

2. The method according to claim 1, characterized in that, The code review model includes a first code review model and a second code review model; The code review model pre-screening includes an initial review by a first code review model and a review by a second code review model. The step of performing code review model pre-screening and consistency verification on the submission records in the candidate queue, and generating vulnerability event records, includes: Based on the local code repository mirror, obtain the actual code differences of the commit records in the candidate queue; The actual code differences are input into the first code review model to perform a preliminary review by the first code review model, and a first judgment result is output. The first judgment result includes the model being determined to be a vulnerability, pending judgment, and the model being determined to be a non-vulnerable. If the first judgment result is that the model is determined to be a vulnerability or to be judged, then the corresponding real code difference is input into the second code review model for second code review model verification, and the second judgment result is output. The second judgment result includes the model being determined to be a vulnerability, to be judged, and the model being determined to be a non-vulnerable. If the first judgment result indicates that the model is not a vulnerability, then the corresponding submission record will be removed; Based on the first judgment result and the second judgment result, a model pre-screening result is generated, which includes model identification vulnerabilities, model pending judgment, blank or model rejection; If the model pre-screening result is a vulnerability identified by the model, the corresponding submission record is taken as a high-confidence vulnerability candidate. The high-confidence vulnerability candidate is subjected to consistency review based on preset judgment parameters to obtain the first review result. The judgment parameters include title, submission information, source code file, actual code difference, candidate general defect enumeration output by the code review model, vulnerability type and judgment reason. If the first review result is a confirmed vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is that the model needs to be judged, then based on the candidate general defect enumeration, vulnerability type, judgment reason and disagreement content output by the code review model, a consistency review is performed to obtain a second review result; If the second review result confirms a vulnerability, a vulnerability event record is generated and a vulnerability number and vulnerability type are assigned; otherwise, the corresponding submission record is skipped or left blank. If the model pre-screening result is blank or the model is rejected, no vulnerability event record will be generated.

3. The method according to claim 2, characterized in that, Based on the first judgment result and the second judgment result, a model pre-screening result is generated. The model pre-screening result includes model identified as having vulnerabilities, model pending judgment, blank, or model rejected, including: If the output of the first code review model indicates that the model is not a vulnerability, then the model pre-screening result is blank or the model is rejected. If the output of the first code review model is that the model has identified a vulnerability, the output of the second code review model is that the model has identified a vulnerability, and the candidate general defect enumerations output by the two models are consistent, then the model pre-screening result is that the model has identified a vulnerability. If the first code review model outputs a vulnerability, the second code review model outputs a pending judgment or a non-vulnerability, or the candidate general defect enumerations output by the two models are inconsistent, then the model pre-screening result is pending judgment. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model determines that it is not a vulnerability", then the model pre-screening result is blank or the model rejects it. If the output of the first code review model is "to be judged" and the output of the second code review model is "the model is determined to be a vulnerability or to be judged", then the model pre-screening result is "the model is to be judged".

4. The method according to claim 1, characterized in that, The code before and after the repair includes the code before the repair and the code after the repair. The process of obtaining code differences between versions based on the vulnerability event records by locating the pre-fix and post-fix versions, and determining the code before and after the fix based on the code differences, includes: Based on the vulnerability event records, the pre-fix version and the post-fix version are located through a preset code location strategy. The code location strategy includes syntax tree parsing, function boundary identification based on abstract syntax tree, function signature matching based on language-specific rules, and function boundary expansion based on difference block context. Obtain the modified files, line numbers, difference blocks, change types, and file extensions between the pre-fix version and the post-fix version, and filter out non-source code files while retaining source code files; Based on the source code file, the modified line numbers, and the difference blocks, determine the affected functions and their location information; Based on the location information, the code of the affected function is extracted from the pre-fix version and the post-fix version, respectively, and used as the pre-fix code and the post-fix code.

5. The method according to claim 1, characterized in that, The code before and after the fix is ​​aligned to generate patch sample pairs, including: Based on code name, code signature, file path, calling context, and code similarity, sample alignment is performed between the code before and the code after repair. Based on the sample alignment results, the internal vulnerability numbers of the vulnerability events are associated to generate patch sample pairs. The patch sample pairs include internal vulnerability number, project group, repository name, commit after fix, commit before fix, source code file path, function code before fix, function code after fix, function name, function signature, change line number, and confirmed vulnerability tag.

6. The method according to claim 1, characterized in that, The commit record includes the commit title, commit information, file path, code differences, function name, class name, method name, variable name, import dependencies, API call, and configuration item name; The security semantic feature set includes a strong security semantic feature set and a weak security semantic feature set; The step of performing security semantic feature mining on the submitted records to generate a security semantic feature set includes: The submission records are normalized to obtain candidate feature units; Security semantic feature mining is performed on the candidate feature units, and a security semantic feature score is calculated. The security semantic feature score includes semantic association score, repair intention score, code evidence score, statistical differentiation score, and stability score. The security semantic feature scores are weighted to obtain the comprehensive score of the candidate feature units, and then sorted based on the comprehensive score. In response to the stability score being not lower than a preset score threshold and the sorting being within a preset first interval, a set of strong security semantic features is generated. If the stability score is not lower than a preset score threshold and the sorting is in a preset second interval, then a set of weak security semantic features is generated.

7. The method according to claim 1, characterized in that, The calculation of the security relevance score of each submitted record relative to the security semantic feature set, and the generation of a candidate queue, includes: Based on the security semantic feature set, a security relevance score is calculated for each submission record. The security relevance score includes semantic security feature hit score, code evidence location weight, repair intent weight, artificial intelligence infrastructure risk weight, and noise penalty item. Based on the security relevance score, the submission records are divided into strong security category, weak security category, and rejection category; Based on the security relevance scores in descending order, the submission records corresponding to the strong security category and the weak security category are sorted to generate a candidate queue, which includes a full candidate queue, a model pre-screening queue, and a review queue.

8. A software vulnerability benchmark construction apparatus, the apparatus comprising: The acquisition module is used to retrieve the commit history of the AI ​​infrastructure code repository; The first generation module is used to perform security semantic feature mining on the submission records, generate a security semantic feature set, calculate the security relevance score of each submission record relative to the security semantic feature set, and generate a candidate queue. The second generation module is used to perform code review model pre-screening and consistency verification on the submission records in the candidate queue, and generate vulnerability event records. The determination module is used to obtain the code differences between the versions based on the vulnerability event records by locating the version before and after the fix, and to determine the code before and after the fix from the version before and after the fix based on the code differences; The third generation module is used to perform sample alignment on the code before and after the repair to generate patch sample pairs; The output module is used to perform quality verification on the vulnerability event records and the patch sample pairs, and output a software vulnerability benchmark.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images