A method and system for screening patches for large language models

Abstract

This invention relates to a referenceless screening method and system for generating patches based on large language models, belonging to the fields of software security and software engineering technology. Specifically, it includes taking the vulnerability source code and its corresponding large language model as input to generate candidate patches. It does not rely on any manually written reference patches, dynamic test cases, or program semantic execution results, but only on the abstract syntax tree structure change information between the candidate patches and the original vulnerability code. It performs multi-stage cascading screening of candidate patches, and adaptively adjusts the screening strategy according to the vulnerability type. Finally, it outputs a queue of candidate patches suitable for subsequent manual review or semantic verification stages. This invention can be widely applied to application scenarios such as automatic software vulnerability repair, intelligent code review, software supply chain security, continuous integration and continuous deployment pipelines, code security auditing, and large language model-assisted software development.

Description

Technical Field

[0001] This invention relates to a referenceless filtering method and system for generating patches for large language models, belonging to the fields of software security and software engineering technology. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) in areas such as source code understanding, code generation, and automated program repair, automatically generating software vulnerability patches using LLMs has become an important direction in software security research. Compared to traditional automated program repair methods based on template matching, search optimization, or rule reasoning, LLMs can combine code context to generate more flexible and diverse repair solutions, thereby significantly improving vulnerability repair efficiency. They are increasingly being applied to practical software engineering scenarios such as software development, code review, and continuous integration / continuous deployment (CI / CD).

[0003] Currently, the effectiveness evaluation of automatically generated patches mainly employs techniques such as reference patch comparison, dynamic testing verification, and static analysis. The reference patch comparison method typically uses the standard fix patch (GoldCode) written by developers as the evaluation criterion, judging patch quality by calculating the similarity between the generated patch and the reference patch. Dynamic testing verification relies on test cases, regression testing, or symbolic execution techniques to verify whether the patch can correctly fix vulnerabilities without introducing new defects. Static analysis methods perform static security checks on the patch through program syntax, control flow, or data flow analysis.

[0004] However, the aforementioned methods are primarily applicable to offline evaluation or experimental environments, and have certain limitations in actual software deployment. For newly discovered software vulnerabilities, a trusted reference patch is usually not available when the patch enters the continuous integration pipeline, thus making it impossible to adopt an evaluation method that relies on a reference patch. Furthermore, methods such as dynamic testing, symbolic execution, and manual code review consume significant computational resources and manpower, making it difficult to meet the real-time screening requirements of massive candidate patches in industrial scenarios. When a large language model generates multiple candidate patches at once, if all of them enter the subsequent semantic verification or manual review stage, it will not only significantly increase the length of the review queue but also easily reduce the overall vulnerability remediation efficiency.

[0005] Furthermore, security patches generated by large language models typically suffer from two types of problems. One type involves patches that excessively modify the original program, introducing extensive code refactoring unrelated to vulnerability remediation, leading to significant changes in program structure. This not only increases the difficulty of manual review but may also introduce new security risks. The other type involves patches that only make superficial code modifications, maintaining high code similarity but failing to eliminate underlying vulnerabilities and thus failing to achieve true security remediation. Existing pre-screening methods, without the need for reference patches and dynamic testing, struggle to simultaneously suppress candidate patches with excessive structural shifts and those exhibiting near-identical superficial modifications at a low cost.

[0006] Therefore, how to achieve this without a trusted reference patch, without performing dynamic tests, and without requiring a first phase? Under complex semantic analysis conditions, relying solely on the structural relationship between the original vulnerable program and candidate patches, the large language model... The generated security patches undergo low-cost pre-queue shaping, prioritizing the exclusion of those with intractable syntax, excessive structural deviations, or other issues. Candidate patches with a risk of near-identical cavity repair are selected to reduce the number of candidates entering the subsequent high-cost review process. This addresses the technical problems that need to be solved in the fields of automated program repair and software security. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention provides a referenceless screening method and system for generating patches based on large language models. This method takes the vulnerability source code and the corresponding candidate patches generated by the large language model as input, without relying on any manually written reference patches, dynamic test cases, or program semantic execution results. It only performs multi-stage cascaded screening of candidate patches based on the abstract syntax tree structure change information between the candidate patches and the original vulnerability code, and adaptively adjusts the screening strategy according to the vulnerability type. Finally, it outputs a queue of candidate patches suitable for subsequent manual review or semantic verification stages.

[0008] To achieve the above objectives, the technical solution adopted in this invention is a referenceless filtering method for generating patches for large language models, comprising the following steps: S1. Obtain the vulnerability program to be patched and the candidate security patches generated by the large language model, perform syntactic integrity checks on the candidate patches, and filter out candidate patches that cannot form a valid abstract syntax tree. S2. Construct abstract syntax trees for the original vulnerable program and the candidate patches that have passed the syntax integrity test after detection, and extract the structural change information between the two to form structural change features that describe the degree of patch modification. S3. Based on the degree of change in the abstract syntax tree of the candidate patches, establish a lower bound constraint on structural changes and perform anti-semantic drift filtering on the candidate patches. S4. Establish upper bound constraints on structural changes for the selected candidate patches and identify void repair patches with insufficient structural changes. S5. Based on the common vulnerability enumeration category to which the candidate patch belongs, conditionally adjust the reasonable structural change range; S6. Retain all candidate patches that meet the screening criteria in the review queue and output them to the subsequent manual code review, large language model semantic analysis or dynamic verification module.

[0009] Preferably, in step S1, the logic for determining grammatical integrity is as follows: define an abstract syntax tree construction function f. AST (·), candidate patch P i The corresponding abstract syntax tree is T. i =f AST (P i ); If T i It is non-empty and T i If it can be parsed effectively, then the syntax determination function G syn (P i If T = 1, proceed to the subsequent structural analysis process; if T i If G is empty or cannot be resolved effectively, then syn (P i If )=0, stop performing subsequent steps on the candidate patch.

[0010] Preferably, in step S2, when constructing the abstract syntax tree, the structure change vector is extracted by counting the number of newly added nodes, deleted nodes, modified node attributes, and adjusted node positions between two abstract syntax trees, and the normalized structure change rate and structure retention rate are calculated.

[0011] Preferably, when extracting the structural change vector, the structural change vector d i =[n ins ,n del ,n upd ,n mov ], Where, n ins n represents the number of new nodes. del n is the number of nodes to be deleted. upd The number of times a node attribute is modified, n mov Adjust the number of nodes at their positions; The normalized structural change rate R i =(n ins +n del +n upd +n mov ) / |V o | , where |V o |The total number of abstract syntax tree nodes for the original vulnerable program; The structural retention rate S i =1-R i , among which, S iThe closer the value is to 1, the higher the structural consistency between the candidate patch and the original vulnerable program; conversely, the lower the value is, the more extensive the modifications made to the program.

[0012] Preferably, in step S3, the rule for anti-semantic drift filtering is set as follows: the lower bound threshold of the structure retention rate is set to τ. L Only retain those that satisfy the candidate patch structure retention rate S i ≥τ L Candidate patches, if S i <τ L If so, the patch is determined to have a semantic drift risk and is filtered out.

[0013] Preferably, in step S4, the void repair identification rule is: setting an upper limit threshold τ for the structure retention rate. U Only retain those that meet the structural retention rate S i ≤τ U Candidate patches; if S i >τ U If so, it is identified as a hole repair patch and filtered out.

[0014] Preferably, in step S5, by obtaining the CWE vulnerability category corresponding to the vulnerability, the upper limit threshold of the structure retention rate is adaptively adjusted according to different vulnerability types to form a reasonable structure change range for the corresponding vulnerability category, and only candidate patches whose structure retention rate falls within this range are retained.

[0015] Preferably, the adaptive adjustment process includes: pre-establishing a mapping relationship Φ(c) between CWE vulnerability categories and structural change ranges. i )=[τ L (c i ),τ U (c i )], Among them, c i τ represents the vulnerability type to which the candidate patch belongs. L (c i ) represents vulnerability category c i The corresponding lower bound of structural change; τ U (c i ) represents vulnerability category c i The corresponding upper bound of the structure retention rate; For minor, localized fixes, the timeframe should be narrowed; for complex vulnerabilities requiring extensive logic adjustments, the timeframe should be broadened. For vulnerabilities satisfying τ... L (c i )≤S i ≤τ U (c i The patch will be retained.

[0016] A filtering system for a no-reference filtering method for generating patches for large language models, comprising: The patch acquisition module is used to connect to the code hosting platform and LLM inference service to obtain the original vulnerability program and all candidate patches generated by LLM, and build a candidate patch set. The syntax detection module is used to call the corresponding programming language parser to complete the construction and verification of the abstract syntax tree and filter out syntax invalid patches. The abstract syntax tree analysis module is used to construct abstract syntax trees for the original program and candidate patches respectively, and establish node mapping relationships. The structural change calculation module is used to count the number of abstract syntax tree editing operations of candidate patches relative to the original vulnerability program, and to calculate the normalized structural change rate and structural retention rate. The vulnerability type analysis module is used to read the vulnerability CWE category, load the corresponding adaptive structural change range parameters, and send them to the queue shaping module; The candidate patch queue shaping module is used to chain syntax validation, double boundary structure constraints, and vulnerability type adaptive interval triple filtering rules to eliminate semantic drift patches, hole repair patches, and syntax error patches. The results output module is used to output the queue of optimized candidate patches after screening, and connects to the manual review, large language model semantic analysis, or dynamic verification module and automatic deployment subsystem.

[0017] Preferably, the system is used as a pre-screening module and deployed in continuous integration and continuous deployment pipelines, automated program repair platforms, intelligent code review platforms, and software supply chain security analysis systems.

[0018] Compared with the prior art, the present invention has the following technical effects: 1) This invention can complete the pre-queue shaping of candidate patches without relying on manually written reference patches. It can quickly evaluate patch quality by utilizing only the abstract syntax tree structure change information between the original vulnerability program and the candidate patches. Therefore, this invention can be applied to new vulnerability repair scenarios where the reference patch has not yet been disclosed or generated, breaking through the dependence of existing patch evaluation methods on reference patches and improving the applicability of automatic program repair technology in real software development environments.

[0019] 2) This invention eliminates the need for dynamic testing, symbolic execution, or complex program semantic analysis during the pre-screening phase. It achieves rapid filtering of candidate patches solely through abstract syntax tree structure change modeling. Compared to traditional patch verification methods that rely on test cases or manual review, this invention effectively reduces computational resource consumption and manual review costs, shortens patch screening time, and improves the overall processing efficiency of continuous integration and continuous deployment pipelines.

[0020] 3) This invention establishes a dual-boundary screening mechanism based on program structure changes, restricting candidate patches to a reasonable range of structural modifications. On the one hand, it suppresses semantically drifting patches caused by excessive generation of large language models through lower bound constraints on structural changes; on the other hand, it identifies hollow patch repairs that only make superficial modifications and do not truly fix vulnerabilities through upper bound constraints on structural preservation rates. The above dual-boundary constraints can simultaneously cover two typical failure modes in the candidate patch screening process, improving the overall quality of candidate patches entering the subsequent verification stage.

[0021] 4) This invention combines the common vulnerability enumeration category to which the vulnerability belongs to construct a reasonable structural change range that adapts to the vulnerability type, and dynamically adjusts the allowed structural change range of candidate patches according to different vulnerability repair modes. Compared with a screening strategy using a fixed threshold, this invention can take into account the characteristics of different vulnerability types such as local repair and complex repair, reduce the probability of falsely filtering valid patches, and improve the generalization ability and applicability of the patch screening method.

[0022] 5) The referenceless queue shaping mechanism proposed in this invention can serve as a pre-screening module in an automated program repair system, working in conjunction with subsequent verification modules such as manual code review, dynamic testing, large language model semantic analysis, and automated deployment. By completing rapid pre-screening before patches enter the high-cost verification process, this invention can effectively reduce invalid candidate patches entering the subsequent review queue, optimize patch verification resource allocation, and improve the processing efficiency and system throughput of the entire vulnerability repair pipeline.

[0023] 6) This invention uses abstract syntax tree (AST) structure changes as a unified representation of program modifications, exhibiting good language independence and scalability. For different programming languages, adaptation can be achieved simply by replacing the corresponding AST parser, without altering the overall filtering process. Therefore, it can be widely applied to software vulnerability patching tasks in various programming languages ​​and has significant engineering deployment value and promising prospects for widespread application.

[0024] In summary, this invention proposes a referenceless security patch queue shaping method and system that requires no reference patch, no dynamic verification, and can adaptively analyze structural changes based on vulnerability types. While ensuring patch screening efficiency, it improves the quality of candidate patches, reduces the verification cost of automatic program repair systems, and provides an efficient, lightweight, and easy-to-deploy engineering solution for large language model-assisted software vulnerability repair. Detailed Implementation

[0025] To make the technical problems, solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] A no-reference filtering method for generating patches for large language models includes the following six steps: candidate patch syntactic integrity detection, candidate patch abstract syntax tree structure change modeling, anti-semantic drift filtering based on the lower bound of the structure preservation rate, hole repair identification based on the upper bound of the structure preservation rate, and reasonable structure change range adjustment based on vulnerability type adaptation.

[0027] Step 1: Check the syntax integrity of candidate patches To prevent grammatically incorrect patches from entering the subsequent screening process, this invention first performs a syntax integrity check on each candidate security patch generated by the large language model. This step serves as the first admission mechanism in the entire candidate patch queue shaping process, filtering out invalid patches caused by code generation errors, missing context, or incomplete syntax, thereby improving the reliability of subsequent structural analysis results.

[0028] Let the vulnerability source code be represented as P. o ={s1,s2,···,s n}, where P o This indicates the original program code that contained the vulnerability, s i Let represent the i-th statement in the program. The large language model generates a candidate patch set P={P1,P2,...c,P...} for the vulnerable program. m}, where m represents the number of candidate patches and Pi represents the i-th candidate patch.

[0029] For each candidate patch, a syntax parser is used to perform syntax analysis on the patch code, and the syntax parsing function is defined as T. i =f AST (P i ), where f AST (·) represents the abstract syntax tree construction function, T i This represents the abstract syntax tree corresponding to the candidate patch. If a candidate patch can successfully complete the construction of the abstract syntax tree, it is considered to meet the syntactic integrity requirements; otherwise, it is considered to have a syntactic error and is directly removed from the candidate patch set.

[0030] Therefore, the syntax determination rule for candidate patches can be defined as follows: if T i If it is not empty, then the syntax evaluation function G syn (P i If T = 1, proceed to the next stage of abstract syntax tree structure analysis; if T i If it is empty, then G syn (P i If )=0, the candidate patch is directly removed.

[0031] Since subsequent structural change calculations depend on the node correspondence between abstract syntax trees, syntax integrity checks can effectively prevent structural analysis failures caused by grammatical error patches, reduce invalid candidate patches from entering the subsequent screening process, and improve the stability and efficiency of the entire no-reference candidate patch queue shaping method.

[0032] Step 2: Modeling changes in the abstract syntax tree structure of candidate patches All candidate patches that pass the syntax integrity check in step one possess a valid program syntax structure. To further describe the extent of modification of the candidate patches relative to the original vulnerable program, this invention uses an abstract syntax tree to uniformly represent the program structure and constructs a structural change model for the candidate patches, providing a basis for structural changes in subsequent patch selection.

[0033] Let T be the abstract syntax tree representation of the original vulnerable program. o =(V o E o ), where V o E represents the set of nodes in the abstract syntax tree of the original program. o This indicates the connection relationships between nodes.

[0034] Correspondingly, the candidate patch is constructed to obtain an abstract syntax tree T. i =(V i E i ), where V i E represents the set of abstract syntax tree nodes for candidate patches. i This represents the corresponding tree structure relationship. To quantify the structural changes that occur during patch modifications, this invention represents the process of converting the original abstract syntax tree into a candidate patch abstract syntax tree as a set of abstract syntax tree editing operations: ∆(T o ,T i )={o1,o2,···,o k}, where the abstract syntax tree structure variation represents the set of abstract syntax tree structure variations, o k This represents the k-th structural editing operation, which includes, but is not limited to, node insertion, node deletion, node update, and node movement.

[0035] Based on various editing operations, the structural change vector d corresponding to the candidate patches is statistically analyzed. i =[n ins ,n del ,n upd ,n mov ], Where, n ins n represents the number of new nodes. del n is the number of nodes to be deleted. upd The number of times a node attribute is modified, n movAdjust the number of nodes at their positions; To eliminate the impact of different program sizes, this invention further defines the normalized structural change rate as R. i =(n ins +n del +n upd +n mov ) / |V o |, where |V o | The total number of nodes in the abstract syntax tree of the original vulnerable program; normalized structure change rate R i Used to describe the degree of overall structural modification of a candidate patch relative to the original vulnerable program. When R i When the value is small, it indicates that the candidate patch mainly made local modifications; when R... i A larger value indicates that the candidate patch involves a large range of program structure adjustments.

[0036] In addition to the overall modification extent, this invention further calculates the structure retention rate between the original program and the candidate patch, which reflects the retention of the original program structure by the candidate patch. Let the structure retention rate be defined as S. i =1-R i , among which, S i The closer the value is to 1, the higher the structural consistency between the candidate patch and the original vulnerable program; conversely, the lower the value is, the more extensive the modifications made to the program.

[0037] By modeling structural changes as described above, candidate patches can be transformed into structural feature representations that reflect program modification behavior, providing a unified quantitative basis for subsequent dual-boundary screening based on the range of structural changes. The normalized structural change rate is primarily used to describe the scale of patch modifications, while the structural retention rate reflects the degree to which the patch preserves the original program structure. Together, they constitute the core structural evaluation indicators in the no-reference patch screening process.

[0038] Step 3: Anti-semantic drift filtering based on the lower bound of structure preservation rate After completing the modeling of candidate patch structural changes, this invention further uses the normalized structural change rate to perform anti-semantic drift screening on candidate patches.

[0039] In the process of generating security patches using large language models, some candidate patches, while passing syntactic parsing, can easily lead to extensive modifications to the original program due to content expansion, context shifts, or code refactoring during generation. This introduces numerous structural changes unrelated to vulnerability remediation. Such patches typically deviate from the vulnerability remediation objective, increasing the cost of subsequent manual review and potentially introducing new program defects. Therefore, this invention establishes a lower bound constraint on structural changes based on the degree of structural variation in candidate patches, enabling early filtering of candidate patches with significant semantic drift characteristics.

[0040] Let step two calculate the candidate patch P. i The corresponding structural retention rate is S i Define the lower bound threshold of the structure retention rate as τ. L , where τ L This represents the lower bound of the structure retention rate, used to identify semantic drift. Whether a candidate patch satisfies the anti-semantic drift constraint is defined as setting the lower bound threshold of the structure retention rate to τ. L Only retain those that satisfy the candidate patch structure retention rate S i ≥τ L Candidate patches will proceed to the next stage of screening if Si < τ. L If so, the patch is determined to have a semantic drift risk and is filtered out.

[0041] This invention uses the rate of structural change as the anti-semantic drift criterion because vulnerability repairs are typically characterized by clear objectives and limited scope of modification. For the vast majority of security vulnerabilities, repairs can be completed simply by modifying the vulnerability triggering conditions, adding boundary checks, improving input validation, or correcting resource management logic, without requiring extensive adjustments to the main program structure. When candidate patches involve adding, deleting, updating, or moving a large number of abstract syntax tree nodes, their modifications often exceed the scope required for vulnerability repair and have a high probability of being semantic drift patches.

[0042] Therefore, this invention utilizes the lower bound constraint of structural change to limit candidate patches to a reasonable range of structural modifications, thereby enabling rapid identification and filtering of semantically drifting patches and providing a first layer of structural constraints for subsequent patch screening.

[0043] Step 4: Void Repair Identification Based on Upper Bound of Structural Retention Rate After completing the anti-semantic drift screening, although the candidate patches can maintain a reasonable maximum range of structural changes, there may still be another type of failure patch, namely, patches that have only made very few or even formal code modifications without truly eliminating the cause of the vulnerability. Therefore, such patches are called hole fix patches.

[0044] Hollow patch fixes typically maintain a high rate of program structure preservation, with modifications limited to variable name replacements, comment adjustments, formatting changes, insertion of irrelevant statements, or other localized edits that do not alter the vulnerability triggering logic. While these patches maintain a high degree of structural consistency with the original vulnerable program, they do not actually achieve effective vulnerability repair. If they are directly submitted to subsequent manual review or dynamic verification processes, it will increase the proportion of invalid patches and reduce the overall processing efficiency of the review queue.

[0045] Therefore, the present invention further utilizes the upper bound threshold of the structure retention rate calculated in step two to establish an upper bound constraint on the structure retention rate of candidate patches, thereby identifying void repair patches with insufficient structural changes.

[0046] Let candidate patch P i The corresponding structural retention rate is S i =1-R i , where R i This represents the normalized structural change rate of the candidate patch.

[0047] Further set the upper limit threshold of the structure retention rate as τ U This is used to identify void repairs. The void repair identification rule for candidate patches is: only those patches that meet the structural retention rate S are retained. i ≤τ U The candidate patches will proceed to the next stage of screening; if S i >τ U If so, it is identified as a hole repair patch and filtered out.

[0048] Vulnerability remediation should lead to adjustments in the program structure corresponding to the cause of the vulnerability. When a candidate patch almost completely maintains the original program structure, it indicates that the patch's modification of the vulnerability-related syntax is insufficient. Although the overall program change is small, it still has a high probability of retaining the original vulnerability. Therefore, by using an upper bound constraint on the structure retention rate, candidate patches that only make superficial modifications and lack actual remediation capabilities can be effectively identified, improving the overall quality of patches entering the subsequent verification stage.

[0049] Step 5: Adjusting the reasonable structural change range based on vulnerability type adaptively After screening in steps three and four, this invention further adaptively adjusts the structural change constraints of candidate patches in conjunction with vulnerability types to improve the accuracy and applicability of patch screening strategies under different vulnerability scenarios.

[0050] Different types of security vulnerabilities require different remediation methods. For example, some vulnerabilities can be remedied simply by adding boundary checks, null value checks, or input validation, and their structural modifications are usually minor. Other vulnerabilities may involve resource release, access control, exception handling, or logical adjustments between multiple functions, and their remediation often requires extensive modifications to the program structure. Therefore, using a uniform threshold for structural changes can easily lead to the false acceptance of locally remediatable vulnerabilities and the false filtering of complex remediatable vulnerabilities, thus reducing the overall screening effectiveness.

[0051] To this end, this invention introduces vulnerability type information and adjusts the reasonable structural change range conditionally according to the Common Weakness Enumeration (CWE) category to which the candidate patch belongs.

[0052] Let the vulnerability category corresponding to the candidate patch be c. i ∈C, where C represents the set of vulnerability categories, c i This indicates the vulnerability type to which the candidate patch belongs. For different vulnerability categories, a corresponding adaptive interval function Φ(c) for structure preservation rate is established. i )=[τ L (c i ),τ U (c i )], where c i τ represents the vulnerability type to which the candidate patch belongs. L (c i ) represents vulnerability category c i The corresponding lower bound of structural change; τ U (c i ) represents vulnerability category c i The corresponding upper bound of the structure retention rate; For minor, localized fixes, the timeframe should be narrowed; for complex vulnerabilities requiring extensive logic adjustments, the timeframe should be broadened. For vulnerabilities satisfying τ... L (c i )≤S i ≤τ U (c i Patches that meet the structural modification criteria for the current vulnerability type will be retained and can proceed to the next review process if the candidate patch's scope of modification is deemed to match the remediation characteristics of the current vulnerability type.

[0053] Furthermore, an adaptive filtering function for vulnerability types can be defined as follows: , Among them, G type (P i )=1 indicates that the candidate patch meets the structural change requirements of the corresponding vulnerability type; G type (P i=0 indicates that the candidate patch does not meet the reasonable repair range for the corresponding vulnerability type.

[0054] Through the aforementioned vulnerability type adaptive adjustment mechanism, this invention can dynamically configure a reasonable structural change range for different vulnerability repair modes, enabling the patch screening process to take into account both local repair and complex repair scenarios, thereby improving the generalization ability and screening accuracy of the no-reference patch screening method across multiple vulnerability types.

[0055] Ultimately, only candidate patches that simultaneously satisfy syntactic integrity constraints, structural change constraints, and vulnerability type adaptive range constraints are retained to form an optimized security candidate patch queue, which is then output to subsequent manual review, dynamic testing, or other patch verification modules.

[0056] The no-reference security patch queue shaping method proposed in this invention can be deployed as a pre-screening module in the automatic software vulnerability repair process in continuous integration and continuous deployment (CI / CD) pipelines, automatic program repair platforms, intelligent code review platforms, and software supply chain security analysis systems to achieve automated screening and queue optimization of candidate patches generated by large language models.

[0057] To implement the aforementioned no-reference security patch queue shaping method, this invention also provides a no-reference patch screening system for security patches generated from large language models. The system can be deployed on servers, continuous integration / continuous deployment (CI / CD) platforms, code review platforms, or other software security analysis environments to automatically screen and optimize the queue of candidate patches generated from large language models.

[0058] The referenceless filtering system mainly includes a patch acquisition module, a syntax detection module, an abstract syntax tree analysis module, a structural change calculation module, a vulnerability type analysis module, a candidate patch queue shaping module, and a result output module. These modules are connected sequentially according to the candidate patch processing flow, and their overall workflow is as follows:

[0059] The patch acquisition module receives the vulnerability program to be patched and the corresponding large language model to generate candidate patches, and establishes a candidate patch set. This module can interface with code hosting platforms, vulnerability remediation platforms, or large language model inference services to achieve automatic acquisition of candidate patches.

[0060] The syntax checking module performs syntax integrity checks on candidate patches to determine whether a candidate patch can successfully construct an abstract syntax tree. For candidate patches with syntax errors, missing syntax, or that cannot be parsed, the subsequent processing is terminated, and the patch is removed from the candidate patch set.

[0061] The abstract syntax tree analysis module is used to construct abstract syntax trees corresponding to the original vulnerable program and candidate patches respectively, and establish the node mapping relationship between the two to provide a unified data representation for subsequent structural change analysis.

[0062] The structural change calculation module is used to statistically analyze the abstract syntax tree editing operations of candidate patches relative to the original vulnerable program, including structural changes such as node insertion, node deletion, node update, and node movement. It also calculates the normalized structural change rate and structural retention rate of the candidate patches, providing a quantitative basis for subsequent double-boundary screening.

[0063] The vulnerability type analysis module is used to obtain the Common WeaknessEnumeration (CWE) category to which the vulnerability to be patched belongs. Based on the repair mode corresponding to the vulnerability category, it adaptively determines the reasonable range of structural changes allowed for candidate patches and sends the corresponding parameters to the candidate patch queue shaping module.

[0064] The candidate patch queue shaping module is used to perform multi-stage cascading screening of candidate patches by comprehensively considering the results of syntax detection, structural change rate, structural retention rate, and reasonable structural change range corresponding to the vulnerability type. Candidate patches with semantic drift risk or hole repair characteristics are filtered, and only candidate patches that meet the screening criteria are retained for subsequent review.

[0065] The output module outputs a set of filtered candidate patches and organizes them into an optimized queue for subsequent review according to a preset order. This queue can then proceed to manual code review, large language model semantic analysis, compilation testing, dynamic testing, or other semantic verification processes to achieve subsequent verification and deployment of security patches.

[0066] The aforementioned functional modules can be deployed on the same computing device or distributed across multiple computing nodes, interacting with each other via a network. This invention does not limit the specific implementation method of each module; each module can be implemented using software programs, hardware circuits, or a combination of both.

[0067] To fully verify the effectiveness of the no-reference security patch queue shaping method proposed in this invention, an experimental environment was constructed based on real software vulnerability remediation tasks, and vulnerability samples from publicly available vulnerability datasets were used as experimental objects. The experimental objects covered different vulnerability types and remediation complexities to evaluate the queue shaping effect of this invention within the scope of the C / C++ security patch testing.

[0068] Experimental Environment: All experiments were conducted using a unified software and hardware environment. The experimental platform was deployed on a Linux operating system. The reference-free patch screening system proposed in this invention was implemented using Python, including functional modules such as candidate patch parsing, abstract syntax tree construction, structural change modeling, reasonable structural change interval analysis, and candidate patch queue shaping.

[0069] During the experiment, abstract syntax tree parsing was performed using an abstract syntax tree parsing tool that supports the target programming language. This embodiment verifies C / C++ patches; for other programming languages, a corresponding parser needs to be configured and the threshold applicability re-verified. Large language model inference, candidate patch selection, and experimental evaluation were all performed in a unified runtime environment, with consistent runtime configurations across all experimental groups to avoid the impact of environmental differences on the results.

[0070] Vulnerability Dataset: A publicly available vulnerability dataset containing real-world vulnerability fix records was used as the experimental subject. Each vulnerability sample includes the vulnerable program, vulnerability description, vulnerability category, and corresponding fix information. The vulnerability category is uniformly identified using Common Weakness Enumeration (CWE), and the vulnerability number is managed using Common Vulnerabilities and Exposures (CVE).

[0071] The experimental samples cover a variety of typical software security vulnerabilities, including but not limited to boundary violations, null pointer deductions, resource management errors, input validation defects, and access control errors, in order to verify the applicability of the present invention in different vulnerability remediation scenarios.

[0072] The main evaluation data used in the paper's experiments came from the BigVul dataset. Two hundred different vulnerable functions were randomly selected from BigVul using a fixed random seed of 42, and security patches were generated using nine large language models. This resulted in a main evaluation data pool containing 5000 valid generated patches. From these 5000 patches, 20% (1000 samples) was used as a validation set for threshold calibration; the remaining 80% (4000 samples) served as an unseen test set to evaluate the final queue shaping effect.

[0073] Candidate patch generation method: For each vulnerability sample in the experimental dataset, the vulnerable program, vulnerability description and related context information are first extracted and input into a large language model with code generation capabilities. The model then automatically generates multiple candidate security patches.

[0074] To ensure experimental consistency, candidate patches in the main evaluation data pool were generated from nine large language models. Seven of these models employed three suggestion configurations: direct repair, post-analysis repair, and multi-stage inference repair. For gpt-5.4 and claude-opus-4.5, no intermediate analysis suggestion configuration was used. This resulted in 5000 valid generated security patches. Subsequently, all candidate patches were input into the proposed no-reference patch filtering system for processing. The system sequentially performed syntactic integrity checks, abstract syntax tree structure change modeling, reasonable structure change range filtering, and adaptive vulnerability type configuration, ultimately forming an optimized candidate patch queue.

[0075] It should be noted that the screening stage of this invention does not directly rely on the internal parameters of the candidate patch generation model. This embodiment has been evaluated on candidate patches generated by various large language models; the applicability of other generation processes needs further verification with specific data.

[0076] Comparison of schemes and evaluation indicators: A comprehensive analysis of the candidate patch screening quality, vulnerability type adaptability, and engineering application effect is conducted using multiple evaluation indicators.

[0077] Comparison scheme: To ensure the fairness of the experimental results, all comparison methods use the same vulnerability dataset, the same large language model generation strategy, and the same set of candidate patches, with only the patch selection method differing.

[0078] The comparison method used in this embodiment includes: (1) an unfiltered candidate patch queue (OriginalQueue). This method directly uses all candidate patches generated by the large language model without any pre-screening, as a basic comparison scheme to verify the effectiveness of the present invention in the candidate patch preprocessing stage.

[0079] (2) Reference Patch-Based Screening Method. This type of method assesses quality by comparing candidate patches with manually written reference patches, and is a commonly used patch evaluation method in the field of existing automated program repair. This embodiment uses the reference patch evaluation method used in the paper as a representative comparison object to verify the screening capability of the present invention without the need for reference patches.

[0080] The experimental results are shown in Table 1: Table 1. Verification results of the overall screening effect of the present invention. Evaluation Project Unscreened Method of the present invention Experimental data scale 5000 patches 5000 patches Test set size 4000 patches 4000 patches Purity of candidate patch queue 63.18% 68.62% Number of low-quality candidate patches blocked – 766 Does it depend on Gold Code? yes no Table 2 Specific screening results of the present invention Evaluation Project Number of candidates to retain Queue purity Number of invalid candidates blocked Error filtering with misplaced candidate count Unscreened 4000 63.18% 0 0 LOC heuristic (≤20 lines) 1522 68.53% 994 1484 The method of this invention (CWE conditional cascade) 2188 76.97% 900 912

[0081] As shown in Table 1, this invention, without requiring a reference patch, effectively improves the quality of the candidate patch queue by filtering candidate patches solely based on the abstract syntax tree structure changes between the original vulnerable program and the candidate patches. Compared to an unfiltered candidate patch queue, this invention increases the purity of the candidate patch queue from 63.18% to 76.97%, while successfully intercepting 900 low-quality candidate patches without relying on a reference patch. The experimental results demonstrate that the proposed syntactic integrity detection, structural change modeling, and reasonable structural change interval filtering mechanism can effectively filter low-quality candidate patches with syntactic anomalies, structural drift, and hole repairs. While maintaining candidate patch coverage, it improves the overall quality of candidate patches entering the subsequent review process, providing a more reliable candidate patch queue for subsequent manual review and semantic verification.

[0082] As shown in Table 2, the purity of the unscreened queue was 63.18% on 4000 unseen test samples. The method of this invention retains 2188 candidate patches, increasing the purity of the retained queue to 76.21%, and intercepts 900 candidate patches marked as invalid by proxy reviewers before they enter manual or model review.

[0083] Compared to heuristic methods that modify no more than 20 lines, the method of this invention achieves slightly higher queue purity while retaining 572 more reviewable candidate patches.

[0084] Meanwhile, the pre-screening structure of this invention sacrifices some recall for improved queue purity and downstream review load. In the test set, the method of this invention incorrectly filtered 912 review-suitable candidate patches; therefore, this method is suitable for high-throughput queue shaping scenarios with limited review resources and should not be used as a replacement for final semantic verification.

[0085] To evaluate the deployment effect of this invention as a primary structural pre-screening module, the method was further deployed before GPT-4o semantic review on 250 manually adjudicated samples. When GPT-4o reviewed all 250 samples alone, it required 250 downstream model calls, with an average latency of 3062.13 milliseconds. After using the method of this invention as a pre-screening layer, only 151 candidate patches entered GPT-4o review, reducing the number of downstream model calls by 39.6% and the average latency to 1839.25 milliseconds, a reduction of approximately 40%. This experiment demonstrates that this invention is suitable as a structural admission layer before high-cost semantic review.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of the present invention.

Claims

1. A referenceless filtering method for generating patches for large language models, characterized in that: Includes the following steps, S1. Obtain the vulnerability program to be patched and the candidate security patches generated by the large language model, perform syntactic integrity checks on the candidate patches, and filter out candidate patches that cannot form a valid abstract syntax tree. S2. Construct abstract syntax trees for the original vulnerable program and the candidate patches that have passed the syntax integrity test after detection, and extract the structural change information between the two to form structural change features that describe the degree of patch modification. S3. Based on the degree of change in the abstract syntax tree of the candidate patches, establish a lower bound constraint on structural changes and perform anti-semantic drift filtering on the candidate patches. S4. Establish upper bound constraints on structural changes for the selected candidate patches and identify void repair patches with insufficient structural changes. S5. Based on the common vulnerability enumeration category to which the candidate patch belongs, conditionally adjust the reasonable structural change range; S6. Retain all candidate patches that meet the screening criteria in the review queue and output them to the subsequent manual code review, large language model semantic analysis or dynamic verification module.

2. The no-reference filtering method for generating patches for large language models according to claim 1, characterized in that: In step S1, the logic for determining syntactic integrity is as follows: Define an abstract syntax tree construction function f. AST (·), candidate patch P i The corresponding abstract syntax tree is T. i =f AST (P i ); like T i It is non-empty and T i If it can be parsed effectively, then the syntax determination function G syn (P i If )=1, proceed to the subsequent structural analysis process; If T i If G is empty or cannot be resolved effectively, then syn (P i If )=0, stop performing subsequent steps on the candidate patch.

3. The no-reference filtering method for generating patches for large language models according to claim 1, characterized in that: In step S2, when constructing the abstract syntax tree, the structure change vector is extracted by counting the number of newly added nodes, deleted nodes, modified node attributes, and adjusted node positions between two abstract syntax trees, and the normalized structure change rate and structure retention rate are calculated.

4. The no-reference filtering method for generating patches for large language models according to claim 3, characterized in that: When extracting the structural change vector, the structural change vector d i =[n ins ,n del ,n upd ,n mov ], Where, n ins n represents the number of new nodes. del n is the number of nodes to be deleted. upd The number of times a node attribute is modified, n mov Adjust the number of nodes at their positions; The normalized structural change rate R i =(n ins +n del +n upd +n mov ) / |V o | , where |V o |The total number of abstract syntax tree nodes for the original vulnerable program; The structural retention rate S i =1-R i , among which, S i The closer the value is to 1, the higher the structural consistency between the candidate patch and the original vulnerable program; conversely, the lower the value is, the more extensive the modifications made to the program.

5. The no-reference filtering method for generating patches for large language models according to claim 2, characterized in that: In step S3, the rule for anti-semantic drift filtering is set as follows: the lower bound threshold of the structure retention rate is set to τ. L Only retain those that satisfy the candidate patch structure retention rate S i ≥τ L Candidate patches, if S i <τ L If so, the patch is determined to have a semantic drift risk and is filtered out.

6. The no-reference filtering method for generating patches for large language models according to claim 3, characterized in that: In step S4, the void repair identification rule is as follows: set an upper bound threshold τ for the structure retention rate. U Only retain those that meet the structural retention rate S i ≤τ U Candidate patches; if S i >τ U If so, it is identified as a hole repair patch and filtered out.

7. The no-reference filtering method for generating patches for large language models according to claim 6, characterized in that: In step S5, by obtaining the CWE vulnerability category corresponding to the vulnerability, the upper limit threshold of the structure retention rate is adaptively adjusted according to different vulnerability types to form a reasonable structure change range for the corresponding vulnerability category, and only candidate patches whose structure retention rate falls within this range are retained.

8. The no-reference filtering method for generating patches for large language models according to claim 7, characterized in that: The adaptive adjustment process includes: pre-establishing a mapping relationship Φ(c) between CWE vulnerability categories and structural change ranges. i )=[τ L (c i ),τ U (c i )], Among them, c i τ represents the vulnerability type to which the candidate patch belongs. L (c i ) represents vulnerability category c i The corresponding lower bound of structural change; τ U (c i ) represents vulnerability category c i The corresponding upper bound of the structure retention rate; For minor, localized fixes, the timeframe should be narrowed; for complex vulnerabilities requiring extensive logic adjustments, the timeframe should be broadened. For vulnerabilities satisfying τ... L (c i )≤S i ≤τ U (c i The patch will be retained.

9. A filtering system for a no-reference filtering method for generating patches for large language models as described in any one of claims 1-8, characterized in that: include The patch acquisition module is used to connect to the code hosting platform and LLM inference service to obtain the original vulnerability program and all candidate patches generated by LLM, and build a candidate patch set; The syntax detection module is used to call the corresponding programming language parser to complete the construction and verification of the abstract syntax tree and filter out syntax invalid patches. The abstract syntax tree analysis module is used to construct abstract syntax trees for the original program and candidate patches respectively, and establish node mapping relationships. The structural change calculation module is used to count the number of abstract syntax tree editing operations of candidate patches relative to the original vulnerability program, and to calculate the normalized structural change rate and structural retention rate. The vulnerability type analysis module is used to read the vulnerability CWE category, load the corresponding adaptive structural change range parameters, and send them to the queue shaping module; The candidate patch queue shaping module is used to chain syntax validation, double boundary structure constraints, and vulnerability type adaptive interval triple filtering rules to eliminate semantic drift patches, hole repair patches, and syntax error patches. The results output module is used to output the queue of optimized candidate patches after screening, and connects to the manual review, large language model semantic analysis, or dynamic verification module and automatic deployment subsystem.

10. A no-reference filtering system for generating patches for large language models according to claim 9, characterized in that: This system is used as a pre-screening module and can be deployed in continuous integration and continuous deployment pipelines, automated program repair platforms, intelligent code review platforms, and software supply chain security analysis systems.