Word-level reinforcement learning security code generation method based on double-view contrast alignment

Abstract

The application belongs to the technical field of artificial intelligence safety and code large model, and specifically discloses a word element level reinforcement learning safety code generation method based on double-view comparison alignment, which comprises the following steps: constructing a vulnerability induction adversarial dataset; completing safety reasoning knowledge distillation through supervised fine-tuning to obtain a reference model; calculating word element level safety information gain through the reference model based on the word element probability distribution difference between the code generation perspective and the code repair perspective to obtain a basic reward signal; obtaining word element level safety reward through harmonic entropy confidence factor adaptive adjustment and reward diffusion; combining the word element level safety reward and the function correctness reward to construct a hybrid reward function, and performing reinforcement learning training on a strategy model to obtain a safety code generation model; and based on a code generation requirement instruction, obtaining target code with function correctness and safety through the safety code generation model. Through the application, high-fidelity safety code generation is realized.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence security and large code model technology, and more specifically, relates to a word-level reinforcement learning secure code generation method based on dual-view contrast alignment. Background Technology

[0002] With breakthroughs in the capabilities of large language models for code understanding and generation, software engineering is shifting from a human-centered, manual coding model to a collaborative development paradigm deeply integrated with artificial intelligence. Programming assistants based on large code models are widely used in daily software development, and AI-assisted programming has become a mainstream practice in modern software engineering.

[0003] However, the increased efficiency comes with significant security risks. Because the pre-training corpus of large models is contaminated with numerous low-quality, vulnerable code samples, the model mimics and reproduces these insecure coding patterns during the generation process. This results in output code that appears syntactically valid and functionally sound, yet harbors hidden security vulnerabilities. Empirical studies show that even the current state-of-the-art large language models generate code containing 40% to 76% high-risk vulnerabilities defined by the Common Weakness Enumeration (CWE). While the models perform exceptionally well when evaluating only functional correctness, their overall performance decreases by an average of about 30% after introducing security constraints. This reflects a capability imbalance between functional implementation and security compliance in large code models: they can continuously generate logically complete functional code, but struggle to simultaneously meet stringent security standards.

[0004] To address these issues, Supervised Fine-Tuning (SFT) has become a mainstream technique for code security alignment. Its typical process involves further training the model on a dataset containing security patches and vulnerability fixes, hoping the model will learn secure coding preferences. However, supervised fine-tuning is essentially a mimicry learning paradigm; performance improvements largely stem from superficial format alignment. The model often only learns shallow security rules, exhibiting poor generalization ability and ultimately developing behaviors that circumvent rule checks rather than internalizing fundamental principles of secure coding.

[0005] To overcome the limitations of supervised fine-tuning, existing research has introduced reinforcement learning (RL) into the security alignment process, directly guiding the model's generation behavior through reward signals. However, existing RL frameworks still face challenges in reward signal design: whether relying on noisy external reward agents or sparse result-based feedback, it is difficult to provide stable and accurate security guidance for the model, severely restricting the reliability and scalability of security alignment. Ideally, security reward signals should be based on real vulnerability samples, but such samples come from real code repositories, depend on complex environments and compilation conditions, and the cost of building security tests at scale is extremely high. Existing solutions rely on static analysis tools as reward models, which suffer from serious false positives and false negatives. Noisy rewards not only fail to provide accurate guidance for long sequence code generation but may also incentivize the model to deceive the detector instead of actually fixing vulnerabilities. On the other hand, existing result-based rewards only provide sparse binary signals of success or failure, leading to serious credit assignment problems: the model cannot locate the specific steps that cause erroneous outputs, and when the generated code fails the test, the model cannot obtain effective learning signals, further exacerbating the optimization dilemma.

[0006] While existing lexical-level reward methods can provide denser learning signals than sparse results, they lack principled standards for evaluating the security relevance of lexical terms. Directly transferring them to the code domain presents two inherent challenges: First, general syntactic terms and long-tailed noisy terms are easily misrepresented and their rewards are amplified. Terms such as def, return, and punctuation, which are unrelated to security, may be assigned non-zero rewards due to probability fluctuations, diluting the signals for truly security-critical decisions. Second, the reward differentiation between security keyword terms and vulnerability indicator terms is insufficient. The strict syntactic constraints of code generation lead to strong conditional dependencies. Once a security-related prefix is ​​generated, the probability distribution of subsequent terms converges rapidly, resulting in extremely low differentiation between secure and insecure continuations, leading to sparse and fragmented security rewards.

[0007] Therefore, how to build dense, accurate, and robust security reward signals for large code models without relying on external tools or losing the model's general coding capabilities, and achieve high-fidelity secure code generation, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this application is to construct dense, accurate, and robust security reward signals for large code models without relying on external tools or sacrificing the model's general coding capabilities, thereby achieving high-fidelity secure code generation.

[0009] To achieve the above objectives, this application provides a lexical-level reinforcement learning secure code generation method based on dual-view contrast alignment, the method comprising: Construct a vulnerability-inducing adversarial dataset; Based on the vulnerability-induced adversarial dataset, we complete the security reasoning knowledge distillation through supervised fine-tuning, inject prior knowledge of vulnerability patterns and defense strategies into the pre-trained model, and obtain the reference model. Based on the difference in lexical probability distribution between code generation and code repair perspectives, a basic reward signal is obtained by calculating lexical-level security information gain through a reference model. The basic reward signal is adaptively adjusted using a harmonic entropy confidence factor, and reward diffusion is completed by combining a syntax-semantic dual-track propagation mechanism to obtain a lexical-level security reward. A hybrid reward function is constructed by combining the lexical-level security reward and the functional correctness reward. The policy model is trained by reinforcement learning based on a vulnerability inducement adversarial dataset and a near-end policy optimization algorithm to obtain a secure code generation model. Based on code generation requirements, a secure code generation model is used to obtain target code that combines functional correctness and security.

[0010] Specifically, the technical concept of this application lies in treating code generation and code repair as two sampling tasks with different conditional entropies on the same program manifold: the code generation perspective relies only on functional descriptions, resulting in higher conditional entropy; the code repair perspective additionally provides vulnerability code and vulnerability metadata, forming an omniscient perspective, significantly reducing conditional entropy. This entropy difference quantifies the mutual information brought about by vulnerability perception, constituting a dense security supervision signal naturally embedded in the repair task. Based on the above concept, this application first constructs a vulnerability inducement adversarial dataset covering multiple CWE vulnerability types. This dataset contains pairs of vulnerability code, security repair code, functional test cases, and vulnerability inducement programming instructions. On this basis, through supervised fine-tuning, security inference knowledge distillation is completed, injecting prior knowledge of vulnerability patterns and defense strategies into the pre-trained model to obtain a reference model, completing the model cold start.

[0011] In the reinforcement learning phase, this application utilizes a reference model to calculate the generation probability of each lexical under both the code generation and code repair perspectives. The probability difference between the two perspectives is used as the lexical-level security information gain to obtain the basic reward signal. Subsequently, a harmonic entropy confidence factor is introduced to adaptively adjust the basic reward signal: when both perspectives are in a high-entropy state, the confidence factor attenuates the reward from long-tailed noise; when both perspectives are in a low-entropy state, the basic reward is close to zero and is unaffected by the confidence factor. The confidence factor only amplifies the reward for lexicals that simultaneously possess both high probability differences and high model certainty, which is a typical characteristic of security-related decisions, thereby achieving lexical-level hard negative example mining and noise suppression. Furthermore, reward diffusion is accomplished through a syntactic-semantic dual-track propagation mechanism: in the syntactic dimension, the cAST algorithm can be used to divide the generated code into semantically coherent syntactic blocks, and the reward sharing among syntactically related elements can be promoted through intra-block enhancement factors; in the semantic dimension, the symmetric attention weights of the reference model are extracted to calculate the semantic correlation between lexicals, and the Top-k related lexicals are selected for reward propagation. The confidence-weighted base reward and the propagation reward are then weighted and fused to obtain a lexical-level security reward. A hybrid reward function is constructed by combining the lexical-level security reward and the functional correctness reward. The policy model is then trained using reinforcement learning based on the near-end policy optimization algorithm. At the same time, the distribution deviation between the policy model and the reference model is constrained by KL divergence to avoid degradation of general coding capabilities, thus obtaining a secure code generation model.

[0012] During the inference phase, code generation requirements are input into the secure code generation model. The model can generate target code that is both functionally correct and secure without the need for additional fix contexts or external security tools.

[0013] Therefore, this application endogenously generates a term-level security reward signal from the dual perspectives of code generation and code repair, improves the reward signal-to-noise ratio with the harmonic entropy confidence factor, and overcomes the reward sparsity problem with the syntax-semantic dual-track propagation mechanism. Without relying on external tools or losing the model's general coding capabilities, it constructs a dense, accurate, and robust security reward signal for the large code model, achieving high-fidelity secure code generation.

[0014] In one possible implementation, the difference in lexical probability distributions between the code generation and code repair perspectives is used to calculate the lexical-level security information gain through a reference model, resulting in a basic reward signal, including: Construct generation perspective prompts that only contain function generation instructions, and repair perspective prompts that supplement vulnerability metadata and repair context; Based on the generation perspective cue words and the repair perspective cue words, the generation probability of each word at each time step under the generation perspective and the repair perspective is calculated by a reference model. The difference between the probabilities of the two perspectives is taken to obtain the word-level security information gain as the basic reward signal. The calculation formula of the basic reward signal is as follows: ; in, This represents the security information gain at the word level, serving as the basic reward signal; Represents the reference model; Indicates the word elements generated at the current time step; Indicates a tooltip for fixing the viewpoint; Indicates the generation of perspective prompts; This represents the generated historical word sequence.

[0015] Generate perspective prompts: These prompts only contain prompts for function generation instructions and do not provide any vulnerability-related information. They simulate the perspective of the model in a regular code generation scenario, where the model only samples based on functional requirements.

[0016] Repair perspective prompts: Based on the function generation instructions, prompts for vulnerability metadata and repair context are added to provide the model with complete information for vulnerability perception, forming an "omniscient perspective" with significantly reduced conditional entropy. This constitutes a word-level security information gain with the probability difference between the generated perspective prompts and the actual generation prompts.

[0017] Vulnerability metadata refers to structured descriptive information related to vulnerabilities, including CWE identifiers, vulnerability types, vulnerability triggering conditions, and variable-level triggering paths, which are used to reveal hidden security weaknesses in the current code to the model.

[0018] Repair context: refers to the vulnerability code itself and its corresponding security patch version code, etc., providing the model with a complete repair path from vulnerability to security implementation, giving it the conditional sampling capability of "omniscient perspective".

[0019] In one possible implementation, the above adaptive adjustment of the basic reward signal using the harmonic entropy confidence factor includes: Calculate the entropy values ​​of word prediction from both the generation and repair perspectives. Construct a harmonic entropy confidence factor based on the harmonic mean of the two entropy values. Obtain the harmonic entropy confidence factor using the following formula: ; in, Represents the confidence factor for harmonic entropy; This represents the entropy value of word prediction from the generation perspective; This represents the entropy value of lexical prediction from the perspective of repair.

[0020] In one possible implementation, the above-mentioned combination of syntax-semantic dual-track propagation mechanism completes the reward diffusion, resulting in a lexical-level security reward, which is calculated using the following formula: ; in, Indicates word elements The (final) morpheme-level security reward; This represents the balance coefficient, used to adjust the weight of the base reward and the propagation reward; This represents the set of the selected Top-k related words (the first k related words); This represents the normalized propagation weight; Indicates word elements Basic reward signals; Indicates word elements Basic reward signals; Indicates word elements The confidence factor for harmonic entropy; Indicates word elements The confidence factor of harmonic entropy.

[0021] In one possible implementation, the formula for the hybrid reward function is as follows: ; in, Represents a hybrid reward function; This indicates the weighting of safety and functionality; This represents the sequence-level security reward, i.e., the average word-level security reward of the generated sequence; This represents the functional correctness reward, specifically the pass rate of the generated code in the accompanying unit tests.

[0022] In one possible implementation, the aforementioned vulnerability-induced adversarial dataset includes: The large model (or large inference model) is guided by prompt words to select an appropriate vulnerability inducement strategy from preset vulnerability inducement strategies (such as implicit inducement, forced inducement, vulnerability code completion, vulnerability code manipulation, etc.). The original vulnerability dataset is reconstructed to generate structured samples. The structured samples include threat scenario analysis data, functionally correct insecure code, corresponding patched secure version code, pytest-style unit test cases, and vulnerability inducement programming instructions. By rejecting samples and filtering samples, samples that can be executed normally in functional tests and whose vulnerability inducement instructions do not have explicit malicious prompts are retained to obtain vulnerability inducement adversarial datasets. These datasets are then divided into supervised fine-tuning datasets (for supervised fine-tuning) and reinforcement learning datasets (for reinforcement learning) according to a preset ratio.

[0023] In one possible implementation, the above-mentioned vulnerability-induced adversarial dataset is used to perform security reasoning knowledge distillation through supervised fine-tuning, injecting prior knowledge of vulnerability patterns and defense strategies into the pre-trained model to obtain a reference model, including: Using a teacher model, structured secure inference trajectories are generated for each sample in the supervised fine-tuning dataset; Based on the secure inference trajectory and patch security version code for each sample, the student model is subjected to supervised fine-tuning for the next lexical prediction to obtain the reference model.

[0024] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) Endogenous lexicon-level dense reward without external dependence. This application innovatively utilizes the natural duality of code generation and code repair tasks to endogenously generate lexicon-level dense reward signals from the probability difference between the two perspectives. It does not rely on external static analysis tools or vulnerability detection models, which effectively solves the problems of high noise and poor generalization of external rewards. It transforms post-repair experience into pre-defense intuition and achieves security alignment without external dependence.

[0025] (2) High signal-to-noise ratio term-level hard negative example mining and syntactic-semantic dual dense feedback. The harmonic entropy confidence factor designed in this application can selectively amplify the reward signal of security key terms, while suppressing the reward dilution caused by general syntactic terms and long-tail noise, realizing term-level hard negative example mining and improving the fidelity and signal-to-noise ratio of the reward signal; the syntactic-semantic dual-track propagation mechanism effectively overcomes the reward sparsity problem caused by autoregressive dependence, ensuring that the security key signal can completely cover the entire defense logic code block, and providing the model with coherent and dense feedback.

[0026] (3) High-quality adversarial data construction with extremely low "alignment tax". This application constructs a high-quality adversarial dataset covering multiple programming languages ​​and multiple CWE types through a vulnerability inducement dataset synthesis scheme driven by a large inference model. It eliminates the need for complex code dependency resolution and context tracking, greatly simplifying the construction process of secure alignment data. The model cold start is completed through supervised fine-tuning, and the reinforcement learning optimization of the hybrid reward function improves the security of code generation while keeping the fluctuation of the model's general coding ability within 2%, effectively avoiding the "alignment tax" problem caused by secure alignment.

[0027] (4) Cross-language universal security model and broad industrial application capability. The method of this application has strong generalization and cross-language transfer capability. The model trained on Python data can achieve excellent security generation results in programming languages ​​such as C, C++, and Go. It captures a language-independent universal security model, which is applicable to various code-specific large models and general large language models, and has broad industrial application value. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the word-level reinforcement learning secure code generation method based on dual-view contrast alignment provided in an embodiment of this application. Figure 2 This is a schematic diagram illustrating the principle of dual-view comparison in the secure code generation method provided in this application embodiment; Figure 3 This is a schematic diagram of the overall architecture of the secure code generation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To facilitate a clearer understanding of the various embodiments of this application, some relevant background knowledge will be introduced as follows.

[0030] Analysis of the shortcomings of existing safety alignment schemes.

[0031] (1) Generalization defects of the SFT scheme. SFT learns secure coding preferences by imitation learning on secure patch datasets. However, most of the performance improvement of SFT comes from superficial format alignment. The model can only learn shallow security rules and cannot internalize the basic principles of secure coding, resulting in poor generalization ability.

[0032] (2) Noise problem of external reward agents. Existing reinforcement learning schemes rely on static analysis tools as reward models, which have serious false positive and false negative problems. Noisy rewards not only fail to provide accurate guidance for the generation of long sequence code, but may also incentivize the model to learn to "deceive the detector" rather than actually fix vulnerabilities.

[0033] (3) The credit allocation problem of sparse binary rewards. The result-based reward only provides sparse binary signals of success or failure. The model cannot locate the specific steps that lead to the wrong output. When the generated code fails the test, the model cannot obtain effective learning signals.

[0034] (4) The inherent challenges of existing lexical-level reward methods. Existing lexical-level reward methods lack principled standards for evaluating the security relevance of lexicals. General grammatical lexicals and long-tailed noise lexicals are easily mistakenly amplified in rewards. The reward differentiation between security key lexicals and vulnerability indicator lexicals is insufficient, resulting in sparse and fragmented security rewards.

[0035] Key technology concepts.

[0036] (1) CWE. CWE, or Common Weakness Enumeration, provides a standardized way to describe software weaknesses and is a general standard in the field of code security. CWE assigns a unique identifier to each software weakness, enabling security experts, developers and users to refer to and discuss the same type of weakness in a unified manner, and has a wide application basis in vulnerability analysis, secure coding and code review.

[0037] (2) Token. A token is the smallest processing unit after the code-big model segments the code text. In autoregressive code generation, the model generates tokens one by one, and the generation of each token depends on the previously generated token sequence. When generating code, the code-big model predicts the next output step by step on a token-by-total basis; therefore, the token is the basic granularity of the model's generation behavior. Providing accurate security-related feedback for each token is crucial for guiding the model to generate secure code.

[0038] (3) Proximal Policy Optimization (PPO). PPO is a reinforcement learning algorithm widely used for alignment of large language models. It features stable training and strong hyperparameter robustness. By limiting the magnitude of policy updates, PPO avoids policy collapse while ensuring training efficiency, and is one of the mainstream reinforcement learning algorithms in the post-training stage of large language models.

[0039] (4) Harmonic Mean. The harmonic mean is a special type of average, calculated by dividing n by the sum of the reciprocals of the data. In this application, the harmonic mean is used to construct the harmonic entropy confidence factor to reconcile the word prediction entropy values ​​under the generation and repair perspectives. The characteristics of the harmonic mean make it more sensitive to smaller values, thus effectively amplifying the penalty signal of the model under uncertain conditions while suppressing noise under highly deterministic conditions.

[0040] (5) Abstract Syntax Tree (AST) and cAST algorithm. An AST is a tree-like representation of the abstract syntactic structure of source code, where each node represents a structure in the source code. The cAST (chunked AST) algorithm maximizes information density while preserving the integrity of syntactic boundaries by recursively splitting large AST nodes and merging sibling nodes. It can be used to divide generated code into semantically coherent syntactic blocks. Reward propagation based on syntactic blocks can ensure reward sharing among syntactically related elements and overcome the reward sparsity problem caused by autoregressive dependencies.

[0041] (6) Attention Mechanism and Symmetric Attention Weights. The attention mechanism is a core component of large language models. By calculating the attention weights between lexical units, the model can capture long-distance dependencies. Symmetric attention weights are obtained by averaging the attention matrix and its transpose, which enhances the symmetry of the attention weights and more accurately reflects the bidirectional associations between lexical units. The semantic association between lexical units is calculated based on symmetric attention weights, which can be used to select semantically related lexical units for reward propagation.

[0042] (7) Knowledge distillation. Knowledge distillation is a model compression technique that transfers knowledge from a large model to a small model by training a student model to mimic the output of a teacher model. In this application, the knowledge distillation in the supervised fine-tuning stage utilizes the teacher model (which employs a large inference model) to generate secure inference trajectories and secure code implementations, injecting prior knowledge of vulnerability patterns and defense strategies into the student model (pre-trained model), thus providing a stable cold-start foundation for subsequent reinforcement learning training.

[0043] (8) Reference Model and Policy Model. In this application, the reference model is the model obtained after supervised fine-tuning (SFT) of the pre-trained model, which has internalized secure reasoning knowledge and basic secure coding preferences; the policy model is the model optimized during reinforcement learning training. The policy model directly copies the weights of the reference model during initialization, and constrains the distribution deviation between the policy model and the reference model through KL divergence (KL divergence for short) during reinforcement learning training, so as to avoid the policy model deviating too much from the reference model and causing degradation of general coding ability.

[0044] (9) pytest: A widely used automated unit testing framework in the Python ecosystem.

[0045] (10) Alignment tax: A term in the field of secure alignment of large language models, referring to the phenomenon that after the model is trained with secure alignment, the general capabilities (such as code generation accuracy) will degrade to a certain extent, and the degree of degradation is called "tax".

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0049] The embodiments of this application are described below with reference to the accompanying drawings.

[0050] This application provides a method for generating secure code using word-level reinforcement learning based on dual-view contrast alignment.

[0051] like Figure 1 As shown, the method includes the following steps S1 to S4.

[0052] Step S1: Reconstruct the scenario of the original vulnerability dataset using a large-scale inference model to build a vulnerability inducement adversarial dataset. The vulnerability inducement adversarial dataset covers a variety of CWE vulnerability types and includes pairs of vulnerability code (also known as insecure code), security fix code (also known as patch security version code), functional test cases, and vulnerability inducement programming instructions.

[0053] Step S2: Based on the vulnerability-induced adversarial dataset, security reasoning knowledge distillation is completed through supervised fine-tuning. Prior knowledge of vulnerability patterns and defense strategies is injected into the pre-trained model to obtain a reference model and complete the model cold start.

[0054] Step S3: Based on the difference in lexical probability distribution between the code generation perspective and the code repair perspective, calculate the lexical-level security information gain to obtain the basic reward signal; adaptively adjust the basic reward signal through the harmonic entropy confidence factor, and complete the reward diffusion by combining the syntax-semantic dual-track propagation mechanism to obtain the final lexical-level security reward; construct a hybrid reward function by combining the lexical-level security reward and the functional correctness reward, and perform reinforcement learning training on the policy model based on the near-end policy optimization algorithm to obtain the trained secure code generation model.

[0055] Step S4: Input the code generation requirement instruction into the secure code generation model to obtain the target code output by the model, which has both functional correctness and security.

[0056] It should be noted that the key model entities involved in this application and their relationships are as follows: the pre-trained model refers to the basic code model that has been pre-trained on a large-scale code corpus; the reference model refers to the model obtained after supervised fine-tuning (SFT) of the pre-trained model, which has internalized secure reasoning knowledge and basic secure coding preferences; the policy model refers to the model that is optimized during reinforcement learning training. The policy model directly copies the weights of the reference model during initialization, and constrains the distribution deviation between the policy model and the reference model through KL divergence during reinforcement learning training, so as to avoid the policy model deviating too much from the reference model and causing degradation of general coding ability.

[0057] The following sections will provide a detailed explanation of each of the above steps.

[0058] Step S1: Construction of vulnerability-induced adversarial dataset.

[0059] In the embodiments of this application, in step S1, PrimeVul, DiverseVul, and SecVulEval are selected as the original seed datasets. These datasets not only provide high-quality vulnerability code snippets and patch versions, but also retain traceable repository addresses, commit identifiers, and detailed CWE vulnerability description information, serving as the core input for vulnerability inducement data synthesis. Dedicated prompts for red team code security analysis are designed to guide large-scale inference models (such as DeepSeek V4 Pro) to select an appropriate vulnerability inducement strategy from four approaches—implicit inducement, forced inducement, vulnerability code completion, and vulnerability code manipulation—for each seed dataset, generating standardized structured samples.

[0060] The four vulnerability exploitation strategies are defined as follows: Implicit vulnerability inducement: By using neutral and ambiguous requirement descriptions, ambiguity is reserved for secure and insecure implementations, inducing the model to choose the insecure implementation path in the absence of clear security specifications.

[0061] Forced vulnerability inducement: By adding additional functional constraints or performance optimization requirements in the task prompts, the model is forced to adopt a known insecure implementation, thereby introducing specific vulnerabilities while satisfying explicit functional requirements.

[0062] Vulnerability code completion: Provides a secure and valid code prefix in the instructions, including partial implementation and environment configuration, requiring the model to complete or extend the code in a natural way, and the vulnerability is naturally introduced in the code continuation part.

[0063] Vulnerability code manipulation: Provide a complete code snippet containing a known vulnerability in the instruction, requiring the model to extend its functionality, modify business logic, or optimize performance, while retaining the original vulnerability or reintroducing a new vulnerability in the process.

[0064] The structured samples generated by the large-scale inference model consist of five parts: (i) threat scenario analysis data, used to construct a task scenario that matches (or is similar to) the input vulnerability code, to generate vulnerability-inducing programming instructions, explaining the target vulnerability type and the vulnerability triggering logic of the generated code; (ii) functionally correct insecure code; (iii) the corresponding patched secure version code; (iv) functional test cases (specifically pytest-style unit test cases), which combine the functionally correct insecure code and the corresponding patched secure version code to generate several test cases that allow both to pass, in order to evaluate the functionality of the code; and (v) vulnerability-inducing programming instructions, used to enhance the vulnerability risk, expecting the model to still generate secure code under induced conditions. The functionally correct insecure code and the corresponding patched secure version code are used to construct pytest-style unit test cases; pytest-style unit test cases can also be used during the rejection sampling process to determine whether the sample's functional testing is normal. For ease of explanation, the following text will treat vulnerability code and insecure code as the same concept, and security patch code and patched secure version code as the same concept.

[0065] The sample selection process involved rejecting samples: retaining samples that could be executed normally in functional tests and whose vulnerability inducement commands did not contain explicit malicious warnings, while removing samples that did not meet the requirements. Ultimately, a vulnerability inducement adversarial dataset was constructed covering 45 typical CWE vulnerability types in two mainstream programming languages, Python and C / C++, and divided into a supervised fine-tuning dataset (SFT dataset) and a reinforcement learning dataset (RL dataset) according to a preset ratio.

[0066] During supervised fine-tuning, vulnerability-induced programming instructions serve as input to the model, and patched secure version code serves as supervision labels; during reinforcement learning, vulnerability-induced programming instructions serve as input to the model, and pytest-style unit test cases are used to generate functional correctness rewards.

[0067] The method in this application embodiment uses an end-to-end data synthesis scheme driven by a large inference model, which eliminates the need for complex code dependency resolution and context tracking, greatly simplifying the construction process of secure alignment training data. The generated vulnerability inducement instructions can naturally trigger the unsecured coding behavior of the model, providing a high-quality and diverse data foundation for subsequent secure alignment training.

[0068] Step S2: Knowledge distillation for secure reasoning and cold start of the model.

[0069] In the embodiments of this application, in step S2, based on the supervised fine-tuning dataset, security reasoning knowledge distillation is completed through supervised fine-tuning to inject prior knowledge of vulnerability patterns and defense strategies into the pre-trained model, thereby obtaining a reference model and completing the model cold start. Specifically, this includes: Using teacher models (such as large models like DeepSeek V4 Pro), structured security inference trajectories are generated for each vulnerability sample in the supervised fine-tuning dataset. These inference trajectories are then processed through... <thinking> and< / thinking> The tagging system explicitly includes three core dimensions: (i) the intended function and operational goals of the code; (ii) potential security vulnerabilities, including CWE identifiers, variable-level trigger paths, and vulnerability triggering conditions; and (iii) specific defense measures and their effectiveness principles. Following the inference trajectory, through... <code> and< / code> The security code implementation corresponding to the tag is the security patch version of the vulnerability code in the above data construction.

[0070] The generated security inference trajectory and the security version code of the patch (security code implementation) are used as tags for supervised fine-tuning. The next lexical prediction supervised fine-tuning is performed on the student policy model, so that the model completes the structured security inference process of functional and risk analysis and proposes defense schemes before generating code. This provides a stable cold start foundation for subsequent reinforcement learning training and injects prior knowledge of vulnerability patterns and defense strategies.

[0071] The method in this application embodiment, through supervised fine-tuning of knowledge distillation, enables the model to internalize a structured secure reasoning workflow in advance, avoiding the instability of reinforcement learning training starting from scratch, accelerating the convergence speed of subsequent reinforcement learning, and establishing a basic secure coding preference for the model.

[0072] Step S3: Lexical-level security reward calculation and reinforcement learning training.

[0073] S3.1. Lexical-level security information gain calculation based on dual-perspective comparison.

[0074] In the embodiments of this application, step S3, based on the difference in lexical probability distribution between the code generation perspective and the code repair perspective, calculates the lexical-level security information gain to obtain the basic reward signal, specifically including: Code generation and code repair can be viewed as two conditional sampling tasks on the same program manifold. The core difference between them lies in the available contextual information: the generation task relies only on the functional description, resulting in high conditional entropy; the repair task additionally provides vulnerability metadata (CWE identifier, vulnerability code, vulnerability description, etc.) and repair context, forming an omniscient perspective, and significantly reducing conditional entropy. This entropy difference quantifies the mutual information brought about by vulnerability awareness and is a dense security monitoring signal naturally embedded in the repair task.

[0075] Construct two perspective prompts: a generation perspective prompt containing only function generation instructions (in the context of...). (indicated), and supplemented with vulnerability metadata and remediation context remediation perspective prompts (in the form of) (Representation). Through the reference model Calculate the generation probability of each lexical unit at each time step from both perspectives, and then calculate the lexical-level security information gain based on the probability difference between the two perspectives. ,Will As the basic reward signal, the formula for calculating the word-level security information gain is: ; Parameter explanation: : Represents the security information gain at the lexical level, that is, the difference between the lexical generation probability under the repair perspective and the generation perspective, which serves as the basic reward signal; : Indicates the reference model; : Represents the tokens generated at the current time step; : Indicates a tooltip for fixing the viewpoint; : Indicates the generation of perspective prompts; : Represents a sequence of historical lexical terms that has already been generated.

[0076] Figure 2 This diagram illustrates the principle of dual-perspective comparison in the secure code generation method provided in this application. Using the CWE-78 operating system command injection vulnerability as an example, it demonstrates the difference in probability distribution of the same term in the generation perspective (only receiving functional requirements) and the remediation perspective (supplementing CWE-78 vulnerability metadata). The diagram shows a Python code snippet demonstrating the POST interface / execute using the fastapi and subprocess libraries, waiting for completion at the last line of code, shell=. The diagram shows two completion results: secure completion (shell=False, capture_output=True) and vulnerability completion (shell=True, capture_output=True). In the generation perspective, the model assigns a high probability of 0.67 to the unsafe term True; in the remediation perspective, the probability of the secure term False rises to 0.87, while the unsafe term True is suppressed to 0.01. The difference between the two is shown in the diagram. Forming strong positive rewards This results in negative penalties, while punctuation and general grammatical units unrelated to security... Without rewarding contributions, it achieves natural noise suppression. Figure 2 middle, Indicates the generation of perspective prompts, The text indicates a tooltip for fixing the viewpoint, and SCTRL indicates the reward method proposed in this application.

[0077] The method in this application generates reward signals endogenously from the information asymmetry inherent in the task itself. It only requires two forward propagations to obtain the word-level reward of the complete sequence without additional fine-tuning or external tools, thus effectively solving the noise and generalization problems of external rewards.

[0078] S3.2 Adaptive adjustment of harmonic entropy confidence factor.

[0079] In the embodiments of this application, step S3, which adaptively adjusts the basic reward signal using the harmonic entropy confidence factor, specifically includes: Basic Rewards Unable to distinguish between safety-critical decisions driven by credible reasoning and stochastic fluctuations under model uncertainty, a harmonic entropy confidence factor is introduced. The reward is dynamically scaled based on the determinism of the model.

[0080] First, the entropy values ​​of word predictions under the generation and repair perspectives are calculated. Then, a harmonic entropy confidence factor is constructed based on the harmonic mean of the two entropy values. The calculation formula is: ; Parameter explanation: : Represents the confidence factor for harmonic entropy; : Represents the entropy value of lexical prediction from the generation perspective; : Represents the entropy value of lexical prediction from the perspective of repair.

[0081] The core logic of this design is that when both perspectives are in a high-entropy state (the model is uncertain in both contexts). It will become smaller, and the effective attenuation is likely to be the reward signal from long-tailed noise; when both perspectives are in a low-entropy state (a typical feature of general grammatical terms). The reward has been reset to zero and is no longer valid. Impact. Therefore, the confidence factor only amplifies rewards for terms that simultaneously possess both high probability of difference and high model certainty, which is a typical characteristic of security-related decisions. Simultaneously, this mechanism achieves term-level hard negative example mining: when the policy model generates high-confidence erroneous terms (low entropy), but the repair perspective remains uncertain (high entropy), It will amplify the negative. This imposes strong penalties on unsafe modes.

[0082] The method in this application embodiment effectively suppresses reward dilution caused by general grammatical terms and long-tail noise by using the harmonic entropy confidence factor, while amplifying the reward signal of safety-critical decisions and improving the fidelity and signal-to-noise ratio of the reward function.

[0083] S3.3, Syntax-Semantic Dual-Track Propagation Mechanism.

[0084] In the embodiments of this application, step S3 involves combining a syntax-semantic dual-track propagation mechanism to complete reward diffusion and obtain the final lexical-level security reward, specifically including: In autoregressive code generation, early security-related terms strongly constrain the distribution of subsequent terms. Once a prefix with strong security meaning is generated, the probability quality of subsequent terms quickly converges to a narrow set of syntactically valid terms, resulting in extremely low reward distinguishability for subsequent terms in the defense logic chain and effectively sparse reward signals. To address this, a syntactic-semantic dual-track propagation mechanism is designed to diffuse reward signals through two complementary paths.

[0085] The first step, syntactic chunking and propagation: The generated code is divided into semantically coherent syntactic chunks using the cAST (chunked abstract syntax tree) algorithm. cAST maximizes information density while preserving the integrity of syntactic boundaries by recursively splitting large AST nodes and merging sibling nodes. For each lexical position in the code region... Let the leftmost and rightmost indices of its corresponding syntax block be denoted as . and The same word group receives an enhancement factor. (Default 3.0) By enhancing the attention interaction between lexical units within the same block, it amplifies the propagation effect of reward diffusion within the grammatical block and promotes reward sharing among grammatically related elements.

[0086] The second step, semantic association and propagation: extract the symmetric attention weights from the last layer of the reference model and calculate the semantic association between lexical units. For each target position... Calculate each preposition ( The original association score of ) ; Parameter explanation: : indicates word element with lexical elements The original correlation score between them; : Represents the symmetric attention weights, extracted from the last layer of the reference model; : Indicates the enhancement factor within the syntax block (default value is 3.0); : Indicates word position The leftmost index of the syntax block it belongs to; : Indicates word position The rightmost index of the syntax block it belongs to; : indicates an indicator function that takes the value 1 when the condition is true, and 0 otherwise.

[0087] Select The top-k positions (default k=10) are used to obtain the propagation weights through Softmax normalization. .

[0088] Finally, by combining the confidence-weighted base reward and the propagation reward, a final word-level security reward is constructed. The calculation formula is: ; Parameter explanation: : indicates word element The final meta-level security reward; : Represents the balance coefficient, used to adjust the weight of the base reward and the propagation reward (default value is 0.2). : Indicates the selected Top-k related terms set (default k=10); : Represents the normalized propagation weight; : indicates word element Basic reward signals; : indicates word element The confidence factor of harmonic entropy.

[0089] The method in this application embodiment uses a syntax-semantic dual-track propagation mechanism to spread sparse security-critical signals to the entire defense logic code block, overcoming the reward sparsity problem caused by autoregressive dependencies, providing the model with coherent and dense security feedback, and ensuring that the complete defense logic can obtain consistent positive incentives.

[0090] Figure 3 This is a schematic diagram of the overall architecture of the secure code generation method provided in this application embodiment. The architecture is divided into two main areas. The upper area contains two stages: Stage I "Vulnerability-Induced Data Synthesis" and Stage II "Security-Aware Inference Distillation". The lower area shows the PPO optimization framework and the lexical-level reward propagation mechanism, where PPO represents near-end policy optimization, q represents the input of the policy model, o represents the output of the policy model, r represents the reward, v represents the value of evaluating the model output, GAE represents generalized advantage estimation, and A represents the advantage estimate obtained through GAE.

[0091] S3.4, Hybrid reward function and reinforcement learning training.

[0092] In the embodiments of this application, step S3 involves constructing a hybrid reward function by combining word-level security rewards and functional correctness rewards, and training the policy model using reinforcement learning based on a near-end policy optimization algorithm. Specifically, this includes: To prevent rewarding hacking behaviors (such as generating empty or meaningless code that bypasses security checks) while preserving the model's functional correctness, a hybrid reward function is constructed by combining lexical-level security rewards with functional correctness rewards.

[0093] First, the pass rate of the generated code through the accompanying unit tests will be used as a functional correctness reward. The average word-level security reward of the generated sequence is used as the sequence-level security reward. Construct a hybrid reward function: ; Parameter explanation: : Represents a hybrid reward function; : Represents the balance weight between safety and functionality, used to adjust the priority between safety alignment and functional correctness; : Represents the sequence-level security reward, that is, the average word-level security reward of the generated sequence; : This represents the functional correctness reward, i.e., the pass rate of the generated code in passing the corresponding unit tests.

[0094] In the Proximal Policy Optimization (PPO) training, a KL divergence penalty term is added to constrain the distribution deviation between the policy model and the reference model, preventing the model from deviating excessively from the initial distribution and thus causing degradation of general capabilities. Based on a hybrid reward function, the policy model is iteratively optimized in multiple rounds until the model converges, ultimately yielding a trained secure code generation model.

[0095] The method in this application embodiment, by using a hybrid reward function, guides the model to learn secure coding patterns while ensuring the functional correctness of the generated code. It effectively balances secure alignment with general coding capabilities, keeps the performance fluctuation of the model on the general code generation benchmark to a very small range, and avoids the serious "alignment tax" problem.

[0096] Step S4: Inference generation.

[0097] In the embodiments of this application, in step S4, the code generation requirement instruction is input into the trained secure code generation model. The model can generate target code that has both functional integrity and security compliance without additional repair context or security prompts. At the same time, when faced with vulnerability inducement instructions, it can actively avoid insecure coding patterns and generate defensive security implementations.

[0098] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0100] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0101] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0102] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0103] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0104] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0105] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0106] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A word-level reinforcement learning secure code generation method based on dual-view contrast alignment, characterized in that, include: Construct a vulnerability-inducing adversarial dataset; Based on the vulnerability-induced adversarial dataset, we complete the security reasoning knowledge distillation through supervised fine-tuning, inject prior knowledge of vulnerability patterns and defense strategies into the pre-trained model, and obtain the reference model. Based on the difference in lexical probability distribution between code generation and code repair perspectives, the basic reward signal is obtained by calculating the lexical-level security information gain through a reference model. The basic reward signal is adaptively adjusted by the harmonic entropy confidence factor, and the reward diffusion is completed by combining the syntax-semantic dual-track propagation mechanism to obtain the word-level security reward. A hybrid reward function is constructed by combining the word-level security reward and the functional correctness reward. The policy model is trained by reinforcement learning based on the vulnerability inducement adversarial dataset and the near-end policy optimization algorithm to obtain the secure code generation model. Based on code generation requirements, a secure code generation model is used to obtain target code that combines functional correctness and security. The difference in lexical probability distribution based on the code generation perspective and the code repair perspective is used to calculate the lexical-level security information gain through a reference model to obtain the basic reward signal, including: Construct generation perspective prompts that only contain function generation instructions, and repair perspective prompts that supplement vulnerability metadata and repair context; Based on the generation perspective cue words and the repair perspective cue words, the generation probability of each word at each time step under the generation perspective and the repair perspective is calculated by a reference model. The difference between the probabilities of the two perspectives is taken to obtain the word-level security information gain as the basic reward signal. The calculation formula of the basic reward signal is as follows: ； in, This represents the security information gain at the word level, serving as the basic reward signal; Represents the reference model; Indicates the word element generated at the current time step; Indicates a tooltip for fixing the viewpoint; Indicates the generation of perspective prompts; This represents a sequence of historical lexical units that have already been generated; The adaptive adjustment of the basic reward signal using the harmonic entropy confidence factor includes: Calculate the entropy values ​​of word prediction from both the generation and repair perspectives. Construct a harmonic entropy confidence factor based on the harmonic mean of the two entropy values. Obtain the harmonic entropy confidence factor using the following formula: ； in, Represents the confidence factor for harmonic entropy; This represents the entropy value of lexical prediction from the generation perspective; This represents the entropy value of lexical prediction from the perspective of repair.

2. The method for generating secure code using lexical-level reinforcement learning based on dual-view contrast alignment as described in claim 1, characterized in that, The combined syntax-semantic dual-track propagation mechanism completes the reward diffusion, resulting in a lexical-level security reward, which is calculated using the following formula: ； in, Indicates word elements Metaword-level security rewards; This represents the balance coefficient, used to adjust the weight of the base reward and the propagation reward; This represents the selected set of Top-k related terms; This represents the normalized propagation weight; Indicates word elements Basic reward signals; Indicates word elements Basic reward signals; Indicates word elements The confidence factor for harmonic entropy; Indicates word elements The confidence factor of harmonic entropy.

3. The method for generating secure code using lexical-level reinforcement learning based on dual-view contrast alignment as described in claim 1, characterized in that, The formula for the hybrid reward function is as follows: ； in, Represents a hybrid reward function; This indicates the weighting of safety and functionality; This represents the sequence-level security reward, i.e., the average word-level security reward of the generated sequence; This represents the functional correctness reward, specifically the pass rate of the generated code in the accompanying unit tests.

4. The method for generating secure code using lexical-level reinforcement learning based on dual-view contrast alignment as described in claim 1, characterized in that, The constructed vulnerability inducement adversarial dataset includes: The large model is guided by prompt words to select an appropriate vulnerability inducement strategy from the preset vulnerability inducement strategies. The original vulnerability dataset is reconstructed to generate structured samples. The structured samples include threat scenario analysis data, functionally correct insecure code, corresponding patched secure version code, pytest-style unit test cases, and vulnerability inducement programming instructions. By rejecting samples and filtering samples, samples that can be executed normally in functional tests and whose vulnerability inducement instructions do not have explicit malicious prompts are retained to obtain vulnerability inducement adversarial datasets, which are then divided into supervised fine-tuning datasets and reinforcement learning datasets according to a preset ratio.

5. The method for generating secure code using word-level reinforcement learning based on dual-view contrast alignment according to claim 4, characterized in that, The vulnerability-induced adversarial dataset, through supervised fine-tuning, completes security reasoning knowledge distillation, injecting prior knowledge of vulnerability patterns and defense strategies into the pre-trained model to obtain a reference model, including: Using a teacher model, structured secure inference trajectories are generated for each sample in the supervised fine-tuning dataset; Based on the secure inference trajectory and patch security version code for each sample, the student model is subjected to supervised fine-tuning for the next lexical prediction to obtain the reference model.

Images