A deep learning vulnerability detection model based on deep reinforcement learning and an adversarial sample generation method and system thereof

Abstract

The application provides a kind of deep learning vulnerability detection model based on deep reinforcement learning adversarial sample generation method and system.The method comprises: obtaining target vulnerability detection model information based on deep learning and prototype vulnerability code set for generating adversarial sample;Effective adversarial code transformation facing target vulnerability detection model is constructed;Deep reinforcement learning framework facing optimal adversarial sample generation is constructed;Using effective adversarial code transformation and deep reinforcement learning framework facing optimal adversarial sample generation, the optimal adversarial sample of target vulnerability detection model is generated.The application uses the features in the code representation of target vulnerability detection model that have important influence on model decision to construct effective adversarial code transformation, and uses deep reinforcement learning algorithm to generate optimal adversarial sample facing target vulnerability detection model, which can more efficiently and specifically generate adversarial sample of target vulnerability detection model based on deep learning.

Description

Technical Field

[0001] This invention relates to the field of information security technology, specifically to a method and system for generating adversarial examples for a deep learning vulnerability detection model based on deep reinforcement learning. Background Technology

[0002] In recent years, the security threats posed by software vulnerabilities have become increasingly severe. The rapid increase in the number of vulnerabilities has led to a surge in cyberattacks worldwide, causing huge economic losses. Therefore, accurately and efficiently detecting and promptly fixing unknown vulnerabilities in software has become a crucial aspect of ensuring information system security.

[0003] With the rapid development of deep learning technology, deep learning-based vulnerability detection models have begun to be applied to the detection of unknown vulnerabilities, achieving excellent detection results. Deep learning technology trains on large-scale code data containing vulnerabilities through various forms of deep neural networks, intelligently extracting the key feature information required for vulnerability detection, thereby achieving accurate and efficient vulnerability detection.

[0004] However, deep learning-based vulnerability detection models are inherently vulnerable, making them susceptible to adversarial attacks. Malicious attackers can modify vulnerable code, causing deep learning-based vulnerability detection models to incorrectly identify vulnerable code as safe code, thus bypassing the detection model and allowing developers to reuse vulnerable code without their knowledge, creating opportunities for malicious attackers to launch attacks.

[0005] Although Chinese patent application CN112989358A discloses a method for generating adversarial examples for deep learning vulnerability detection models, which generates adversarial examples by implementing two types of code transformations—data transformation and control transformation—for vulnerable code snippets, this method suffers from high randomness and low efficiency, making it difficult to efficiently and specifically generate adversarial examples for different types of deep learning-based target vulnerability detection models. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method and system for generating adversarial examples for deep learning vulnerability detection models based on deep reinforcement learning. The aim is to solve the problems of high randomness and low efficiency in existing adversarial example generation methods, thereby achieving efficient and targeted generation of adversarial examples for deep learning-based target vulnerability detection models.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for generating adversarial examples based on a deep learning vulnerability detection model using deep reinforcement learning, comprising the following steps:

[0009] 1) Obtain information on deep learning-based target vulnerability detection models and a set of prototype vulnerability codes used to generate adversarial examples;

[0010] 2) Construct an effective counter-code transformation model for target-oriented vulnerability detection;

[0011] 3) Construct a deep reinforcement learning framework for generating optimal adversarial examples;

[0012] 4) Using an effective adversarial code transformation and a deep reinforcement learning framework for generating optimal adversarial examples, the optimal adversarial examples for the target vulnerability detection model are generated.

[0013] In step 1) above, the target vulnerability detection model information based on deep learning includes model instances that can perform vulnerability detection, model structure information of the detection model, and code representation information used by the detection model; the prototype vulnerability code set used to generate adversarial samples includes vulnerability code samples that the target vulnerability detection model can correctly classify as having vulnerabilities.

[0014] In step 2) above, an effective adversarial code transformation for the target vulnerability detection model is constructed through the following steps:

[0015] 2a) Calculate the importance of features in the code representation of the target vulnerability detection model in the model decision-making process;

[0016] 2b) Based on the feature importance in the code representation of the target vulnerability detection model, obtain adversarial-sensitive features;

[0017] 2c) Analyze the correspondence between adversarial sensitive features and code transformations to obtain candidate adversarial code transformations;

[0018] 2d) Based on the target constraints of the adversarial sample attack and the candidate adversarial code transformation, obtain the effective adversarial code transformation.

[0019] In step 2a) above, the method for calculating the importance of features in the code representation of the target vulnerability detection model in the model decision-making process includes the local interpretation method.

[0020] In step 2c) above, the method for analyzing the correspondence between adversarial sensitivity features and code transformation includes factor analysis.

[0021] In step 2c) above, candidate adversarial code transformation refers to one or more code transformations that can cause changes in adversarial sensitive features, including adding, deleting, and modifying code at the granularity of variables, statements, or basic blocks.

[0022] In step 3) above, a deep reinforcement learning framework for generating optimal adversarial examples is constructed through the following steps:

[0023] 3a) Model the decision-making process for generating optimal adversarial examples using a Markov decision process model;

[0024] 3b) Establish a deep reinforcement learning environment model for generating optimal adversarial examples;

[0025] 3c) Construct a DQN (Deep Q-Networks) valuation network for decision-making in a deep reinforcement learning environment model;

[0026] 3d) Train the DQN valuation network to obtain the DQN valuation network for generating optimal adversarial examples.

[0027] In step 4) above, the optimal adversarial sample for the target vulnerability detection model is generated using a deep reinforcement learning framework for generating optimal adversarial samples through the following steps:

[0028] 4a) Select prototype vulnerability code samples as input for generating adversarial examples, using a deep reinforcement learning framework oriented towards generating optimal adversarial examples;

[0029] 4b) Use the DQN valuation network to select effective adversarial code transformations applied to prototype vulnerability code samples to obtain process vulnerability code samples;

[0030] 4c) Determine whether the vulnerability code sample in the process can cause the target vulnerability detection model to misclassify it, or whether the current generation round number l is equal to the upper limit of the generation round number Z;

[0031] 4d) If so, the process vulnerability code sample will be used as the optimal adversarial sample output for the target vulnerability detection model;

[0032] 4e) If not, continue to use the DQN valuation network to select effective adversarial code transformations applied to process vulnerability code samples.

[0033] This invention also provides a deep learning vulnerability detection model adversarial example generation system based on deep reinforcement learning, which includes:

[0034] The information acquisition module is used to acquire information about the target vulnerability detection model based on deep learning and a set of prototype vulnerability codes used to generate adversarial examples.

[0035] An effective code transformation build module is used to construct effective code transformation-resistant target vulnerability detection models;

[0036] The Deep Reinforcement Learning Framework Building Module is used to build a deep reinforcement learning framework for generating optimal adversarial examples.

[0037] The optimal adversarial sample generation module is used to generate optimal adversarial samples for the target vulnerability detection model using effective adversarial code transformation and a deep reinforcement learning framework oriented towards optimal adversarial sample generation.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1) This invention utilizes the features in the code representation of the target vulnerability detection model that have a significant impact on the model's decision-making to construct effective adversarial code transformation, which can more effectively utilize the vulnerabilities of the deep learning-based target vulnerability detection model contained in these features to carry out adversarial attacks at the code transformation level.

[0040] 2) This invention employs a deep reinforcement learning algorithm to select an optimal combination of effective adversarial code transformations to generate optimal adversarial samples for target vulnerability detection models. Compared to traditional optimization combination selection methods, the deep reinforcement learning algorithm uses a deep neural network to fit the optimization strategy, resulting in faster computation and stronger optimization capabilities, thus generating optimal adversarial samples for target vulnerability detection models more efficiently. Attached Figure Description

[0041] Figure 1 This is a flowchart of an adversarial example generation method based on a deep reinforcement learning-based deep learning vulnerability detection model.

[0042] Figure 2 This is a flowchart for constructing an effective countermeasure against code transformation in a target-oriented vulnerability detection model.

[0043] Figure 3 This is a flowchart of constructing a deep reinforcement learning framework for generating optimal adversarial examples.

[0044] Figure 4 This is a flowchart of generating optimal adversarial examples for a target vulnerability detection model using a deep reinforcement learning framework. Detailed Implementation

[0045] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings.

[0046] This embodiment provides a method for generating adversarial examples based on a deep learning vulnerability detection model using deep reinforcement learning. The overall process is as follows: Figure 1 As shown, the main steps include:

[0047] 1) Obtain the target vulnerability detection model information and the prototype vulnerability code set used to generate adversarial samples. The specific implementation instructions are as follows:

[0048] 1a) Obtain information on target vulnerability detection models based on deep learning, including but not limited to model instances capable of vulnerability detection, model structure information of the detection model, code representation information used by the detection model, and other relevant information that can guide the accurate and efficient generation of adversarial samples.

[0049] The detection model's structure includes an embedding layer, a neural network layer, and an activation function layer. The embedding layer maps the code representation of the code sample to be detected into a fixed-length continuous-value feature vector, which is then input to the neural network layer. The neural network layer includes, but is not limited to, convolutional neural networks, recurrent neural networks, long short-term memory networks, or graph neural networks, and may also be a combination of multiple networks. The activation function layer includes, but is not limited to, activation functions such as ReLU, sigmoid, or softmax.

[0050] The code representation information used by the detection model includes, but is not limited to, features that can represent the syntax and semantics of the program source code, such as text sequences, grayscale images, and program dependency graphs extracted from the program source code using program analysis techniques. In this embodiment, the code representation used by the detection model is formally defined as a feature set F = {F1, F2, ..., F...} l}

[0051] 1b) Obtain the prototype vulnerability code set VC = {VC1, VC2, ..., VC} used to generate adversarial examples. n The prototype vulnerability code set used to generate adversarial examples includes vulnerability code samples that the target vulnerability detection model can correctly classify as having vulnerabilities. The prototype vulnerability code can take forms including, but is not limited to, code fragments at the function or slice level.

[0052] 2) Construct an effective adversarial code transformation for a target-oriented vulnerability detection model. The construction process is as follows: Figure 2 As shown, the specific implementation instructions are as follows:

[0053] 2a) Calculate the importance of features in the code representation of the target vulnerability detection model in the model decision-making process. Combining the target vulnerability detection model information and the prototype vulnerability code, apply local interpretation methods or other deep learning model interpretation methods to evaluate the classification results of the target vulnerability detection model and each feature F in the code representation. i The relationship between them is explained, and the feature F is calculated. i Importance value in the decision-making process of detection models For all features F i Importance value Perform normalization. (Go to 2b).

[0054] 2b) Based on the feature importance in the code representation of the target vulnerability detection model, obtain adversarial-sensitive features. Combine this with an importance threshold θ. I Filter out importance values Greater than the importance threshold θ I All features F i Construct an adversarial sensitive feature set SF = {SF1, SF2, ..., SF} mPreferably, in this embodiment, the importance threshold θ is... I Set it to 0.7. (Go to 2c).

[0055] 2c) Analyze the correspondence between adversarial sensitive features and code transformations to obtain candidate adversarial code transformations. Combine the code representation extraction rules of the target vulnerability detection model or other useful information, and use factor analysis or other analytical methods to obtain the SF for each adversarial sensitive feature. i The correspondence between the code transformations and the candidate adversarial code transformation set P = {P1, P2, ..., P...} is used to obtain the candidate adversarial code transformation set P = {P1, P2, ..., P...}. o The candidate adversarial code transformation refers to one or more code transformations that can cause changes in adversarial-sensitive features, including but not limited to addition, deletion, and modification operations on code at the granularity of variables, statements, or basic blocks. (Go to 2d).

[0056] 2d) Based on the target constraints of the adversarial attack and the candidate adversarial code transformations, obtain the effective adversarial code transformations. Combining the target constraints of the adversarial attack, in the candidate adversarial code transformation set P = {P1, P2, ..., P...} o} is used to filter and obtain all candidate adversarial code transformations P that meet the constraints of the adversarial example attack target. i A set of effective adversarial code transformations VP = {VP1, VP2, ..., VP} is constructed. p The target constraints of the adversarial sample attack include, but are not limited to, forcing the target vulnerability detection model to misclassify, ensuring that the original vulnerability can be triggered, ensuring that the code correctly executes the original function, or that the code transformation only involves a limited code segment.

[0057] 3) Construct a deep reinforcement learning framework for generating optimal adversarial examples. The construction process is as follows: Figure 3 As shown, the specific implementation instructions are as follows:

[0058] 3a) Model the decision-making process for generating the optimal adversarial example using a Markov decision process model. Based on the definition of a Markov decision process model, the decision-making process for generating the optimal adversarial example is modeled as a multi-round decision-making process, i.e., in the t-th round of decision-making, the input prototype vulnerability code sample VC... seed After applying (t-1) effective adversarial code transformations, it is converted into a process vulnerability code sample TVC. t The process vulnerability code sample TVC needs to be reviewed. t Optimize decisions based on the effective adversarial code transformations applied. (Go to 3b).

[0059] 3b) Establish a deep reinforcement learning environment model for optimal adversarial example generation. The deep reinforcement learning environment model for optimal adversarial example generation includes a state space S, an action space A, and a reward function R. Preferably, in this embodiment, the embedding layer of the target vulnerability detection model is derived from the process vulnerability code sample TVC. t Extracted feature vectors Represents the state at the t-th round of decision-making. Utilizing the effective adversarial code transformation set VP = {VP1, VP2, ..., VP p Construct an action space A = {A1, A2, ..., A} p}, where each action A i Corresponding to an effective countermeasure code transformation VP i Defined in state S t Select action A below t The reward function R(S) t A t )as follows:

[0060]

[0061] Where, ΔP normal Indicates the application of action A t Before and after, the target vulnerability detection model will use process vulnerability code samples (TVCs) t The change in the probability value classified as having no vulnerability is represented by λ, which is an additive factor. Preferably, in this embodiment, the additive factor λ is set to 70. (Go to 3c).

[0062] 3c) Construct a DQN estimation network for decision-making within a deep reinforcement learning environment model. In state S... t Below, the expected function Q(S) fitted using the DQN estimation network Q is obtained. t A i Calculate each action A i The reward is used to select the optimal action A that maximizes the expected function value using the ε-greedy strategy. t Execution. Preferably, in this embodiment, the neural network structure of the DQN network is a convolutional neural network. (Go to 3d).

[0063] 3d) Initialize the dual DQN network training model. Randomly initialize the parameters φ of the evaluation network Q, and initialize the target network Q. target The parameter φ target =φ. Initialize the current training epoch loop = 0, the target network update epoch R, the upper limit of the training epoch G, and the memory unit size M N. Preferably, in this embodiment, the target network Q targetThe neural network structure is the same as the evaluation network Q. The number of update rounds R for the target network is set to 5, the upper limit of the number of training rounds G is set to 500, and the size N of the memory unit M is set to 16. (Go to 3e).

[0064] 3e) The decision data of the sampled estimation network Q is stored in the memory unit M. In the deep reinforcement learning environment model, the random sampled estimation network Q is in state S. t Action A of making a decision t The corresponding reward value R(S) t A t ) and execution A t The state S that is then transitioned to t+1 Construct a quadruple (S t A t ,R(S t A t ),S t+1 ) Add to memory unit. If the size of the decision data stored in the memory unit is less than the size N of the memory unit, continue sampling and return to step 3e). If the size of the decision data already stored in the memory unit is equal to the size N of the memory unit, replace the decision data in the memory unit using a first-in-first-out strategy and proceed to step 3f).

[0065] 3f) Train the valuation network Q using the loss function Loss. Use the decision use cases (S) in the memory unit M. t A t ,R(S t A t ),S t+1 During training, the corresponding loss function Loss is defined as follows:

[0066]

[0067] Wherein, γ represents the discount factor, and preferably, in this embodiment, the discount factor γ is set to 0.9. Combined with the loss function Loss, the gradient descent method Q←Q+βLoss is used to update the parameters φ of the estimated network Q, where β represents the learning rate. Preferably, in this embodiment, the learning rate β is set to 0.01. (Go to 3g).

[0068] 3g) Update target network Q target Determine if the current loop number satisfies loop mod R = 0. If so, update the target network Q using the parameters φ of the estimated network Q at the current loop number. target , so that φ target =φ, otherwise, do not update the target network Q. target (Switch to 3h).

[0069] 3h) Determine if the current epoch number loop is equal to the training epoch limit G. If yes, end the training; otherwise, set loop = loop + 1 and return to step 3e).

[0070] 4) Using an effective adversarial code transformation and a deep reinforcement learning framework for generating optimal adversarial examples, the optimal adversarial examples for the target vulnerability detection model are generated. The generation process is as follows: Figure 4 As shown, the specific implementation instructions are as follows:

[0071] 4a) Select prototype vulnerability code samples (VC) from the prototype vulnerability code set for adversarial sample generation. seed The current generation round number l = 0 and the upper limit of the generation round number Z are initialized. Preferably, in this embodiment, the upper limit of the generation round number Z is set to 20. (Go to 4b).

[0072] 4b) The prototype vulnerability code sample VC seed Input a deep reinforcement learning framework for generating optimal adversarial examples, denoted as the process vulnerability code sample TVC0 generated when the number of rounds l=0. (Go to 4c).

[0073] 4c) Utilize the DQN valuation network Q to select effective adversarial code transformations applied to the process vulnerability sample TVC in the current generation round l. l Received TVC l+1 (Go to 4d).

[0074] 4d) Determine if the condition for terminating the generation of the optimal adversarial example is met. If TVC l+1 If the target vulnerability detection model misclassifies the vulnerability, or if the current generation round l equals the upper limit of generation rounds Z, then end the generation process and proceed to step 4e). Otherwise, l = l + 1, and return to step 4c.

[0075] 4e) Sample TVC code for process vulnerability l+1 This serves as the optimal adversarial sample output for the target vulnerability detection model.

[0076] Based on the same inventive concept, another embodiment of the present invention provides a deep learning vulnerability detection model adversarial example generation system based on deep reinforcement learning, comprising:

[0077] The information acquisition module is used to acquire information about the target vulnerability detection model based on deep learning and a set of prototype vulnerability codes used to generate adversarial examples.

[0078] An effective code transformation build module is used to construct effective code transformation-resistant target vulnerability detection models;

[0079] The Deep Reinforcement Learning Framework Building Module is used to build a deep reinforcement learning framework for generating optimal adversarial examples.

[0080] The optimal adversarial sample generation module is used to generate optimal adversarial samples for the target vulnerability detection model using effective adversarial code transformation and a deep reinforcement learning framework oriented towards optimal adversarial sample generation.

[0081] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0082] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0083] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0084] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for generating adversarial examples based on a deep learning vulnerability detection model using deep reinforcement learning, comprising the following steps: Acquire information on deep learning-based target vulnerability detection models and a set of prototype vulnerability codes used to generate adversarial examples; Constructing an effective countermeasure against code transformation in a target-oriented vulnerability detection model; Construct a deep reinforcement learning framework for generating optimal adversarial examples; Using an effective adversarial code transformation and a deep reinforcement learning framework oriented towards the generation of optimal adversarial examples, the optimal adversarial examples for the target vulnerability detection model are generated. The effective adversarial code transformation for constructing the target-oriented vulnerability detection model includes: The importance of features in the code representation of a target vulnerability detection model in the model decision-making process; Based on the feature importance in the code representation of the target vulnerability detection model, obtain adversarial-sensitive features; Analyze the correspondence between adversarial sensitive features and code transformations to obtain candidate adversarial code transformations; Based on the target constraints of the adversarial example attack and the candidate adversarial code transformations, obtain effective adversarial code transformations; the step of obtaining effective adversarial code transformations based on the target constraints of the adversarial example attack and the candidate adversarial code transformations includes: combining the target constraints of the adversarial example attack, filtering in the candidate adversarial code transformation set, obtaining all candidate adversarial code transformations that meet the target constraints of the adversarial example attack, and obtaining the effective adversarial code transformation set. The construction of the deep reinforcement learning framework for optimal adversarial example generation includes: Model the decision-making process for generating optimal adversarial examples using a Markov decision process model; Establish a deep reinforcement learning environment model for generating optimal adversarial examples; Construct a DQN valuation network for decision-making within a deep reinforcement learning environment model; Train the DQN valuation network to obtain the DQN valuation network for generating optimal adversarial examples; The deep reinforcement learning framework, which uses effective adversarial code transformation and optimal adversarial sample generation, generates optimal adversarial samples for the target vulnerability detection model, including: We select prototype vulnerability code samples as input for generating adversarial examples and use a deep reinforcement learning framework for generating optimal adversarial examples. The DQN valuation network is used to select effective adversarial code transformations applied to prototype vulnerability code samples to obtain process vulnerability code samples. The determination process determines whether the vulnerability code sample can cause the target vulnerability detection model to misclassify it, or whether the current generation round is equal to the upper limit of the generation round; If so, the process vulnerability code sample will be used as the optimal adversarial sample output for the target vulnerability detection model; If not, continue using the DQN valuation network to select effective code samples that resist code transformations applied to the process vulnerability.

2. The method according to claim 1, characterized in that, The deep learning-based target vulnerability detection model information includes model instances capable of vulnerability detection, model structure information of the detection model, and code representation information used by the detection model; the prototype vulnerability code set used to generate adversarial samples includes vulnerability code samples that the target vulnerability detection model can correctly classify as having vulnerabilities.

3. The method according to claim 1, characterized in that, The methods for calculating the importance of features in the code representation of the target vulnerability detection model in the model decision-making process include local interpretation methods; the methods for analyzing the correspondence between adversarial sensitive features and code transformations include factor analysis.

4. The method according to claim 3, characterized in that, The candidate adversarial code transformation refers to one or more code transformations that can cause changes in adversarial sensitive features, including adding, deleting, or modifying code in variables, statements, or basic blocks.

5. A deep learning vulnerability detection model adversarial example generation system based on deep reinforcement learning, employing the method described in any one of claims 1 to 4, characterized in that, include: The information acquisition module is used to acquire information about the target vulnerability detection model based on deep learning and a set of prototype vulnerability codes used to generate adversarial examples. An effective code transformation build module is used to construct effective code transformation-resistant target vulnerability detection models; The Deep Reinforcement Learning Framework Building Module is used to build a deep reinforcement learning framework for generating optimal adversarial examples. The optimal adversarial sample generation module is used to generate optimal adversarial samples for the target vulnerability detection model using effective adversarial code transformation and a deep reinforcement learning framework oriented towards optimal adversarial sample generation.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 4.

Images