Method for detecting unauthorized data of code pre-training model

Abstract

The invention discloses a method for detecting unauthorized data of a code pre-training model. The method comprises the following steps: S1, sampling and preprocessing code data; s2, aiming at the target model of the code, training a shadow model to simulate the behavior of the target model; s3, adopting a signal extractor to extract signals of pre-training tasks of a target model and a shadow model of the code, and aiming at the loss of different pre-training tasks, taking the signals as feature input of a subsequent weighted classification model for subsequent reasoning; s4, performing difficulty correction on the signal output by the signal extractor by adopting a difficulty correction model so as to enable the signal output by the signal extractor not to be interfered by the sample difficulty; and S5, learning an output signal of the signal extractor and various signals corrected by the difficulty correction model by adopting a weighted classification model, and giving a judgment on whether the pre-training model uses a certain piece of unauthorized data or not.

Description

Technical Field

[0001] The present invention relates to the technical field of data detection, and in particular to a method for detecting unauthorized data in a code pre-training model. Background Art

[0002] Recently, a variety of code pre-trained language models, such as CodeBERT and Code Llama, have emerged and demonstrated powerful capabilities. CPLMs are pre-trained on a large amount of code data publicly available on platforms such as GitHub and StackOverflow. CPLMs can be fine-tuned or directly used for code-related tasks such as code refactoring and code search, which can reduce the intellectual burden of developers and promote software development and maintenance even when there is not much data for downstream tasks. However, using code data to train CPLMs may cause patent infringement and legal violations. GitHub launched a programming tool called Copilot, driven by OpenAI Codex, however, because Copilot is trained on code that may be collected from open source projects, Copilot faces allegations of violating open source licenses. Although it is still controversial whether using open source code to train code pre-trained models will cause intellectual property infringement, the lawsuit has reminded researchers and companies working on CPLMs that code data from open source projects is not always free training data. Therefore, it is particularly important to identify whether a trained code pre-trained model uses a specific unauthorized code snippet as its training data. Summary of the invention

[0003] To solve the above problems, the present invention proposes a method for detecting unauthorized data in a code pre-training model. The purpose of the method is to detect whether the suspected infringing data is used in the training process of a certain code pre-training model, that is, whether the data appears in the training data set of a certain code model; the present invention first simulates the behavior of the target model through a shadow model, and then extracts the loss of the shadow model for the training data through a signal extractor as a signal, and then corrects the difficulty of the sample through a difficulty correction model, and finally gives a weighted classification model to determine whether a suspected infringement is used in the training task of a certain code model. Experiments show that the present invention can perform member reasoning tasks well.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for detecting unauthorized data in a code pre-training model comprises the following steps:

[0006] S1. Sampling and preprocessing code data;

[0007] S2. For the target model of the code, train a shadow model to simulate the behavior of the target model;

[0008] S3, using a signal extractor to extract the signals of the pre-training tasks of the target model and the shadow model of the code, and use the losses of different pre-training tasks as feature inputs of the subsequent weighted classification model for subsequent reasoning;

[0009] S4, using a difficulty correction model to perform difficulty correction on the signal output by the signal extractor, so as to prevent the signal output by the signal extractor from being interfered by the difficulty of the sample;

[0010] S5. Use a weighted classification model to learn the output signal of the signal extractor and the various signals corrected by the difficulty correction model, and give a judgment on whether the pre-trained model has used a certain unauthorized data.

[0011] Preferably, the sampling process of the code data in step S1 is: assuming that the code data obeys distribution D, distribution D represents all samples of the code data, a shadow data set Ds is sampled from distribution D to construct a shadow model, and a difficulty correction model data set Dc is sampled from all samples of the code data to construct a difficulty correction model.

[0012] Preferably, the preprocessing process of the code data in step S1 is: removing noise in the code data, including removing original comments in the code and character strings in the original code, and uniformly replacing them with special symbols; wherein the special symbol is STR.

[0013] Preferably, the shadow model in step S2 resamples a sub-dataset from the entire code dataset and is trained through the same task as the target model to simulate the behavior of the target model; wherein, a shadow dataset is sampled to train the shadow model, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black box environment and infer the member state of the code.

[0014] Preferably, in step S3, the code data is input into the code model, and the signal output of the code model is obtained through a signal extractor to obtain signals for different pre-training tasks. The signal extractor is used to capture member data or non-member data.

[0015] Preferably, the difficulty correction model in step S4 adopts the same model architecture as the target model. The difficulty correction model is used to determine the difficulty of the sample and is pre-trained using a data set that does not intersect with the shadow model data set and the target model data set. The pre-training task is the same as the target model. The specific calculation formula of the correction process is:

[0016]

[0017] in, represents the final calibration value of the i-th sample; a signal representing the target model or the shadow model; Represents the signal of the difficulty correction model; ∈ and Does not add up to zero.

[0018] Preferably, in step S5, a weighted classification model based on the code signal is established, and the weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks; wherein, generative tasks are learned through a self-attention mechanism network, and non-generative tasks are learned through a multi-layer perceptron, and the confidence scores of the sub-networks are weighted by the multi-layer perceptron to obtain the final confidence scores.

[0019] Preferably, in step S5, the calculation formula of the loss function of the weighted classification model is:

[0020] L(θ,m,n)=α-C(m)+C(n),

[0021] Among them, L(θ,m,n) is the loss function of the weighted classification model, which fits the model parameters according to the member samples and non-member samples; θ is the model parameter; m is the member sample; n is the non-member sample; C(m) is the output of the weighted classification model for the member sample; C(n) is the output of the weighted classification model for the non-member sample; α is a hyperparameter.

[0022] After adopting the above technical solution, the present invention has the following beneficial effects:

[0023] 1. The purpose of the present invention is to detect whether suspected infringing data is used in the training process of a certain code pre-training model, that is, whether the data appears in the training data set of a certain code model; the present invention first simulates the behavior of the target model through a shadow model, and then extracts the loss of the shadow model for the training data through a signal extractor as a signal, and then corrects the difficulty of the sample through a difficulty correction model, and finally gives a weighted classification model to determine whether a suspected infringement is used in the training task of a certain code model. Experiments show that the present invention can perform member reasoning tasks well.

[0024] 2. This paper fills the gap in code membership reasoning tasks and defines two reasoning levels for code pre-training language models: white-box reasoning and black-box reasoning. This paper applies various reasoning techniques, including signal extraction from pre-training tasks, hard sample calibration, and weighted reasoning, to accurately identify code membership states.

[0025] 3. The accuracy of the present invention in black box reasoning is not much different from that in white box reasoning, indicating that the unauthorized data detection method of the code pre-training model of the present invention has high accuracy in practice and has significant practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 It is a flowchart of the present invention;

[0028] Figure 3 is a schematic diagram of a signal extractor of the present invention;

[0029] Figure 4 This is an analysis diagram of the difficulty correction model of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] like Figures 1 to 4 As shown, a method for detecting unauthorized data of a code pre-training model includes the following steps:

[0032] S1. Sampling and preprocessing code data;

[0033] The sampling process of the code data in step S1 is as follows: assuming that the code data obeys distribution D, distribution D represents all samples of the code data, a shadow data set Ds is sampled from all samples of the code data to construct a shadow model, and a difficulty correction model data set Dc is sampled from all samples of the code data to construct a difficulty correction model;

[0034] The preprocessing process of the code data in step S1 is: removing noise in the code data, including removing the original comments in the code and the character strings in the original code, and uniformly replacing them with special symbols; wherein the special symbol is STR;

[0035] S2. For the target model of the code, train a shadow model to simulate the behavior of the target model;

[0036] The shadow model in step S2 resamples a sub-dataset from the entire code dataset and is trained through the same task as the target model to simulate the behavior of the target model; wherein a shadow dataset is sampled to train the shadow model, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black box environment and infer the member state of the code;

[0037] S3, using a signal extractor to extract the signals of the pre-training tasks of the target model and the shadow model of the code, and use the losses of different pre-training tasks as feature inputs of the subsequent weighted classification model for subsequent reasoning;

[0038] In step S3, the code data is input into the code model, and the signal output of the code model is obtained by a signal extractor to obtain signals for different pre-training tasks, and the signal extractor is used to capture member data or non-member data;

[0039] S4, using a difficulty correction model to perform difficulty correction on the signal output by the signal extractor, so as to prevent the signal output by the signal extractor from being interfered by the difficulty of the sample;

[0040] The difficulty correction model in step S4 adopts the same model architecture as the target model. The difficulty correction model is used to determine the difficulty of the sample and is pre-trained using a data set that does not intersect with the shadow model data set and the target model data set. The pre-training task is the same as the target model. The specific calculation formula of the correction process is:

[0041]

[0042] in, represents the final calibration value of the i-th sample; a signal representing the target model or the shadow model; Represents the signal of the difficulty correction model; ∈ and The sum does not equal zero;

[0043] S5. Use a weighted classification model to learn the output signal of the signal extractor and the various signals corrected by the difficulty correction model, and give a judgment on whether the pre-trained model has used a certain piece of unauthorized data;

[0044] In step S5, a weighted classification model based on code signals is established, and the weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks; wherein, generative tasks are learned through a self-attention mechanism network, and non-generative tasks are learned through a multi-layer perceptron, and the confidence scores of the sub-networks are weighted by the multi-layer perceptron to obtain a final confidence score;

[0045] In step S5, the calculation formula of the loss function of the weighted classification model is:

[0046] L(θ,m,n)=α-C(m)+C(n),

[0047] Among them, L(θ,m,n) is the loss function of the weighted classification model, which fits the model parameters according to the member samples and non-member samples; θ is the model parameter; m is the member sample; n is the non-member sample; C(m) is the output of the weighted classification model for the member sample; C(n) is the output of the weighted classification model for the non-member sample; α is a hyperparameter.

[0048] Performance Test:

[0049] This method selects four representative code pre-training models as target models, including CodeBERT (Codebert: A pre-trained model for programming and natural languages), CodeT5 (Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.), Deepseek-Coder (Deepseek-coder: When the large language model meets programming-the rise of code intelligence) and CodeLlama (Code llama: Open foundation models for code.). CodeBERT is a bimodal CPLM based on BERT. CodeT5 is a CPLM based on encoder-decoder. DeepseekCoder is a decoder-only model, and the present invention adopts deepseek-coder-1.3b-base with 1.3 billion parameters. CodeLlama is a decoder-only model based on Llama 2, and the present invention adopts codellama-7b-base with 7 billion parameters. For CodeBERT and CodeT5, the present invention pre-trains them from scratch to generate target models, and the generated CodeBERT has 125 million parameters and CodeT5 has 220 million parameters. For the larger CPLMs DeepseekCoder and CodeLlama, the present method continues to train based on their published models to generate target models.

[0050] For CodeBERT and CodeT5, the CSN dataset (Code-searchnet challenge: Evaluating the state of semantic code search.) was selected because their authors pre-trained CodeBERT and CodeT5 on CSN. The CSN dataset contains more than 6 million code snippets from open source projects on GitHub, covering six programming languages ​​(Python, Java, JavaScript, Go, Ruby, and PHP). Due to limited computing resources, the present invention only uses Python code snippets in CSN. The code snippets are associated with metadata such as code descriptions written in natural language. The present invention extracts non-overlapping paragraphs from CSN to pre-train the target model, shadow model, and difficulty correction model. Specifically, the present invention extracts 100,000 data records for pre-training the target model, and 50,000 for pre-training the shadow model and difficulty correction model. For testing, the present invention extracts 10,000 member data records and 10,000 non-member data records.

[0051] For DeepseekCoder and CodeLlama, the present invention uses the Magicoder-Evol-Instruct-110k (MEI) dataset (Magicoder: Source code is all you need). It should be noted that MEI is generated by GPT-4, which can avoid data leakage problems. Specifically, the present invention extracts 30,000 data records from MEI for training the target model, and 20,000 for training the shadow model and calibration model. For testing, the present invention extracts 5,000 member data records and 5,000 non-member data records.

[0052] The present invention runs experiments on a machine equipped with two Intel(R) Xeon(R) Silver 4214R CPUs @ 2.40GHz, 256GB main memory, and eight NVIDIA GeForce RTX 3090. Since the pre-training implementations of CodeBERT and CodeT5 are not public, the present invention implements these models according to their original papers. For training CodeLlama, the present invention uses deepspeed and ZERO optimizations and CPU offloading (Zero: memory optimizations toward trainingtrillion parameter models.). The present invention sets the batch size to 64.

[0053] The present invention can be used to detect unauthorized code training data. For a piece of suspected infringing data, the model behavior is obtained through the shadow model, the signal is obtained through the extractor, and the final prediction is given through the inference model. Table 1 shows the experimental results of four code pre-training models.

[0054] Table 1: Experimental results of four code pre-training models

[0055]

[0056] In Table 1, the abbreviations bb and wb represent black-box reasoning and white-box reasoning, respectively. perb and ppl represent perturbation and confusion, respectively. From Table 1, we can draw the following conclusions: member data and non-member data cannot be easily separated based on code features (i.e., data distribution), and the results of FastText show this: its AUC score is close to 0.5.

[0057] The present invention exhibits superior performance than the baseline FastText, perturbation and perplexity, showing that the present invention can meet the needs of code membership reasoning to a certain extent.

[0058] The AUC of white-box reasoning is not much higher than that of black-box reasoning. Therefore, the distribution of the target model training dataset has relatively little impact on the inference accuracy. In other words, high-accuracy CMI can be achieved in the black-box setting.

[0059] The AUC scores of CodeT5 (about 0.8), DeepseekCoder (about 0.7), and CodeLlama (about 0.9) are much higher than CodeBERT (about 0.6), mainly due to their different model structures and parameter sizes. To investigate whether our method is affected by high false alarm rates, the true positive rate (TPR) at different false alarm rates (0.01%, 0.1%, 1%) is shown in Table 1. Our method overcomes the problem of high false alarm rates, which is a common problem in existing membership inference (MI) work (Watson et al., 2022), because at low false alarm rates, BUZZER's TPR is much higher than the baseline's TPR. Our method shows higher TPR on larger CPLMs DeepseekCoder and CodeLlama.

[0060] Table 2 shows the ablation experiment results with and without the difficulty correction model.

[0061] Table 2: Ablation experiment results with and without difficulty correction model

[0062]

[0063] Table 2 shows the inference results with and without the difficulty correction model. Obviously, the difficulty correction model can significantly improve the CMI performance. Specifically, for CodeBERT, it improves the AUC score by 0.08 and for CodeT5 by 0.137. For DeepseekCoder and CodeLlama, the impact of calibration on black-box reasoning is more significant. The difficulty correction model is effective in both white-box reasoning and black-box reasoning.

[0064] exist Figure 4 In the figure, the lowest score of the calibrated member data is about 0.52 (lower curve in (a)), while the highest score of the calibrated non-member data is about 0.41 (lower curve in (b)), and the gap after calibration is 0.11. The lowest score of the uncalibrated member data is about 0.62 (upper curve in (a)), while the highest score of the non-member data without calibration is about 0.59 (upper curve in (b)), and the gap without calibration is 0.03, which is smaller than the gap after calibration (0.11). Therefore, it can be concluded that calibration increases the gap between member and non-member data, thereby improving AUC.

[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for detecting unauthorized data in a code pre-training model, characterized in that: The following steps are involved: S1. Sampling and preprocessing code data; S2. For the target model of the code, train a shadow model to simulate the behavior of the target model; S3, using a signal extractor to extract the signals of the pre-training tasks of the target model and the shadow model of the code, and use the losses of different pre-training tasks as feature inputs of the subsequent weighted classification model for subsequent reasoning; S4, using a difficulty correction model to perform difficulty correction on the signal output by the signal extractor, so as to prevent the signal output by the signal extractor from being interfered by the difficulty of the sample; S5. Use a weighted classification model to learn the output signal of the signal extractor and the various signals corrected by the difficulty correction model, and give a judgment on whether the pre-trained model has used a certain unauthorized data.

2. A method for detecting unauthorized data in a code pre-training model according to claim 1, characterized in that: The sampling process of the code data in step S1 is: assuming that the code data obeys distribution D, distribution D represents all samples of the code data, a shadow data set Ds is sampled from all samples of the code data to construct a shadow model, and a difficulty correction model data set Dc is sampled from all samples of the code data to construct a difficulty correction model.

3. The method for detecting unauthorized data in a code pre-training model according to claim 1, characterized in that: The preprocessing process of the code data in step S1 is: removing noise in the code data, including removing the original comments in the code and the character strings in the original code, and uniformly replacing them with special symbols; wherein the special symbol is STR.

4. A method for detecting unauthorized data in a code pre-training model as claimed in claim 1, characterized in that: The shadow model in step S2 resamples a sub-dataset from the entire code dataset and is trained through the same task as the target model to simulate the behavior of the target model; wherein, a shadow dataset is sampled to train the shadow model, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black box environment and infer the member status of the code.

5. The method for detecting unauthorized data in a code pre-training model according to claim 1, characterized in that: In step S3, the code data is input into the code model, and the signal output of the code model is obtained through a signal extractor to obtain signals for different pre-training tasks. The signal extractor is used to capture member data or non-member data.

6. A method for detecting unauthorized data in a code pre-training model according to claim 1, characterized in that: The difficulty correction model in step S4 adopts the same model architecture as the target model. The difficulty correction model is used to determine the difficulty of the sample and is pre-trained using a data set that does not intersect with the shadow model data set and the target model data set. The pre-training task is the same as the target model. The specific calculation formula of the correction process is: in, represents the final calibration value of the i-th sample; a signal representing the target model or the shadow model; Represents the signal of the difficulty correction model; ∈ and Does not add up to zero.

7. A method for detecting unauthorized data in a code pre-training model according to claim 1, characterized in that: In step S5, a weighted classification model based on code signals is established. The weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks; wherein, generative tasks are learned through a self-attention mechanism network, and non-generative tasks are learned through a multi-layer perceptron. The confidence scores of the sub-networks are weighted by the multi-layer perceptron to obtain the final confidence scores.

8. A method for detecting unauthorized data in a code pre-training model as claimed in claim 1, characterized in that: In step S5, the calculation formula of the loss function of the weighted classification model is: L(θ,m,n)=α-C(m)+C(n), Among them, L(θ,m,n) is the loss function of the weighted classification model, which fits the model parameters according to the member samples and non-member samples; θ is the model parameter; m is the member sample; n is the non-member sample; C(m) is the output of the weighted classification model for the member sample; C(n) is the output of the weighted classification model for the non-member sample; α is a hyperparameter.

Images