A code pre-training model unauthorized data detection method

Abstract

The application discloses a kind of code pre-training model unauthorized data detection method, comprising the following steps: S1, code data is sampled and preprocessed;S2, for the target model of code, train a shadow model to simulate the behavior of target model;S3, using signal extractor, extract the signal of the pre-training task of the target model and shadow model of code, for the loss of different pre-training tasks, as the feature input of subsequent weighted classification model, for subsequent inference;S4, using difficulty correction model, the signal output by signal extractor is corrected in difficulty, for making the signal output by signal extractor not be disturbed by sample difficulty;S5, using weighted classification model learns the output signal of signal extractor and the multiple signals corrected after difficulty correction model correction, and give whether the judgment of pre-training model uses certain unauthorized data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data detection, and in particular to a code pre-training model unauthorized data detection method. BACKGROUND

[0002] Recently, various code pre-training language models such as CodeBERT and Code Llama have emerged and shown strong capabilities. CPLMs are pre-trained on a large amount of code data publicly available on platforms such as GitHub and StackOverflow. CPLMs can be fine-tuned or directly used for code refactoring and code search and other code-related tasks. Even in the case of little downstream task data, it can reduce the intellectual burden of developers and promote software development and maintenance. However, training CPLMs using code data may cause patent infringement and legal violations. GitHub launched a programming tool called Copilot, which is driven by OpenAI Codex. However, since Copilot is trained on code that may have been collected from open source projects, Copilot faces allegations of violating open source licenses. Although it is still controversial whether using open source code to train code pre-training models will cause intellectual property infringement, lawsuits have reminded researchers and companies engaged in CPLMs research that code data from open source projects is not all free training data. Therefore, it is particularly important to identify whether a trained code pre-training model has used a specific unauthorized code snippet as its training data. SUMMARY

[0003] To solve the above problems, the present application provides a code pre-training model unauthorized data detection method, the purpose of which is to detect whether suspected infringing data has been used in the training process of a certain code pre-training model, that is, whether the data appears in the training data set of a certain code model; the present application first simulates the behavior of the target model through the shadow model, then extracts the loss of the shadow model for the training data through the signal extractor as the signal, and finally corrects the difficulty of the sample through the difficulty correction model, and finally gives a judgment through the weighted classification model whether a suspected infringement is used in the training task of a certain code model. Experiments show that the present application can better perform member reasoning tasks.

[0004] To achieve the above purpose, the present application adopts the following technical scheme:

[0005] A code pre-training model unauthorized data detection method, comprising the following steps:

[0006] S1, sampling and preprocessing code data;

[0007] S2, training a shadow model for the target model of the code to simulate the behavior of the target model;

[0008] S3, using a signal extractor to extract the signals of the target model of the code and the pre-training task of the shadow model, and using the loss of different pre-training tasks as the feature input of the subsequent weighted classification model, for subsequent reasoning;

[0009] S4, using a difficulty correction model to correct the signals output by the signal extractor, so that the signals output by the signal extractor are not disturbed by the difficulty of the samples;

[0010] S5, using a weighted classification model to learn the output signals of the signal extractor and the multiple signals corrected by the difficulty correction model, and giving a judgment of whether the pre-training model has used a certain unauthorized data.

[0011] Preferably, the sampling process of the code data in step S1 is: assuming that the code data is subject to distribution D, the distribution D represents the entire sample of the code data, the shadow data set Ds is sampled from the entire sample of the code data for constructing the shadow model, and the difficulty correction model data set Dc is sampled from the entire sample of the code data for constructing the difficulty correction model.

[0012] Preferably, the preprocessing process of the code data in step S1 is: removing the noise in the code data, including removing the original comments in the code and the original strings in the code, and replacing them with special symbols; wherein the special symbol is STR.

[0013] Preferably, in step S2, the shadow model resamples the shadow data set from the entire code data set and is trained through the same task as the target model to simulate the behavior of the target model; wherein the shadow model is trained by sampling a shadow data set, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black box environment and infer the member state of the code.

[0014] Preferably, in step S3, the code data is input into the code model, and the signal output of the code model is obtained through the signal extractor, and the signals for different pre-training tasks are obtained, and the signal extractor is used to capture the member data or non-member data.

[0015] Preferably, in step S4, the difficulty correction model uses the same model architecture as the target model, the difficulty correction model is used to determine the difficulty of the sample, and a data set that does not intersect with the shadow model data set and the target model data set is used for pre-training, the pre-training task is the same as the target model, and the specific calculation formula of the correction process is:

[0016]

[0017] wherein, represents the final calibration value of the i-th sample; represents the signal of the target model or the shadow model; represents the signal of the difficulty correction model; and The sum of the two is not equal to zero.

[0018] Preferably, in step S5, a weighted classification model based on the code signal is established, the weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks; wherein, a self-attention mechanism network is used to learn a generative task, a multi-layer perception is used to learn a non-generative task, and the confidence score of the sub-network is weighted by the multi-layer perception to obtain a final confidence score.

[0019] Preferably, in step S5, the calculation formula of the loss function of the weighted classification model is:

[0020] L(θ,m,n)=α-C(m)+C(n),

[0021] wherein, L(θ,m,n) is a loss function of the weighted classification model, which is used to fit the model parameters according to the member samples and the non-member samples; θ is a model parameter; m is a member sample; n is a non-member sample; C(m) is an output of the weighted classification model for the member sample; C(n) is an output of the weighted classification model for the non-member sample; and α is a hyperparameter.

[0022] After the above technical solution is adopted, the present application has the following beneficial effects:

[0023] 1. The purpose of the present application is to detect whether the suspected infringing data is used in the training process of a certain code pre-training model, i.e., whether the data appears in the training data set of a certain code model; the present application first simulates the behavior of the target model through the shadow model, then extracts the loss of the shadow model for the training data as a signal through the signal extractor, and finally corrects the difficulty of the sample through the difficulty correction model, and finally gives a judgment on whether a suspected infringement is used in the training task of a certain code model through the weighted classification model. Experiments show that the present application can better perform the member reasoning task.

[0024] 2. The present application fills the gap of the code member reasoning task, and defines two reasoning levels for the code pre-training language model: white-box reasoning and black-box reasoning. The present application applies various reasoning techniques, including signal extraction from pre-training tasks, difficult sample calibration and weighted reasoning, to accurately identify the code member state.

[0025] 3. The accuracy of this invention in black-box reasoning is not much different from that in white-box reasoning, indicating that the method for detecting unauthorized data in the code pre-training model of this invention has high accuracy in practice and has significant practical value and application prospects. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention;

[0027] Figure 2 This is a flowchart of the present invention;

[0028] Figure 3 This is a schematic diagram of the signal extractor of the present invention;

[0029] Figure 4 This is an analysis diagram of the difficulty correction model of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] like Figures 1 to 4 As shown, a method for detecting unauthorized data in a code pre-trained model includes the following steps:

[0032] S1. Sample and preprocess the code data;

[0033] The sampling process for code data in step S1 is as follows: Assuming that the code data follows a distribution D, where distribution D represents all samples of code data, a shadow dataset Ds is sampled from all samples of code data to construct a shadow model, and a difficulty correction model dataset Dc is sampled from all samples of code data to construct a difficulty correction model.

[0034] The preprocessing of code data in step S1 is as follows: remove noise from the code data, including removing the original comments and strings in the original code, and replacing them with special symbols; where the special symbols are STR.

[0035] S2. For the target model in the code, train a shadow model to simulate the behavior of the target model;

[0036] The shadow model described in step S2 is resampled from the full code dataset and trained on the same task as the target model to simulate the behavior of the target model; wherein, by sampling a shadow dataset to train the shadow model, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black-box environment to infer the member state of the code;

[0037] S3, a signal extractor is used to extract signals of the target model and the shadow model of the code, and the loss of different pre-training tasks is used as the feature input of the subsequent weighted classification model, for subsequent reasoning;

[0038] In step S3, the code data is input into the code model, and the signal output of the code model is obtained through the signal extractor, and signals for different pre-training tasks are obtained. The signal extractor is used to capture member data or non-member data;

[0039] S4, a difficulty correction model is used to correct the signal output by the signal extractor, so that the signal output by the signal extractor is not disturbed by the difficulty of the sample;

[0040] In step S4, the difficulty correction model uses the same model architecture as the target model. The difficulty correction model is used to determine the difficulty of the sample, and is pre-trained using a data set that does not intersect with the shadow model data set and the target model data set. The pre-training task is the same as the target model. The specific calculation formula of the correction process is:

[0041]

[0042] Wherein, represents the final calibration value of the i-th sample; represents the signal of the target model or the shadow model; represents the signal of the difficulty correction model; and The sum is not equal to zero.

[0043] S5, a weighted classification model is used to learn the output signal of the signal extractor and the corrected multiple signals of the difficulty correction model, and to give a judgment on whether the pre-training model has used a certain unauthorized data;

[0044] In step S5, a weighted classification model based on code signals is established. The weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks. Through a self-attention mechanism network, a generative task is learned, and through a multi-layer perception, a non-generative task is learned. The confidence score of the sub-network is weighted by a multi-layer perception to obtain the final confidence score.

[0045] In step S5, the calculation formula of the loss function of the weighted classification model is:

[0046] L(θ,m,n)=α-C(m)+C(n),

[0047] wherein L(0, m, n) is a loss function of the weighted classification model, the model parameters are fitted according to the member samples and the non-member samples; 0 is the model parameter; m is the member sample; n is the non-member sample; C(m) is the output of the weighted classification model for the member sample; C(n) is the output of the weighted classification model for the non-member sample; a is a hyperparameter.

[0048] Performance test:

[0049] The method selects four representative code pre-training models as target models, including CodeBERT(Codebert: A pre-trained model for programming and natural languages), CodeT5(Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.), Deepseek-Coder(Deepseek-coder: When the large language model meets programming-the rise of code intelligence) and CodeLlama(Code llama: Open foundation models for code.). CodeBERT is a BERT-based dual-modal CPLM. CodeT5 is an encoder-decoder-based CPLM. DeepseekCoder is a decoder-only model, and the present application adopts deepseek-coder-1.3b-base with 1.3 billion parameters. CodeLlama is a decoder-only model based on Llama 2, and the present application adopts codellama-7b-base with 7 billion parameters. For CodeBERT and CodeT5, the present application pre-trains them from scratch to generate target models, and the generated CodeBERT has 125 million parameters, and CodeT5 has 220 million parameters. For larger CPLMs DeepseekCoder and CodeLlama, the method continues to train based on the models published by them to generate target models.

[0050] For CodeBERT and CodeT5, we select the CSN dataset (Code-searchnet challenge: Evaluating the state of semantic code search.) because their authors pretrain CodeBERT and CodeT5 on CSN. The CSN dataset contains over 6 million code snippets from open-source projects on GitHub, covering six programming languages (Python, Java, JavaScript, Go, Ruby, and PHP). Due to limited computational resources, we only use Python code snippets in CSN. Code snippets are associated with metadata such as code descriptions written in natural language. We extract disjoint passages from CSN to pretrain the target model, shadow model, and difficulty correction model. Specifically, we extract 100,000 data records for pretraining the target model, 50,000 for pretraining the shadow model and difficulty correction model. For testing, we extract 10,000 member data records and 10,000 non-member data records.

[0051] For DeepseekCoder and CodeLlama, we use the Magicoder-Evol-Instruct-110k (MEI) dataset (Magicoder: Source code is all you need). Note that MEI is generated by GPT-4, which can avoid data leakage issues. Specifically, we extract 30,000 data records from MEI for training the target model, 20,000 for training the shadow model and calibration model. For testing, we extract 5,000 member data records and 5,000 non-member data records.

[0052] We run experiments on a machine equipped with two Intel(R) Xeon(R) Silver 4214R CPUs @ 2.40GHz, 256GB main memory, and eight NVIDIA GeForce RTX 3090s. Since the pretraining implementation of CodeBERT and CodeT5 is not public, we implement these models according to their original papers. For training CodeLlama, we use deepspeed and ZERO optimization and CPU offloading (Zero: memory optimizations toward training trillion parameter models.). We set the batch size to 64.

[0053] The present application can be used for code unauthorized training data detection. For a suspected infringing data, the model behavior is obtained through the shadow model, the signal is obtained through the extractor, and finally the final prediction is given through the inference model. Table 1 is the experimental results of four code pre-training models.

[0054] Table 1: Experimental results of four code pre-training models

[0055]

[0056] In table 1, the abbreviations bb and wb represent black-box inference and white-box inference, respectively. Perb and ppl represent perturbation and perplexity, respectively. From table 1, the following conclusions can be drawn: member data and non-member data cannot be easily separated according to code characteristics (i.e., data distribution), and the results of FastText show this: its AUC score is close to 0.5.

[0057] The present application exhibits superior performance over the baseline FastText, perturbation and perplexity, showing that the present application can meet the needs of code membership inference to some extent.

[0058] The AUC of white-box inference is not much higher than that of black-box inference. Therefore, the distribution of the target model training data set has a relatively small impact on inference accuracy. In other words, high-precision CMI can be achieved in a black-box setting.

[0059] The AUC scores of CodeT5 (about 0.8), DeepseekCoder (about 0.7) and CodeLlama (about 0.9) are much higher than that of CodeBERT (about 0.6), mainly due to their different model structures and parameter sizes. In order to investigate whether the present method is affected by high false positive rate, the true positive rate (TPR) under different false positive rates (0.01%, 0.1%, 1%) is shown in table 1. The present method overcomes the problem of high false positive rate, which is a common problem in existing membership inference (MI) work (Watson et al., 2022), because at low false positive rate, the TPR of BUZZER is much higher than that of the baseline. The present application shows higher TPR on larger CPLMs DeepseekCoder and CodeLlama.

[0060] Table 2 shows the ablation experiment results with and without the difficulty correction model.

[0061] Table 2: Ablation experiment results with and without the difficulty correction model

[0062]

[0063] Table 2 can show the inference results using and not using the difficulty correction model. Apparently, the difficulty correction model can significantly improve the CMI performance. Specifically, for CodeBERT, it improves the AUC score by 0.08, and for CodeT5, it improves the AUC score by 0.137. For DeepseekCoder and CodeLlama, the calibration has a more significant impact on the black-box inference. The difficulty correction model is effective in both white-box and black-box inference.

[0064] In Figure 4 , the lowest score of the calibrated member data is about 0.52 ((the lower curve in (a)), while the highest score of the non-member data after calibration is about 0.41 ((the lower curve in (b)), and the gap after calibration is 0.11. The lowest score of the uncalibrated member data is about 0.62 ((the upper curve in (a)), while the highest score of the non-member data when uncalibrated is about 0.59 ((the upper curve in (b)), and the uncalibrated gap is 0.03, which is smaller than the calibrated gap (0.11). Therefore, it can be concluded that calibration increases the gap between member and non-member data, thereby improving the AUC.

[0065] The above merely provides the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting unauthorized data of a code pre-training model, characterized in that, The method comprises the following steps: S1, sampling and preprocessing code data; S2, training a shadow model for the target model of the code to simulate the behavior of the target model; In step S2, the shadow model resamples the shadow data set from the entire code data set, and is trained through the same task as the target model to simulate the behavior of the target model; wherein, by sampling a shadow data set to train the shadow model, the member data and non-member data of the shadow model are known, and the shadow model is used to replace the target model in a black box environment and infer the member state of the code; S3, using a signal extractor to extract the signals of the pre-training tasks of the target model and the shadow model of the code, and using the loss of different pre-training tasks as the feature input of the subsequent weighted classification model for subsequent reasoning; S4, using a difficulty correction model to correct the signals output by the signal extractor, so as to prevent the signals output by the signal extractor from being disturbed by the difficulty of the sample; In step S4, the difficulty correction model uses the same model architecture as the target model, and the difficulty correction model is used to determine the difficulty of the sample and is pre-trained using a data set that does not intersect with the shadow model data set and the target model data set, the pre-training task is the same as the target model, and the specific calculation formula of the correction process is: wherein, represents the final calibration value of the i-th sample; represents the signal of the target model or the shadow model; represents the signal of the difficulty correction model; ∈ and the sum is not equal to zero; S5, using a weighted classification model to learn the output signals of the signal extractor and the multiple signals corrected by the difficulty correction model, and giving a judgment whether the pre-training model has used a certain unauthorized data; In step S5, the calculation formula of the loss function of the weighted classification model is: L(θ,m,n)=α-C(m)+C(n), Wherein, L(θ,m,n) is the loss function of the weighted classification model, which is used to fit the model parameters according to the member samples and non-member samples; θ is the model parameter; m is the member sample; n is the non-member sample; C(m) is the output of the weighted classification model for the member sample; C(n) is the output of the weighted classification model for the non-member sample; α is a hyperparameter. 2.The method of claim 1, wherein, In step S1, the sampling process of the code data is as follows: assuming that the code data obeys distribution D, distribution D represents all samples of the code data, a shadow data set Ds is sampled from all samples of the code data for constructing a shadow model, and a difficulty correction model data set Dc is sampled from all samples of the code data for constructing a difficulty correction model. 3.The method of claim 1, wherein, In step S1, the preprocessing process of the code data is as follows: removing the noise in the code data, including removing the original comments in the code and the original strings in the code, and replacing them with special symbols; wherein, the special symbol is STR. 4.The method of claim 1, wherein: In step S3, the code data is input into the code model, the signal output of the code model is obtained through the signal extractor, the signals for different pre-training tasks are obtained, and the signal extractor is used to capture the member data or non-member data.

5. The method of claim 1, wherein the code pre-training model is a BERT model. In step S5, a weighted classification model based on code signals is established, the weighted classification model includes different types of sub-networks for processing signals given by different types of code pre-training tasks;Wherein, the self-attention mechanism network is used to learn the generative task, and the multilayer perception is used to learn the non-generative task, and the confidence score of the sub-network is weighted by the multilayer perception to obtain the final confidence score.

Images