Data leakage detection method based on semantic preserving disturbance and related device

Abstract

The invention provides a data leakage detection method and device based on semantic preserving disturbance, electronic equipment and a storage medium, and relates to the field of data processing. The method comprises the following steps: acquiring a to-be-detected code evaluation sample, and analyzing the code evaluation sample to obtain a structured sample; calling a preset disturbance operator set to execute semantic preserving type transformation on the structured samples, and generating a sample set containing original samples and variant samples; inputting each sample in the sample set into the detection model, and calculating the generation confusion degree of each sample; comparing and analyzing the generation confusion degree of each sample to obtain a comparison result; and outputting a leakage detection conclusion according to a comparison result, and recording a leakage confidence indication index. Therefore, according to the scheme of the invention, the memory trace of the disturbance amplification model is kept by utilizing code semantics, and a judgment mechanism without a threshold value is constructed in combination with confusion degree relative comparison, so that the leakage risk of the code evaluation sample can be detected under a complete black box condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular, the present application relates to a data leakage detection method and device based on semantic preservation disturbance, an electronic device and a storage medium. BACKGROUND

[0002] With the development of Transformer architecture and large-scale pre-training technology, Code Large Language Models (Code LLMs) have become an important tool in the field of software engineering and are widely used in automatic code completion, program synthesis, defect repair, and algorithm generation, etc. Currently, mainstream Code LLMs are generally divided into two categories: one is the encoder-decoder architecture model, such as CodeT5, CodeT5+, etc. This type of model is based on sequence-to-sequence modeling and is good at code generation and code understanding tasks; the other is the decoder-only architecture model, such as StarCoder, CodeLlama, etc. This type of model has stronger general generation ability and context reasoning ability through large-scale self-recurrent pre-training. The above models are usually pre-trained based on massive open source code corpus, and further improve the alignment ability to natural language instructions and the completion quality of complex code tasks through instruction fine-tuning, reinforcement learning optimization, and retrieval enhancement, etc.

[0003] An accurate, fair, and operable data leakage detection method is urgently needed in the existing code large model evaluation to effectively identify whether the model has memorized or indirectly utilized the training samples in the evaluation benchmark. Existing methods based on text memory analysis cannot depict the structured input and output, algorithm patterns, and program structure leakage forms specific to the code field, while the strategies based on dataset overlap comparison or release time alignment have serious limitations due to incomplete access to pre-training data and unreliable time information. Therefore, the existing methods can only provide incomplete or weak proxy leakage judgment, which is difficult to support fair evaluation of code large models.

[0004] As can be seen from the above, there is an urgent need for a data leakage detection method based on semantic preservation disturbance to solve the above problems. SUMMARY

[0005] The present application provides a data leakage detection method, device, electronic device and storage medium based on semantic preservation disturbance. The present application can amplify the memory traces of the code semantic preservation disturbance amplification model, and construct a threshold-free judgment mechanism by combining the perplexity relative comparison, so as to detect the leakage risk of the code evaluation sample under the condition of complete black box, thereby providing reliable technical support for building a more fair and reliable code large model evaluation system. The technical solutions are as follows: According to an aspect of the present application, a data leakage detection method based on semantic-preserving perturbation is provided, the method comprising: obtaining a code evaluation sample to be detected, and performing parsing processing on the code evaluation sample to obtain a structured sample; calling a preset set of perturbation operators to perform semantic-preserving transformation on the structured sample to generate a sample set, wherein the sample set contains an original sample and a variant sample obtained after the original sample is subjected to semantic equivalence; inputting each sample in the sample set into a detection model respectively, and calculating a generation perplexity of each sample; performing comparative analysis on the generation perplexity of each sample to obtain a comparison result; outputting a leakage detection conclusion according to the comparison result, and recording a leakage confidence indication index.

[0006] According to an aspect of the present application, a data leakage detection device based on semantic-preserving perturbation is provided, the device comprising an acquisition unit, a processing unit, a calculation unit, an analysis unit and an output unit, wherein: the acquisition unit is configured to obtain a code evaluation sample to be detected, and perform parsing processing on the code evaluation sample to obtain a structured sample, wherein the structured sample contains different constituent elements; the processing unit is configured to call a preset set of perturbation operators to perform semantic-preserving transformation on the structured sample to generate a sample set containing an original sample and a variant sample; the calculation unit is configured to input each sample in the sample set into a detection model respectively, and calculate a generation perplexity of each sample; the analysis unit is configured to perform comparative analysis on the generation perplexity of each sample to obtain a comparison result; and the output unit is configured to output a leakage detection conclusion according to the comparison result, and record a leakage confidence indication index.

[0007] According to an aspect of the present application, an electronic device comprises at least one processor and at least one memory, wherein the memory has stored thereon program instructions or code; the program instructions or code are loaded and executed by the processor, so that the electronic device implements the data leakage detection method based on semantic-preserving perturbation as described above.

[0008] According to an aspect of the present application, a storage medium has stored thereon program instructions or code, which are loaded and executed by a processor to implement the data leakage detection method based on semantic-preserving perturbation as described above.

[0009] According to an aspect of the present application, a computer program product comprises program instructions or code stored in a storage medium, a processor of an electronic device reads the program instructions or code from the storage medium, loads and executes the program instructions or code, so that the electronic device implements the data leakage detection method based on semantic-preserving perturbation as described above.

[0010] The technical scheme provided by the present application has the beneficial effects that In the above technical scheme, the present application provides a data leakage detection method, device, electronic equipment and storage medium based on semantic-preserving perturbation, relating to the field of data processing. Among them, the method comprises: first, obtaining a code evaluation sample to be detected, and performing parsing processing on the code evaluation sample to obtain a structured sample; then, calling a preset perturbation operator set to perform semantic-preserving transformation on the structured sample to generate a sample set containing an original sample and a variant sample; and inputting each sample in the sample set into a detection model respectively, and calculating the generation perplexity of each sample; and comparing and analyzing the generation perplexity of each sample to obtain a comparison result; finally, outputting a leakage detection conclusion according to the comparison result, and recording a leakage confidence indication index. In this way, the memory traces of the code semantic-preserving perturbation amplification model are utilized by the present application, and a threshold-free decision mechanism is constructed by combining the relative comparison of perplexity, so that the leakage risk of the code evaluation sample can be detected under the condition of a completely black box, thereby providing reliable technical support for constructing a more fair and reliable code large model evaluation system. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced.

[0012] Figure 1 is a system function module diagram of a data leakage detection method based on semantic-preserving perturbation according to an example embodiment; Figure 2 is a flowchart of a data leakage detection method based on semantic-preserving perturbation according to an example embodiment; Figure 3 is a whole flowchart of a data leakage detection method of a code large model based on semantic-preserving perturbation and model confidence comparison according to an example embodiment; Figure 4 is a structural block diagram of a data leakage detection device based on semantic-preserving perturbation according to an example embodiment; Figure 5 is a structural block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0013] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0014] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0015] The following is an introduction and explanation of the relevant academic background involved in this application: Thanks to the development of the aforementioned models and training techniques, Code LLM has achieved significant performance improvements on standardized benchmarks, especially on mainstream code generation test sets such as HumanEval. The model's Pass@1 accuracy has increased from less than 5% in the early stages to nearly or over 95% currently, approaching a "perfect score." Therefore, such public benchmarks have become a core evaluation method for assessing the code generation capabilities, inference abilities, and practical value of large-scale code generation models.

[0016] Because Code LLM's pre-training heavily relies on large-scale open-source code repositories, public code platforms, and developer community data, and because benchmark sets such as HumanEval may have previously appeared publicly or exist in equivalent forms in internet code resources, there is a potential problem of cross-contamination between the training and test sets. If evaluation samples appear in the model training corpus in their raw or structured equivalent forms, the model may generate correct code during the evaluation phase through "memory or approximate reproduction" rather than "true generalization reasoning," leading to significantly overestimation of the evaluation results and severely compromising the fairness, credibility, and reproducibility of the evaluation.

[0017] To address the aforementioned issues, existing research has attempted to propose various data leakage detection and mitigation solutions, whose technical approaches mainly fall into the following categories: (1) A prompt-based leak detection method.

[0018] This type of method assumes that if the text to be detected has been used by the model for training, the model will typically exhibit significantly lower perplexity or abnormally high word prediction probabilities when generating or evaluating that text. Representative implementations include: (a) Perplexity-based Detection: This method measures the perplexity generated by the model on the input text, using it as an important indicator of whether the model has been trained. Some studies have modified the perplexity calculation results by performing format conversion, case variation, or introducing distracting text fragments to enhance the sensitivity to memorized text.

[0019] (b) Min-K% Probability Detection Method: This type of method assumes that untrained texts usually contain a small number of "low-probability anomalous words", while training texts are more consistent in the overall word probability distribution. Therefore, the mean of the K% words with the lowest probability in the input text is calculated as the leakage confidence score.

[0020] (c) Local maximum likelihood pattern discrimination method (Min-K%++): Theoretically, the training samples are regarded as local extreme points in the model likelihood space, and leakage detection is transformed into a "likelihood extreme point detection" problem by measuring the probabilistic local structure.

[0021] (d) Divergence Calibration Detection Method (DC-PDD): To address the issue that high-frequency words in ordinary text may mask probability differences, a calibration mechanism based on cross-entropy and distribution divergence is introduced. This method enhances detection robustness by comparing the differences between the model word distribution and the empirical distribution.

[0022] The methods described above are primarily designed for natural language text scenarios. Their advantage lies in their ability to perform detection under black-box conditions without access to the model's training corpus or weight parameters. However, these methods heavily rely on probability distribution assumptions and threshold segmentation strategies. The threshold is typically chosen based on timestamps in general text, lacking reliability and generalizability.

[0023] (2) Data leakage detection method based on data overlap comparison For specific application scenarios of code models, some works attempt to quantify potential leakage risks by statistically analyzing the proportion of repeated or similar segments between evaluation samples and known pre-trained code sets. The basic idea of ​​this type of method is to retrieve evaluation samples and their approximate implementations from publicly available code corpora, calculate code duplication indices using techniques such as string matching, hash fingerprinting, and abstract syntax tree matching, and then estimate whether the evaluation samples are likely to be used for model training.

[0024] However, this technical solution has a fundamental flaw: since the complete pre-training corpus of most large code models is not publicly available, overlap detection can only be carried out in an extremely limited subset of data. Objectively, it is impossible to measure the global intersection between the real training corpus and the evaluation benchmark. Therefore, its detection results are highly incomplete and unverifiable, and it is difficult to use them as a basis for determining credible leakage.

[0025] In summary, although existing methods for data leakage detection have achieved some success, the following significant problems still exist: The model does not address the unique leakage behaviors specific to the code domain.

[0026] It is highly dependent on the availability or transparency of pre-trained data, which is difficult to achieve in reality.

[0027] It can only provide partial or weakly proxies of leakage judgment results, which is difficult to support a systematic review of large-scale model evaluation.

[0028] Therefore, under current technological conditions, there is an urgent need for a data leakage detection technology that can be implemented without accessing the internal information of the model, is targeted at code evaluation tasks, is quantifiable, and generalizable, so as to effectively ensure the fairness and credibility of the large code model evaluation process.

[0029] Therefore, the data leakage detection method based on semantic preservation perturbation provided in this application can utilize the memory traces of the model by amplifying the semantic preservation perturbation of the code, and combine it with the relative comparison of perplexity to construct a decision mechanism that does not require a threshold, so as to detect the leakage risk of code evaluation samples under completely black-box conditions.

[0030] Accordingly, the data leakage detection method based on semantic perturbation is applicable to data leakage detection devices based on semantic perturbation. These devices can be deployed on electronic devices, which may be computer devices configured with a von Neumann architecture, such as desktop computers, laptops, servers, etc.; electronic devices may be electronic devices with central control functions, such as gateways; and electronic devices may be portable mobile electronic devices, such as smartphones, tablets, etc.

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] like Figure 1 The system functional module diagram shown illustrates that this solution mainly comprises three core functional modules: an input sample parsing module, a code semantics-preserving perturbation generation module, and a model confidence comparison and leakage judgment module. These modules operate sequentially and collaboratively to form a closed-loop detection system, used for automatic data leakage analysis of a single code evaluation sample.

[0033] Please see Figure 2This application provides a flowchart of a data leakage detection method based on semantically preserved perturbation. This method is applicable to electronic devices, which can be servers or other devices with data analysis and processing capabilities, without specific limitations.

[0034] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0035] like Figure 2 As shown, the method may include the following steps: Step S210: Obtain the code evaluation sample to be detected, and parse the code evaluation sample to obtain a structured sample.

[0036] Specifically, the model acquires code evaluation samples to be detected, and then uses methods such as... Figure 1 The input sample parsing module shown performs structured decomposition (i.e., parsing) on ​​the code evaluation samples to obtain structured samples.

[0037] The code evaluation sample typically consists of the following four sub-elements: problem description text. Test case specifications Function interface definition Reference implementation code Therefore, a single sample example can be represented as: Step S220: Call the preset set of perturbation operators to perform a semantic-preserving transformation on the structured samples to generate a sample set.

[0038] The sample set includes the original samples and variant samples obtained by semantically equivalencing the original samples.

[0039] Specifically, the model uses, for example Figure 1 The code semantic preservation perturbation generation module shown is responsible for performing semantic preservation perturbation transformations on the input structured samples to construct a sample set.

[0040] For example, the code semantics-preserving perturbation generation module has a built-in set of perturbation operators, defined as: in, This represents the text restatement perturbation operator for the problem description; This represents the test case variables and format perturbation operators; Indicates the renaming operator for consistency of function interface identifiers; This represents a code variable refactoring operator. It is derived from input instances. Applying the above perturbation operator, semantically equivalent perturbation instances are generated: And it satisfies the following constraints: ,in The above constraints indicate that the functional semantics of the samples are equivalent. The criterion for this is that the reference implementation code or its perturbation variants can both run correctly and return consistent results under the corresponding test case set.

[0041] In this way, for the original input sample It can obtain a sample containing the original sample and The sample set of equivalent transformations: Step S230: Input each sample in the sample set into the detection model and calculate the generation perplexity of each sample.

[0042] Specifically, the model confidence comparison and leakage judgment module performs the following checks on each sample set: Calculate the perplexity (PPL) of the large language model. The perplexity of each sample is defined as follows: in: Represents the large language model to be detected; This represents the word of the i-th sample at time step m; Indicates the sample length.

[0043] Step S240: Compare and analyze the generated perplexity of each sample to obtain the comparison results.

[0044] Subsequently, this module performs the following comparison and judgment: When satisfied In other words, if the original sample consistently obtains the lowest perplexity value, it is determined that the sample has a risk of being leaked into the model's memory.

[0045] Step S250: Output the leakage detection conclusion based on the comparison results and record the leakage confidence indicator.

[0046] Finally, based on the comparative analysis results of step S240, it is determined whether there is a risk of data leakage. After outputting the detection conclusion, the leakage confidence indicator is recorded.

[0047] Under the above embodiments, the method proposed in this application first obtains code evaluation samples to be detected and parses them to obtain structured samples; then, it calls a preset set of perturbation operators to perform semantic-preserving transformation on the structured samples, generating a sample set containing original samples and variant samples; then, it inputs each sample in the sample set into the detection model to calculate the generation perplexity of each sample; then, it compares and analyzes the generation perplexity of each sample to obtain comparison results; finally, it outputs a leakage detection conclusion based on the comparison results and records the leakage confidence indicator. Thus, by utilizing the semantic-preserving perturbation to amplify the memory traces of the model and combining it with a relative comparison of perplexity to construct a threshold-free decision mechanism, the leakage risk of code evaluation samples can be detected under completely black-box conditions, thereby providing reliable technical support for building a fairer and more trustworthy large-scale code model evaluation system.

[0048] In one exemplary embodiment, the structured sample includes different components, including problem description text, test case specifications, function interface definitions, and reference implementation code.

[0049] For example, after obtaining the code evaluation samples to be detected (such as a single task instance in HumanEval or MBPP), the model performs a structured decomposition of the samples to extract four core elements: Problem description text (d): Code requirements in natural language form (e.g., "Write a function to calculate the average of an array"); Test case specification (t): includes input data, output validation rules, execution environment constraints, etc. (e.g., output 2.0 when inputting [1,2,3]). Function interface definition (f): function name, parameter list, return value declaration, etc. (e.g., def calculate_avg(arr:List[float]) -> float); Reference implementation code (c): The specific code logic to fulfill the requirements (such as looping through the array, summing the results, and then taking the average).

[0050] In one exemplary embodiment, the set of perturbation operators includes a problem description text restatement perturbation operator, a test case variable and format perturbation operator, a function interface identifier consistency renaming operator, and a code variable refactoring operator.

[0051] In an exemplary embodiment, the method of invoking a preset set of perturbation operators to perform semantically preserving transformations on the structured samples includes the following steps: invoking the problem description text restatement perturbation operator to perform equivalent rewriting of the problem description text through synonym replacement, language recombination, and multi-round rewriting; invoking the test case variable and format perturbation operator to perform equivalent transformations on the naming method, input format, or numerical value of input variables in the test case specification, wherein the equivalent transformations maintain the core verification logic of the test cases unchanged; invoking the function interface identifier consistency renaming operator to perform end-to-end consistency renaming of function names and parameter names in the function interface definition, and synchronously adjust the corresponding function call and parameter reference logic in the code; and invoking the code variable reconstruction operator to perform equivalent replacements on non-core variable names in the reference implementation code, wherein the equivalent replacements support equivalent structural rearrangement of independent statements without execution dependencies.

[0052] Specifically, the above steps are performed by the code semantics-preserving perturbation generation module, whose goal is to generate multiple sets of equivalent variant samples and construct a detection control set without changing the functional semantics of the code. The module has a built-in set of four perturbation operators, defined as follows: For example, semantic-preserving transformations are performed on different elements of the structured sample respectively: : Restate the problem description text by performing synonym substitution, language restructuring, or other equivalent methods; Make consistent modifications to the variable naming and input format of test cases; Perform end-to-end consistent renaming of identifiers for function interfaces; : Performs equivalent replacements for variable names in the reference implementation code.

[0053] For example, The disturbance process includes: calling The operator generates 2-3 versions of equivalent descriptions through "synonym substitution + sentence recombination".

[0054] The disturbance process includes: calling Operators make equivalent modifications to the non-core parts of test cases. For example, changing the variable name test_input1 to case_val_a in a Python test case, or changing the input format from "list input" to "tuple input" (the code reading logic needs to be adapted accordingly), while retaining core constraints such as the input value range and output validation logic to avoid compromising the effectiveness of the test.

[0055] Disturbance: Call The operator performs end-to-end consistent renaming based on the AST identifier mapping table. For example, it renames the function calculate_sum(a,b) to compute_total(x,y), automatically modifies the identifiers at all function call points and parameter reference points, and verifies the interface syntax validity using a syntax checking tool (such as Python's pyflakes). If a syntax error is found, it automatically rolls back.

[0056] The disturbance process includes: calling The operator performs equivalent substitutions only on non-core variables and supports two types of transformations, as detailed below: Variable name renaming: Change temp in temp = a + b to cache_val, and ensure that all references to the variable are updated synchronously by traversing the AST; Equivalent structure replacement: rewrite the for i in range(10) loop as i = 0; while i<10: i+=1, but you need to first confirm that there is no statement execution order dependency through dependency analysis, and then ensure the structure is legal through syntax compilation verification.

[0057] For example, in the step of generating the sample set, the core is to build a full-process guarantee system around functional semantic equivalence. This system controls the process layer by layer from three dimensions: rule constraints before the disturbance, process control during the disturbance, and double verification after the disturbance. This ensures that the disturbance only changes the sample form and does not change the core functional semantics, as detailed below: Before perturbation: Define immutable boundaries and anchor core semantics. Before performing any perturbation operations, first clarify the core semantic anchor points of each element of the sample through structured analysis, determine the key parts of the sample's function, and only allow transformations of non-core, non-functional formal content to avoid semantic deviations from the source.

[0058] Specifically, regarding the boundary delineation of the problem description text, core requirement keywords are extracted: such as "sorting arrays," "calculating the sum of two numbers," and "verifying mobile phone number format," which directly define the functional goals and are marked as "non-replaceable / modifiable." At the same time, the scope of modification is clarified: only sentence structure (active / passive, causal / conditional), non-core modifiers (such as "quick writing" can be changed to "implement"), and expression style (colloquial / formal) are allowed to be adjusted, while the core intent of the requirement is prohibited from being modified.

[0059] For the boundary determination of test case specifications, by analyzing the core verification logic of test cases: including the input value range (such as "input is an integer between 1 and 100"), the output determination rule (such as "the return value needs to be greater than 0"), and the exception scenario verification (such as "return None when the input is empty"), it is marked as "unmodifiable"; at the same time, clarify the modifiable range: only allow adjusting variable names (such as modifying `test_val` to `case_data`), input data formats (lists can be modified to tuples, JSON can be modified to key-value pairs), and comment texts, and prohibit modifying the verification logic itself.

[0060] For the boundary determination of function interface definitions: Based on the analysis of the abstract syntax tree (AST), analyze the core attributes of the interface: including the function input parameter types (such as `int` / `List`), return value types, and core business identifiers (such as `transfer_amount` in financial codes), and mark them as "unmodifiable"; at the same time, clarify the modifiable range: only allow adjusting general identifiers without business meaning (such as modifying `a` / `b` to `x` / `y`, `func1` to `func_temp`), and prohibit modifying the number, type, and return logic of the interface parameters.

[0061] For the boundary determination of reference implementation code ($c$): Distinguish core / non-core logic through AST + dependency analysis. Among them, the core logic: the core steps of the algorithm (such as comparison and exchange in sorting, hash calculation in encryption), conditional judgment thresholds (such as `if score>60`), and loop termination conditions (such as `i<len(arr)`), are marked as "unmodifiable"; the non-core logic: temporary variable naming (such as modifying `temp` to `cache`), the order of statements without dependencies (such as the order of log printing and variable assignment within the same loop), and comment texts are defined as the "modifiable range".

[0062] Furthermore, in the perturbation: structured operations + linked modifications to avoid logical breaks. When executing specific perturbation operators, adopt the "AST-driven structured transformation + full-linkage modification" strategy to ensure that each transformation step does not damage the syntactic legality and logical coherence of the code. The control methods for the four types of operators are as follows: Process control of the problem description text restatement operator: Adopt "template-driven + semantic anchoring" rewriting: Execute the transformation based on the preset code requirement rewriting template (such as "active voice → passive voice", "simple sentence → compound sentence"), and forcefully retain all core requirement keywords during the rewriting process; real-time semantic verification: For each generated rewritten text, immediately calculate the similarity with the original text through a pre-trained semantic similarity model (such as BERT). If the similarity is lower than the preset threshold (such as 90%), automatically terminate this round of rewriting and regenerate.

[0063] Process control of test case variables and format perturbation operators, and linkage adaptation of format changes: If the input format of a test case is modified (e.g., "list" to "tuple"), the corresponding input reading logic in the code must be adjusted synchronously (e.g., changing `arr = list(input)` to `arr = tuple(input)`) to ensure that the code is compatible with the new format; Global consistency of variable renaming: When modifying the variable name in a test case, all references to the variable in the code are replaced synchronously through regular expression matching + AST traversal to avoid execution failure caused by "the test case variable name has been changed but not in the code".

[0064] The process control of renaming function interface identifiers is achieved through full-link traversal replacement using the Abstract Syntax Tree (AST): First, the AST is used to locate all occurrences of function names and parameter names (including definitions, calls, and comment references), and a mapping table is established where the "original identifier" can be modified into the "new identifier". Batch replacement + real-time syntax checking: After batch replacing all related nodes based on the mapping table, programming language syntax checking tools (such as Python's `pyflakes` and Java's `javac`) are immediately called to verify the syntax. If errors such as "undefined identifier" occur, the replacement operation is automatically rolled back.

[0065] Process control of code variable refactoring operators, such as: global replacement of non-core variables: only renaming variables marked as "non-core" is performed, and the definition, reference and assignment of variables are updated synchronously through AST traversal; setting dependency checks for statement structure transformations: for structure transformations such as "for loop and while loop", the statement is first confirmed to have no execution dependencies (such as the subsequent statement does not depend on the variable state of the previous statement) through dependency analysis tools (such as CodeQL) before the transformation is performed to avoid breaking the execution logic.

[0066] Under the aforementioned embodiments, this application proposes a code sample construction mechanism based on semantically preserved perturbation. For the code samples to be detected, various semantically equivalent transformation operators are designed. All transformations ensure that the functional semantics and execution results of the code remain unchanged, thereby generating a set of multiple versions of functionally equivalent code samples. By generating multiple versions of equivalent samples through systematic code transformation operations, the change in the model's confidence in perturbed samples becomes the core criterion for determining whether a sample has been memorized. This provides a necessary comparative basis for subsequent model memory detection and is a core technical means to distinguish between training memory samples and non-training samples.

[0067] In an exemplary embodiment, after the method calls a preset set of perturbation operators to perform a semantically preserving transformation on the structured sample to generate a sample set, the method includes the following steps: performing a verification step on the variant sample, the verification step including: calling a syntax checking tool of the corresponding programming language to perform a syntax scan on the function interface definition and the reference implementation code of the variant sample, and filtering out variant samples with syntax errors; reusing the test case specifications of the original sample to run the reference implementation code of the variant sample, and comparing the execution results of the variant sample with those of the original sample; if the execution results are inconsistent, the variant is determined to be invalid and discarded.

[0068] For example, the solution in this application can call the compiler / interpreter of the corresponding programming language (such as `python -m py_compile` for Python and `javac` for Java) by calling the syntax checking tool of the corresponding programming language (such as pyflakes for Python) to perform compile-level syntax checking on the perturbed function interface and reference implementation code, so as to filter out syntax error samples, and set an error handling mechanism: if a syntax error is detected (such as missing semicolon, undefined identifier, indentation error), the variant is directly marked as invalid, discarded and a new variant sample is regenerated.

[0069] Furthermore, by reusing the test cases t of the original sample, the complete set of test cases from the original sample can be applied to the perturbed code. All test cases are run, and the output results are recorded. The execution results are then compared with the original code to see if they are consistent. If the output results of all test cases are completely consistent with the output of the original code, it is considered "semantic preservation," and the variant is retained. If any test case result is inconsistent (e.g., deviation in calculation results, different handling of abnormal scenarios), it is considered "semantic deviation," and the variant is discarded.

[0070] For example, rewrite "write a function to calculate the sum of two numbers" as "implement a function that can return the sum of two numbers". At the same time, use a semantic similarity model (such as BERT) to check the semantic consistency of the text before and after the rewrite. If the similarity is lower than the threshold, the variant is discarded.

[0071] In one possible example, supplementary boundary value checks can also be set up: for special scenarios such as empty input, extreme value input, and abnormal input, additional special tests are performed to ensure that the core semantics remain consistent under extreme conditions.

[0072] In another possible example, full-process logging can be set up: record the operator type, modification location, original content, transformed content, and verification results for each round of perturbation. If semantic deviations are found later, the problematic operator can be quickly located and its rules optimized.

[0073] Further optimization of abnormal sample clustering: "pseudo-equivalent samples" that failed historical verification (such as samples with valid surface syntax but functional deviations) are included in a blacklist, their perturbation features are extracted and the operator rules are updated to avoid the recurrence of the same type of error.

[0074] Under the above embodiments, by setting a variant sample verification process to ensure the accuracy of the control data used for detection, it can be ensured that each generated perturbation variant sample meets the semantic preservation requirement of "different forms, completely identical functions", providing a reliable control sample for subsequent confidence comparison, thereby ensuring the reliability of the final data leakage detection results.

[0075] In an exemplary embodiment, the method for comparing and analyzing the generated perplexity of each sample further includes the following steps: comparing and analyzing the distribution of the original sample and its corresponding variant sample in the perplexity space; if the detection model detects that the perplexity corresponding to the original sample is less than the perplexity of the variant sample, then determining that the original sample has a leakage risk; if the detection model detects that the perplexity corresponding to the original sample is not less than the perplexity of the variant sample, then determining that the original sample does not have a significant leakage risk; outputting the leakage detection conclusion and recording the leakage confidence indicator, wherein the leakage detection conclusion is used to indicate whether the original sample has the leakage risk.

[0076] Specifically, the proposed solution uses pre-set judgment rules: if the perplexity of the original sample is the minimum value in the sample set, then the original sample is judged to have a risk of data leakage due to being memorized by the model; otherwise, if the perplexity of the original sample is not significantly lower than that of the variant sample, then there is no obvious leakage.

[0077] Furthermore, based on the above judgment results, the detection conclusion (leakage / no leakage) is output, and key indicators are recorded, including the perplexity values ​​of the original sample and each variant sample, the average perplexity difference, the stability verification results, etc., to form a complete detection report.

[0078] Furthermore, if a leakage risk is determined, the sample can be removed from the evaluation benchmark or its evaluation score can be adjusted by weighting to ensure the fairness of the model performance evaluation.

[0079] Therefore, in this solution, based on the method described in the above embodiments, the entire detection process does not require access to the model training corpus, internal weights, or temporal information. It is entirely based on black-box interaction and specifically solves the problem of detecting code domain structural equivalence leakage through semantically preserved perturbations. After obtaining the detection results based on perplexity, the model can be further optimized based on the detection results to improve the model's accuracy.

[0080] To better understand the differences between the above content and existing technologies, the following will combine... Figure 3 This paper presents a flowchart illustrating the overall process of a data leakage detection method for large code models based on semantically preserved perturbation and model confidence comparison. Specifically, for example... Figure 3 As shown: For example, Figure 3 The application effects of the three core perturbation operators are intuitively demonstrated through specific code examples. It is clear that the perturbation only changes the formal representation of the sample (function name, parameter name, variable name, test case input data) without changing the core functional logic and execution results. This verifies the implementation effect of semantically preserving perturbation and provides a reliable reference sample for subsequent confidence comparison.

[0081] Figure 3 The layout is a left-right comparison. The left side shows the common existing detection method workflow, using the original sample (target sample x) as input data. As you can see, most traditional methods rely on scoring functions and threshold settings for sample leakage detection. The right side shows three groups of different types of perturbation variant samples (x1, x2, x11). Figure 3 The right side clearly demonstrates the core workflow of the detection system provided by this solution. Through the sequential collaboration of five stages—"parse-perturbation-measurement-comparison-judgment"—automatic detection of data leakage in large code models is achieved. The process does not rely on model training corpora or internal parameters and is completed entirely based on black-box interaction, intuitively presenting the core design concept of "no threshold, semantic preservation, and confidence comparison".

[0082] The application effects of the three core perturbation operators are intuitively demonstrated through specific code examples. It is clarified that perturbations only change the formal representation of the samples (function names, parameter names, variable names, and test case input data), without altering the core functional logic and execution results. This verifies the effectiveness of semantically preserving perturbations and provides a reliable reference sample for subsequent confidence comparisons. The core judgment logic for confidence comparison is presented visually, intuitively demonstrating the core rule of "lowest PPL in the original sample → leakage judgment". The bar chart clearly shows the confidence difference between the remembered and unremembered samples, and the flowchart clarifies the execution steps for threshold-free judgments, verifying the feasibility and intuitiveness of the "confidence difference comparison" mechanism of this invention.

[0083] It should be noted that the above Figure 3 The entire processing flow on the right can be mapped to steps S210-S250 and their corresponding sub-explanations, which will not be repeated here.

[0084] It should be noted that the data leakage detection method based on semantically preserved perturbation and model confidence comparison provided in this application for code-wide models has been verified for its feasibility and effectiveness through extensive experiments. The experiments cover a variety of mainstream code-wide language models (CodeLLM), including Qwen2.5-7B, and have been systematically evaluated.

[0085] Experimental results show that the proposed method significantly outperforms existing baseline methods in key metrics such as accuracy, precision, recall, and F1-score, fully demonstrating the feasibility and reliability of the proposed solution in practical applications. Further experiments demonstrate that the proposed method exhibits good robustness and generalization ability.

[0086] Specifically, please refer to Table 1 below. Table 1 presents the performance comparison results of the proposed method with other baseline methods. Detailed data can be found in Table 1. Table 1 Furthermore, experiments validated the advantages of the semantically preserved perturbation and confidence comparison strategy. Compared to methods relying solely on the original sample probability or the Min-K% method (K is set to 20 in Table 1), this approach generates semantically equivalent perturbed samples and analyzes the confidence difference between the original and perturbed samples, effectively amplifying the distinguishing features between remembered and non-remembered samples and reducing the false negative rate common in traditional methods. This result demonstrates that a dedicated perturbation strategy for code data has significant advantages in detecting model memory behavior.

[0087] In summary, the technical solution provided by this invention has been proven through experiments to have the characteristics of high precision, strong robustness, and good scalability, which fully demonstrates the feasibility and practical value of the method, device, and system of this invention in detecting data leakage in large code models.

[0088] The following are embodiments of the apparatus described in this application, which can be used to execute the data leakage detection method based on semantic preservation perturbation involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the data leakage detection method based on semantic preservation perturbation involved in this application.

[0089] Please see Figure 4 This application provides a data leakage detection device 400 based on semantically preserved perturbation. The device includes an acquisition unit 401, a processing unit 402, a calculation unit 403, an analysis unit 404, and an output unit 405, wherein: The acquisition unit 401 is used to acquire the code evaluation sample to be detected, and to parse the code evaluation sample to obtain a structured sample, wherein the structured sample contains different components. The processing unit 402 is used to call a preset set of perturbation operators to perform a semantic-preserving transformation on the structured sample, generating a sample set containing the original sample and n sets of semantically equivalent variant samples; The computing unit 403 is used to input each sample in the sample set into the detection model and calculate the generation perplexity of each sample. The analysis unit 404 is used to perform comparative analysis on the generation perplexity of each sample to obtain comparison results; The output unit 405 is used to output a leakage detection conclusion based on the comparison results and record the leakage confidence indicator.

[0090] Therefore, in this scheme, based on the aforementioned data leakage detection device based on semantic preservation perturbation, firstly, the acquisition unit acquires the code evaluation samples to be detected and parses them to obtain structured samples; then, the processing unit calls a preset set of perturbation operators to perform semantic preservation transformation on the structured samples, generating a sample set containing original samples and variant samples; then, each sample in the sample set is input into the detection model, and the calculation unit calculates the generation perplexity of each sample; then, the analysis unit compares and analyzes the generation perplexity of each sample to obtain comparison results; finally, the output unit outputs the leakage detection conclusion based on the comparison results and records the leakage confidence indicator. Thus, by utilizing the semantic preservation perturbation of the code to amplify the model's memory traces and combining it with a relative comparison of perplexity to construct a threshold-free decision mechanism, the leakage risk of code evaluation samples can be detected under completely black-box conditions, thereby providing reliable technical support for building a fairer and more trustworthy large-scale code model evaluation system.

[0091] In one exemplary embodiment, the structured sample comprises different components, including problem description text, test case specifications, function interface definitions, and reference implementation code.

[0092] In one exemplary embodiment, the set of perturbation operators includes a problem description text restatement perturbation operator, a test case variable and format perturbation operator, a function interface identifier consistency renaming operator, and a code variable refactoring operator.

[0093] In an exemplary embodiment, the processing unit 403 is further configured to: invoke a preset set of perturbation operators to perform a semantically preserving transformation on the structured sample; The problem description text restatement perturbation operator is invoked to perform equivalent rewriting of the problem description text through synonym substitution, language recombination, and multi-round rewriting. The test case variables and format perturbation operators are invoked to perform equivalent transformations on the naming conventions, input formats, or values ​​of the input variables in the test case specifications, wherein the equivalent transformations maintain the core verification logic of the test cases unchanged; The function interface identifier consistency renaming operator is invoked to perform end-to-end consistency renaming of the function name and parameter name in the function interface definition, and the corresponding function call and parameter reference logic in the code is adjusted synchronously. The code variable refactoring operator is invoked to perform equivalent replacements on non-core variable names in the reference implementation code. The equivalent replacement supports equivalent structural rearrangement of independent statements without execution dependencies.

[0094] In an exemplary embodiment, after the processing unit 403 and the analysis unit 404 generate a sample set by invoking a preset set of perturbation operators to perform a semantically preserving transformation on the structured samples, the processing unit 403 is further configured to: Perform a verification step on the variant sample, the verification step including: The syntax checking tool of the corresponding programming language is invoked to perform a syntax scan on the function interface definition and the reference implementation code of the variant sample, and the variant sample containing syntax errors is filtered out; Run the reference implementation code of the variant sample using the test case specifications of the original sample, and compare the execution results of the variant sample with those of the original sample; If the execution results are inconsistent, the variant is deemed invalid and discarded.

[0095] In an exemplary embodiment, the analysis unit 404 and the output unit 405 are further configured to: perform comparative analysis on the generated perplexity of each sample. The distribution of the original samples and their corresponding variant samples in the perplexity space is compared and analyzed. If the detection model detects that the perplexity of the original sample is less than that of the variant sample, it determines that the original sample has a leakage risk. If the detection model detects that the perplexity of the original sample is not less than the perplexity of the variant sample, then it is determined that the original sample does not have a significant risk of leakage. Output the leakage detection conclusion and record the leakage confidence indicator, wherein the leakage detection conclusion is used to indicate whether the original sample has the leakage risk.

[0096] It should be noted that the data leakage detection device based on semantic perturbation provided in the above embodiments is only illustrated by the division of the above functional modules when performing gene detection and analysis. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the data leakage detection device based on semantic perturbation will be divided into different functional modules to complete all or part of the functions described above.

[0097] Furthermore, the data leakage detection device based on semantic perturbation and the data leakage detection method based on semantic perturbation provided in the above embodiments belong to the same concept. The specific way each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0098] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0099] This application also provides an electronic device. Figure 5 This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 5000 includes at least one processor 5001 and at least one memory 5003.

[0100] The data interaction between the processor 5001 and the memory 5003 can be achieved through at least one communication bus 5002. This communication bus 5002 may include a path for transmitting data between the processor 5001 and the memory 5003. The communication bus 5002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 5002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0101] Optionally, the electronic device 5000 may further include a transceiver 5004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 5004 is not limited to one unit, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of this application.

[0102] Processor 5001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0103] The memory 5003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 500, but not limited thereto.

[0104] The memory 5003 stores program instructions or code, and the processor 5001 can read the program instructions or code stored in the memory 5003 through the communication bus 5002.

[0105] When the program instructions or code are executed by the processor 5001, the data leakage detection method based on semantically preserving perturbation in the above embodiments is implemented.

[0106] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by the processor, the processor will execute the data leakage detection method based on semantic preservation perturbation provided in this application.

[0107] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0108] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0109] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0111] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0115] Compared with related technologies, the data leakage detection method based on semantically preserved perturbation proposed in this application generates multiple equivalent samples through systematic code transformation operations, making the change in the model's confidence in the perturbation samples the core basis for judging whether a sample has been memorized. Compared with traditional methods based on overlap rate or time inference, this invention can effectively reduce the false positive rate and false negative rate, and is especially suitable for code samples with different syntaxes and similar inheritance structures, achieving high-precision identification of complex code patterns. Through semantically preserved perturbation generation, model confidence comparison, differential analysis judgment, and automated system implementation, it has significant advantages over existing technologies in terms of not relying on training data, not requiring threshold setting, wide applicability, and high detection accuracy, effectively solving the core technical problem in data leakage detection of large code models.

[0116] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0117] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting data leakage based on semantic-preserving perturbation, characterized in that, The method comprises the following steps: acquiring a code evaluation sample to be detected, and performing parsing processing on the code evaluation sample to obtain a structured sample; calling a preset perturbation operator set to perform semantic-preserving transformation on the structured sample to generate a sample set, wherein the sample set includes an original sample and a variant sample obtained by performing semantic equivalence on the original sample; inputting each sample in the sample set into a detection model respectively, and calculating the generation perplexity of each sample; comparing and analyzing the generation perplexity of each sample to obtain a comparison result; outputting a leakage detection conclusion according to the comparison result, and recording a leakage confidence indication index.

2. The method of claim 1, wherein, Wherein, the structured sample includes different constituent elements, and the constituent elements include problem description text, test case specification, function interface definition, and reference implementation code.

3. The method of claim 2, wherein, The perturbation operator set includes problem description text restatement perturbation operator, test case variable and format perturbation operator, function interface identifier consistency renaming operator, and code variable restructuring operator.

4. The method of claim 3, wherein, The method of calling the preset perturbation operator set to perform semantic-preserving transformation on the structured sample comprises: calling the problem description text restatement perturbation operator to perform equivalent rewriting on the problem description text in the form of synonymous replacement, language restructuring, and multi-round rewriting; calling the test case variable and format perturbation operator to perform equivalent transformation on the input variable naming method, input format, or numerical value in the test case specification, wherein the equivalent transformation keeps the test case core check logic unchanged; calling the function interface identifier consistency renaming operator to perform full-link consistency renaming on the function name and parameter name in the function interface definition, and synchronously adjusting the corresponding function call and parameter reference logic in the code; calling the code variable restructuring operator to perform equivalent replacement on the non-core variable name in the reference implementation code, wherein the equivalent replacement supports equivalent structure rearrangement on independent statements without execution dependency.

5. The method of claim 4, wherein, After the method of calling the preset perturbation operator set to perform semantic-preserving transformation on the structured sample to generate a sample set, the method further comprises: performing a check step on the variant sample, wherein the check step comprises: calling a syntax checking tool of a corresponding programming language to perform syntax scanning on the function interface definition and the reference implementation code of the variant sample, and screening out the variant sample with syntax errors; multiplexing the test case specification of the original sample to run the reference implementation code of the variant sample, and comparing the execution results of the variant sample and the original sample; if the execution results are inconsistent, it is determined that the variant sample is invalid and is discarded.

6. The method of claim 1, wherein, The method of comparing and analyzing the generation perplexity of each sample comprises: comparing and analyzing the distribution of the original sample and its corresponding variant sample in the perplexity space in the sample set; if the detection model detects that the perplexity of the original sample is less than the perplexity of the variant sample, it is determined that the original sample has a leakage risk; If the detection model detects that the perplexity of the original sample is not less than the perplexity of the variant sample, it is determined that the original sample does not have obvious leakage risk; Output the leakage detection conclusion and record the leakage confidence indication index, wherein the leakage detection conclusion is used to indicate whether the original sample has the leakage risk.

7. A data leakage detection device based on semantically preserved perturbation, characterized in that, The device comprises an acquisition unit, a processing unit, a calculation unit, an analysis unit and an output unit, wherein: The acquisition unit is configured to acquire a code evaluation sample to be detected, and perform parsing processing on the code evaluation sample to obtain a structured sample, wherein the structured sample comprises different constituent elements; The processing unit is configured to call a preset set of perturbation operators to perform semantic-preserving transformation on the structured sample, to generate a sample set comprising an original sample and n groups of semantic-equivalent variant samples; The calculation unit is configured to input each sample in the sample set into a detection model respectively, and calculate the generated perplexity of each sample; The analysis unit is configured to perform comparative analysis on the generated perplexity of each sample to obtain a comparison result; The output unit is configured to output a leakage detection conclusion according to the comparison result, and record a leakage confidence indication index.

8. An electronic device, comprising: Comprise: At least one processor and at least one memory, wherein, Program instructions or codes are stored on the memory; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the data leakage detection method based on semantic-preserving perturbation as claimed in any one of claims 1 to 6.

9. A storage medium having stored thereon program instructions or code, characterized in that, The program instructions or codes are loaded and executed by the processor to implement the data leakage detection method based on semantic-preserving perturbation as claimed in any one of claims 1 to 6.

10. A computer program product comprising a computer program, the computer program being read and executed by a processor of a computer device, so that the computer device executes the data leakage detection method based on semantic-preserving perturbation as claimed in any one of claims 1 to 6.

Images