Methods, devices, electronic devices, readable storage media, and computer program products for protecting large language model datasets based on polluting lexical units.

Abstract

This application discloses a method, apparatus, electronic device, readable storage medium, and computer program product for protecting a large language model dataset based on polluted lexical units, belonging to the field of artificial intelligence technology. The method includes: generating a polluted data statistical template by statistically analyzing each data item in the original dataset to obtain the high-frequency common words and grammatical structure characteristics of the original dataset; generating a polluted lexical sequence including high-frequency common words and polluted lexical units, wherein the polluted lexical units include: user-created words that do not exist in reality, generated by arranging common characters, and their corresponding parts of speech; generating polluted sentences using the polluted lexical sequence based on the grammatical structure characteristics of the original dataset; filling the corresponding data item of each data item in the original dataset with the polluted sentences according to the existence probability recorded in the polluted data statistical template, generating a polluted dataset; polluting the original dataset using the polluted sentences and the polluted dataset, and storing the pollution locations to generate a protected dataset.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, readable storage medium, and computer program product for protecting large language model datasets based on polluted word units. Background Technology

[0002] Currently, Large Language Models (LLMs) are one of the hottest research areas in the field of artificial intelligence in recent years. High-quality datasets are often the key to determining the quality of large language models. Therefore, protecting datasets will become a key task for intelligent AI companies in the future.

[0003] Current known dataset security protection schemes use common encryption methods to encrypt the dataset and then decrypt it before using it for training. However, the decryption process is usually quite complex, requiring the decryption module to be loaded into the training program, which generates additional runtime load, thereby reducing the training efficiency of the dataset and increasing hardware and time costs. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, readable storage medium, and computer program product for protecting large language model datasets based on polluted lexical units. It can solve the problem that existing dataset protection schemes require loading the decryption module into the training program, which generates additional runtime load and reduces the training efficiency of the dataset.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] Firstly, a method for protecting large language model datasets based on polluting lexical units is provided, including:

[0007] Obtain the original dataset;

[0008] The data items of each data item in the original dataset are statistically analyzed to generate a polluted data statistical template, and the high-frequency common words and grammatical structure characteristics of the original dataset are obtained; wherein, the polluted data statistical template records the existence probability of each data item;

[0009] Generate a pollution lexical sequence, the pollution lexical sequence including: the high-frequency common words and pollution lexical elements, wherein the pollution lexical elements include: user-created words that do not exist in reality and their corresponding parts of speech generated by arranging common characters;

[0010] Polluted sentences are generated using the polluted lexical sequence based on the syntactic structure characteristics of the original dataset;

[0011] According to the existence probability recorded in the pollution data statistics template, the pollution statement is filled into the corresponding data item of each data in the original dataset to obtain the pollution data of the corresponding data item, and a pollution dataset is generated.

[0012] The original dataset is contaminated using the contaminated statements and the contaminated dataset, and the contaminated locations are stored to generate a protected dataset.

[0013] Secondly, a protection device for large language model datasets based on polluting lexical units is provided, including:

[0014] A data preprocessing module is used to obtain the original dataset; to perform statistics on each data item in the original dataset, generate a contaminated data statistical template, and obtain the high-frequency common words and grammatical structure characteristics of the original dataset; wherein, the contaminated data statistical template records the existence probability of each data item;

[0015] A contaminated data generation module is used to generate a contaminated lexical sequence, which includes: the high-frequency common words and contaminated lexical units, wherein the contaminated lexical units include: user-created words that do not exist in reality and their corresponding parts of speech generated by arranging common characters; using the contaminated lexical sequence to generate contaminated sentences according to the grammatical structure characteristics of the original dataset; and filling the contaminated sentences into the corresponding data items of each data item in the original dataset according to the existence probability recorded in the contaminated data statistics template to obtain the contaminated data of the corresponding data items, thereby generating a contaminated dataset.

[0016] The protected dataset generation module is used to contaminate the original dataset using the contaminated statements and the contaminated dataset, and store the contaminated locations to obtain the protected dataset.

[0017] Thirdly, an electronic device is provided, comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described method for protecting large language model datasets based on polluting lexical units.

[0018] Fourthly, a readable storage medium is provided, wherein at least one computer program is stored in the readable storage medium, the computer program being loaded and executed by a processor to implement the above-described method for protecting large language model datasets based on polluting lexical units.

[0019] Fifthly, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the method for protecting large language model datasets based on polluting lexical units provided in the various optional implementations described above.

[0020] The method, apparatus, electronic device, readable storage medium, and computer program product for protecting large language model datasets based on contaminated lexical units provided in this application embodiment enable LLM to avoid loading the decryption module into the training program. By simply filtering out the contaminated data, the training efficiency of the protected dataset can be greatly improved, and hardware and time costs can be reduced. Furthermore, if an attacker steals the contaminated dataset for training a large LLM model, not only will the original training effect be lost, but the various indicators of the large model will also be significantly reduced. In some cases, the unfiltered contaminated content may even cause the model performance to crash, thereby protecting the security of the original dataset.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 This invention illustrates a flowchart of a method for protecting a large language model dataset based on polluting lexical units, provided in an exemplary embodiment of this application.

[0024] Figure 2 This application shows a block diagram illustrating an exemplary embodiment of a large language model dataset protection device based on polluting lexical units.

[0025] Figure 3 This invention provides a block diagram of an exemplary embodiment of a large language model dataset protection device based on polluting lexical units.

[0026] Figure 4 A structural block diagram of an electronic device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] To address the issue that existing dataset protection schemes require loading the decryption module into the training program, thereby generating additional runtime load and reducing dataset training efficiency, this application provides a method for protecting large language model datasets based on polluted lexical units.

[0029] First, the terms used in the embodiments of this application will be explained.

[0030] Lexical units: In LLM (Local Mode) sentence reading, lexical units are unique units within a sentence that possess consistent and significant meaning. Besides words from the dictionary, lexical units also include suffixes, common expressions, and names. For example, the English word "running" can be segmented into two lexical units: "run" and "-ing"; some common expressions like "in the end" are treated as a single lexical unit; and names like "New York" are also lexical units. The more lexical units LLM encodes, the faster the sentence "reading" becomes, while consuming less computational power, thus reducing response costs.

[0031] Part of speech: such as noun, verb, adjective, etc.

[0032] Pollution lexical units are self-created lexical units that do not exist in reality, lack true semantic meaning, and are randomly assigned a certain part of speech. LLM cannot understand their meaning, thus inducing more "illusions" and even making the originally established moral safeguards extremely vulnerable.

[0033] Figure 1 A flowchart illustrating an exemplary embodiment of this application is shown, illustrating a method for protecting a large language model dataset based on polluted lexical units. This method can be executed by a large language model dataset protection device based on polluted lexical units. This device can be implemented in hardware and / or software and can be configured in an electronic device, such as integrated into a server device. Figure 1 As shown, this method for protecting large language model datasets based on polluting lexical units mainly includes the following steps (S101-S106):

[0034] S101, Obtain the original dataset;

[0035] In this embodiment, the original dataset P0 includes multiple data entries, and each data entry consists of multiple data items. Specifically, the original dataset P0 includes various data formats. For ease of understanding, this embodiment will use the structured dataset of Alpaca as an example to introduce the method for protecting large language model datasets based on polluted lexical units provided in this embodiment. Those skilled in the art will understand that the method described in this embodiment using the structured dataset of Alpaca as an example is merely an illustrative example and does not constitute a limitation on this application.

[0036] In one application example, one data format of alpaca is: [

[0038] {

[0039] "instruction": "User instruction (required)",

[0040] "input": "User input (optional)",

[0041] "output": "Model response (required)",

[0042] "system": "System prompt word (optional)",

[0043] } ]

[0045] This application example demonstrates the data structure of a single data entry from the original dataset P0. In this example, each data entry includes four data items: "instruction", "input", "output", and "system".

[0046] As an optional implementation method in this embodiment, obtaining the original dataset includes: if the initial dataset includes multiple languages, splitting the initial dataset according to language types to obtain multiple original datasets; wherein each original dataset includes only one language. In this embodiment, if the initial dataset contains multiple languages, it needs to be split according to language types and then each original dataset is processed separately. Thus, each original dataset contains only one language, ensuring that the polluting lexical units match the language characteristics of the original datasets, ensuring that the polluting statements are highly similar to the data in the original datasets in terms of language patterns, and enhancing concealment.

[0047] S102, perform statistics on each data item in the original dataset to generate a contaminated data statistical template, and obtain the high-frequency common words and grammatical structure characteristics of the original dataset; wherein, the contaminated data statistical template records the existence probability of each data item;

[0048] In this embodiment, to generate contaminated data that is indistinguishable from genuine data, it is first necessary to find the linguistic feature patterns in the original dataset. As an optional implementation method in this embodiment, statistics are performed on each data item in the original dataset to generate a contaminated data statistical template, and the high-frequency commonly used words and grammatical structure characteristics of the original dataset are obtained, including the following steps:

[0049] a1. Perform existence probability statistics on each data item in the original dataset to generate a polluted data statistics template;

[0050] In this embodiment, taking the original dataset as an alpaca structured dataset as an example, the original dataset includes four data items: "instruction", "input", "output", and "system". The existence probability and distribution pattern of each data item (e.g., "instruction", "input", "output", and "system") in the dataset are analyzed. For example, if in the original dataset of 10,000 data items, the "input" data item exists in 5,000 (50%), and the "system" data item exists in 0, the statistical result is: "input" data item existence probability: 50%, "system" data item existence probability: 0%. Based on this statistical result, a contaminated data statistical template is generated, recording the existence probability of each data item to guide the data item filling strategy when generating the contaminated dataset Q. For example, each data item in the generated contaminated dataset Q has a 50% probability of generating the content of the "input" data item, and the "system" data item is always empty. Of course, in this embodiment, the existence probability of each data item will also be dynamically adjusted according to user-inputted adjustment instructions or when preset conditions are met, and is not fixed.

[0051] In this embodiment, by employing probability statistics and other methods, the data items in the original dataset data structure are statistically analyzed to generate a polluted data statistical template that records the probability of the existence of each data item. This determines the probability of each data item appearing when polluted data is generated subsequently, ensuring that the structure of the subsequently generated polluted data is highly consistent with the original dataset, thus achieving a pollution effect and enhancing its deceptiveness to attackers.

[0052] a2. String segmentation of each data item in the original dataset and linguistic feature analysis and statistics are performed to obtain high-frequency commonly used words. Syntactic structure characteristics are obtained by constructing a syntax tree. Among them, the grammatical structure characteristics include the probability of use of different grammatical structures in the original dataset.

[0053] In this embodiment, in addition to ensuring that the generated contaminated data is structurally highly consistent with the original dataset, it is also necessary to maintain consistency with the original dataset in terms of language patterns. Therefore, besides statistically analyzing the existence probability of data items in the original dataset, this embodiment also performs statistical analysis on the content of each data item.

[0054] Specifically, word segmentation is performed on the strings of each data item in each data entry. Different word segmentation tools can be used for different languages. For example, Jieba can be used for Chinese, and the word segmentation module in the Natural Language Toolkit (NLTK) can be used for English. These tools can segment strings into individual words or units based on the vocabulary and grammar rules of the language. The specific word segmentation principle is as follows: in addition to words in the dictionary, units also include suffixes, common expressions, names, etc. For example, in English, "running" can be segmented into two units: "run" and "-ing"; some common expressions like "in the end" are treated as a single unit; and names like "NewYork" are also units. Through this word segmentation method, the strings in the data items are transformed into meaningful basic units for subsequent analysis.

[0055] After word segmentation, the frequency of each word element in the original dataset is counted. For example, after word segmentation, "the" appears 1000 times, "generate" appears 200 times, and so on. This statistical analysis identifies the high-frequency words in the dataset. A subset of these high-frequency words are randomly selected when generating contamination word sequences, and these sequences are combined with contamination words to create a confusing effect.

[0056] Simultaneously, part-of-speech tagging tools (such as the part-of-speech tagger in NLTK) are used to tag each lexical unit with its part of speech, such as noun, verb, adjective, etc., and the frequency of words with different parts of speech is counted. For example, nouns may account for 40% of the total number of words, and verbs for 30%. In this embodiment, as an optional approach, the proportion of words with different parts of speech in the dataset can be recorded in the form of a table or dictionary, for example, {"noun": 40%, "verb": 30%, "adjective": 20%, "adverb": 10%}. This helps to determine the appropriate part-of-speech proportions for polluting lexical units, making the generated polluted sentences similar to the original dataset in terms of part-of-speech distribution.

[0057] In this embodiment, grammatical structure characteristics are obtained by constructing a syntax tree. A syntax tree is used to represent the grammatical structure of a sentence, displaying the relationships between various components in the sentence in a tree-like form, such as subject-verb-object, attributive, adverbial, and complement. Specifically, a natural language processing library (such as Stanford CoreNLP) can be used to construct the syntax tree. By analyzing the syntax tree, the frequency of different grammatical structures is statistically analyzed. For example, the subject-verb-object structure appears 60% of the time, the subject-verb-complement structure appears 30% of the time, and other structures appear 10% of the time. Information such as the connection relationships between different nodes in the syntax tree and the structure of subtrees can also be statistically analyzed. The high-frequency commonly used words and grammatical structure characteristics obtained through the above statistics will be used to guide the subsequent generation of polluted sentences using polluted word sequences, ensuring that the generated sentences conform to the grammatical structure characteristics of the original dataset.

[0058] In summary, by obtaining statistical templates, frequently used words, and grammatical structure characteristics of contaminated data through steps a1 and a2, we can guide the subsequent generation of contaminated data that is highly consistent with the original dataset in terms of structure and language patterns and achieves the contamination effect, thereby enhancing its deceptiveness to attackers.

[0059] S103, Generate a pollution word sequence. The pollution word sequence includes: high-frequency common words and pollution words. Among them, pollution words include: user-created words that do not exist in reality and their corresponding parts of speech, generated by arranging common characters.

[0060] In this embodiment, common characters are first randomly arranged to generate words that do not originally exist, i.e., "polluting words". Next, these "polluting words" are labeled and randomly assigned different parts of speech. These labeled polluting words are then combined with previously obtained high-frequency common words to form a "polluting word sequence". In some embodiments, a portion of high-frequency common words and polluting words are randomly selected and mixed. Polluting words constitute the majority of the sequence (e.g., 80%) and are randomly assigned parts of speech; high-frequency common words constitute a minority of the sequence (e.g., 20%) and retain their original parts of speech. Of course, the mixing ratio can be adjusted according to needs (e.g., 9:1). High-frequency words are only used to obfuscate attackers and only constitute a minority of the sequence. In this embodiment, high-frequency common words only serve an obfuscating function to prevent polluted data from being easily filtered out, and therefore only constitute a small portion of the polluting word sequence.

[0061] In one application example, suppose:

[0062] Pollution morphemes: "glorp" (verb), "zorxal" (noun), "tranix" (adjective);

[0063] Frequently used words: “generate” (verb), “user” (noun), “clear” (adjective).

[0064] The contamination word sequence generated with a mixing ratio of 3:2 is: ("glorp", "verb"), ("user", "noun"), ("tranix", "adjective"), ("generate", "verb"), ("zorxal", "noun").

[0065] In this embodiment, a small number of real high-frequency words (such as "generate") are mixed with a large number of polluting words in the polluting word sequence, which enhances the concealment of the polluting word sequence, making it difficult to identify the polluting words and making it difficult for thieves to separate the effective information.

[0066] S104, using the polluting word sequence to generate polluting sentences based on the grammatical structure characteristics of the original dataset;

[0067] After generating the contaminated lexical sequence, various contaminated sentences can be randomly generated using the lexical units in the sequence. As an optional implementation in this embodiment, generating contaminated sentences using the contaminated lexical sequence based on the syntactic structure characteristics of the original dataset includes the following steps:

[0068] b1. Randomly select the corresponding syntax structure according to the probability of use of different syntax structures in the original dataset;

[0069] In this embodiment, the frequency of different grammatical structures was statistically analyzed by analyzing the syntax tree. In actual implementation, the corresponding grammatical structures are selected according to their usage probability; for example, 60% of the structures are subject-verb-object, 30% are subject-verb-complement, and 10% are other structures.

[0070] b2. Randomly select word segments that match the part of speech from the polluted word segment sequence according to the part-of-speech requirements in the corresponding grammatical structure;

[0071] In this embodiment, taking the subject-verb-object structure as an example, according to the part-of-speech requirements in the grammatical structure of "subject (noun) + predicate (verb) + object (noun)," word elements with matching parts of speech are randomly selected from the polluted word element sequence. Taking the polluted word element sequence in step S103 as an example, "glorp" (verb), "zorxal" (noun), and "user" (noun) are randomly selected.

[0072] Taking the subject-verb-complement structure as an example, based on the part-of-speech requirements in the grammatical structure of "subject (noun) + linking verb (such as "is") + complement (adjective)," word elements that match the part of speech are randomly selected from the polluted word element sequence. Taking the polluted word element sequence in step S103 as an example, "user" (noun) and "tranix" (adjective) are randomly selected.

[0073] In this embodiment, there are many ways to generate polluting sentences. As one option, one can recursively select lexical units and then concatenate them according to grammatical rules to form a complete sentence. Of course, other language models can also be used, which will not be elaborated here.

[0074] b3. Generate polluting statements by concatenating word class matching lemmas according to the rules of the corresponding grammatical structure.

[0075] In one application example, the polluting statement generated based on the subject-verb-object syntax structure is: zorxal glorp user.

[0076] In another application example, the polluted statement generated based on the subject-relation table syntax structure is: User is tranix.

[0077] The sentences generated in this way conform to the language patterns of the original dataset in terms of grammatical structure, but contain meaningless polluting lexical units. Under normal circumstances, they will not have any actual semantic meaning, thus achieving the purpose of generating polluting sentences. This makes the data deceptive without affecting model training, thereby preventing the data from being used illegally.

[0078] S105, according to the existence probability recorded in the contaminated data statistics template, fill the contaminated statements into the corresponding data items of each data in the original dataset to obtain the contaminated data of the corresponding data items, and generate the contaminated dataset;

[0079] In this embodiment, as an optional implementation, according to the existence probability recorded in the contaminated data statistics template, contaminated statements are filled into the corresponding data items of each data item in the original dataset to obtain the contaminated data of the corresponding data items, generating a contaminated dataset, including:

[0080] For each data item in the original dataset, the polluting statements are filled into the corresponding data items in the following way to obtain the polluted data corresponding to the data item: Iterate through the existence probability of all data items in a data item, and fill the corresponding data item with the polluting statements according to the existence probability of each data item to obtain a polluted data item.

[0081] Specifically, taking the original dataset as an example of an alpaca structured dataset, we iterate through the four data items of the original dataset: "instruction," "input," "output," and "system." For instance, if the probability of the "input" data item being present is 50%, then for each generated corrupted data item, there is a 50% chance that a corrupted statement will be filled in this data item. On the other hand, if the probability of the "system" data item being present is 0%, then for each generated corrupted data item, no corrupted statement will be filled in this data item. This process continues until all data items in the original dataset are transformed into corrupted data. The resulting corrupted dataset is highly consistent with the original dataset in terms of structure and language patterns and effectively achieves the desired corruption effect.

[0082] The following example, which generates two pollution data points, is used to illustrate the process in comparison with step S105.

[0083] The process for generating the first pollution data is as follows:

[0084] Iterate through the existence probability of the four data items and fill in the pollution statement:

[0085] The data item "instruction" has a 100% probability of existence and is required. The polluting statement "glorpzorxal tranix user" will be randomly selected and filled into this data item. The data item "input" has a 50% probability of existence and will be randomly selected and filled into this data item. The polluting statement "quarn flim input" will be randomly selected and filled into this data item. The data item "output" has a 100% probability of existence and is required. The polluting statement "dral flemb generate result" will be randomly selected and filled into this data item. The data item "system" has a 0% probability of existence and will not be filled.

[0086] The first pollution data point generated is: [

[0088] {

[0089] "instruction": "glorp zorxal tranix user",

[0090] "input": "quarn flim input",

[0091] "output": " dral flemb generated result",

[0092] } ]

[0094] The process for generating the first pollution data is as follows:

[0095] Iterate through the existence probability of the four data items and fill in the pollution statement:

[0096] The data item "instruction" has a 100% probability of existence and is required. A randomly selected polluting statement "tranixquarn generate request" will be used to fill this data item. The data item "input" has a 50% probability of existence and will not be filled in this time. The data item "output" has a 100% probability of existence and is required. A randomly selected polluting statement "glorp quarn output" will be used to fill this data item. The data item "system" has a 0% probability of existence and will not be filled in.

[0097] The second pollution data point generated is: [

[0099] {

[0100] "instruction": "tranix quarn generate request",

[0101] "output": "glorp quarn output"

[0102] } ]

[0104] Those skilled in the art will understand that the embodiments of this application are described using the generation process of the above two pollution data as an example, and are merely illustrative and do not constitute a limitation on this application.

[0105] Therefore, the probability of the existence of data items in the polluted dataset Q generated in this embodiment is completely consistent with that of the original dataset P0 (e.g., "input" exists only 50%), making it impossible for attackers to identify polluted data through the difference in the missing data item rate. Furthermore, the filled polluted data is highly similar to the real data in grammatical structure, making it difficult to filter out simply.

[0106] S106 uses polluted statements and polluted datasets to pollute the original dataset, stores the polluted locations, and generates a protected dataset.

[0107] In this embodiment, the goal of contaminating the original dataset is to insert contaminated statements into the original dataset P0 and accurately record the contaminated locations so that the contaminated parts can be automatically removed by a filter generated based on the contaminated locations during subsequent training, without affecting the uncontaminated original dataset.

[0108] In specific implementation, inserting polluting statements into the original dataset P0 includes two methods: one is to insert polluting statements into certain data items of the original dataset P0, and the other is to insert the entire polluted data from the polluted dataset Q after a certain data item in the original dataset P0. In this embodiment, the pollution positions include: a first pollution position and a second pollution position. As an optional implementation method in this embodiment, the original dataset is polluted using polluting statements and the polluted dataset, and the pollution positions are stored to obtain a protected dataset, including the following steps (c1-c2):

[0109] c1. Insert a pollution statement into the target entry data of the original dataset and store the first pollution position of the pollution statement insertion to generate the process dataset; wherein, the first pollution position includes: the data entry number of the target entry data in the original dataset, the pollution data item in the target entry data, and the starting string coordinates and ending string coordinates of the pollution statement in the pollution data item;

[0110] Step c1 represents the first type of contamination. The target data entries are the original data to be contaminated. In this embodiment, the target data entries include multiple entries. For example, a portion of the data (e.g., 10% of the data) is randomly selected from the original dataset P0, and contamination statements are inserted into multiple original data entries to generate the contaminated process dataset P1. Optionally, for each selected original data entry, the data entry to be contaminated is selected based on the probability of its existence. For example, the data entry "instruction" is mandatory and is contaminated first.

[0111] In this embodiment, the index of the target entry data in the original dataset is the index of the original data to be contaminated in the original dataset. For example, the 5th original data item is selected for contamination. The key of the contaminated data item is selected, for example, "instruction" is selected to insert a contaminated statement. The starting string coordinates start_pos of the contaminated statement in the contaminated data item represent the starting position of the contamination, that is, from which character in the original data the contaminated statement is inserted. For example, if it is inserted after the 6th character, start_pos=6. The ending string coordinates end_pos of the contaminated statement in the contaminated data item represent the ending position of the contamination, which is the starting string coordinates start_pos + the string length of the contaminated statement. For example, if the string length of the contaminated statement is 11, then end_pos=6+11=17. Finally, the "index", "key", "start_pos", and "end_pos" of the insertion position are recorded as the first contamination position and stored in the contamination position information table C0. This information is used to generate filter F1 later, which filters out these inserted contaminated statements during LLM training.

[0112] The following is an application example, using the target entry data as data entry number index=0 in the original dataset P0, to explain the implementation of step c1. Specifically, it includes the following steps:

[0113] d1. Select the target entry data (index "index=0"), the content of which is as follows: [

[0115] {

[0116] "instruction": "Generate a brief introduction about artificial intelligence",

[0117] "input": "",

[0118] "output": "Artificial intelligence is the science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence."

[0119] } ]

[0121] d2. Select the data item to be contaminated, for example, select the data item "instruction" for contamination;

[0122] d3. Select the pollution statement to be inserted, for example, "glorp zorxal" (11 characters long, including spaces);

[0123] d4. Insert a pollution statement into the data item "instruction";

[0124] The original sentence for the data item "instruction" is "Generate a brief introduction about artificial intelligence" (19 characters long). The insertion position is after the 6th character (i.e. after "generate a brief introduction"). The polluted statement becomes: "Generate a brief introduction about artificial intelligence by glorp zorxal".

[0125] d5. Record the location of the first contamination and store it in C0.

[0126] Specifically, the first pollution location information for C0 is as follows:

[0127] {"index":0,

[0128] "key": "instruction",

[0129] "start_pos": 6,

[0130] "end_pos": 17

[0131] }

[0132] Specifically, the contaminated data in the contaminated process dataset P1 with data entry index=0 is: [

[0134] {

[0135] "instruction": "Generate a brief introduction to artificial intelligence from glorp zorxal".

[0136] "input": "",

[0137] "output": "Artificial intelligence is the science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence."

[0138] } ]

[0140] Those skilled in the art will understand that the embodiments of this application, using the example of contaminating the target entry data with data entry sequence number index=0, are merely illustrative and do not constitute a limitation on this application. In specific implementations, the same contamination statement can be inserted into multiple data items. For example, in the above application example, a contamination statement can also be inserted into the data item "output". In this embodiment, the insertion position and length of the contamination statement are randomized to prevent attackers from identifying the contamination content through a fixed pattern. The same method can be used to insert contamination statements into other data entry sequence numbers of the original dataset P0, which will not be elaborated here.

[0141] c2. Insert the contaminated data from the contaminated dataset into the process dataset, store the second contaminated position of the contaminated dataset insertion, and update the data entry sequence number of the first contaminated position to generate the protected dataset; wherein, the second contaminated position includes: the data entry sequence number of the inserted contaminated data in the process dataset.

[0142] Step c2 represents the second type of contamination, inserting an entire contaminated data entry from the contaminated dataset Q into the process dataset P1, becoming a contaminated data entry in the protective dataset P2. Specifically, the entire contaminated dataset Q can be randomly inserted after a contaminated data entry in P1, for example, after the last contaminated data entry in P1. Alternatively, individual contaminated data entries from the contaminated dataset Q can be inserted into P1 according to preset rules (e.g., sequentially inserting one contaminated data entry from Q after each contaminated data entry in P1) or randomly inserted into any position in P1, generating the protective dataset P2. P1 and Q are merged into P2, and the data entry sequence numbers also change. The inserted data entry sequence numbers are then sequentially numbered, and the position information of the inserted contaminated data in the renumbered protective dataset P2 is recorded, i.e., the renumbered data entry sequence number "index" is recorded as the second contamination position, stored in the contamination position information table C2. This information is used to generate filter F2 later, filtering out these inserted contaminated statements during LLM training. Since the sequence number has changed, the sequence number of the data entry in the first contaminated position also needs to be updated, but the "key", "start_pos", and "end_pos" in the first contaminated position remain unchanged. C0 is updated to C1.

[0143] In step c2, only the data entry sequence number "index" is considered. Therefore, C2 primarily stores the index values ​​of these inserted contaminated data in the protection dataset P2. C0 contains the contamination location information recorded when P0 was previously contaminated. As data from Q is inserted into P1, the index positions of the data in P1 change. Therefore, C0 needs to be adjusted and updated based on these updated location information "index" to obtain C1, ensuring that the first contamination location information in C1 remains accurate in the new protection dataset.

[0144] The implementation of step c2 is explained and described below in an application example.

[0145] In this application example, the contents of the process dataset P1, the pollution dataset Q, and the first pollution location information C0 are as follows:

[0146] Process dataset P1 includes 3 data entries, as follows:

[0147] P1 = [

[0148] {"instruction": "Task 1 description", "input": "Relevant input 1", "output": "Task 1 output"}

[0149] {"instruction": "Task 2 description", "input": "Relevant input 2", "output": "Task 2 output"}

[0150] {"instruction": "Task 3 description", "input": "Relevant input 3", "output": "Task 3 output"} ]

[0152] The data entry number for the first data entry is "index=0", the data entry number for the second data entry is "index=1", and the data entry number for the third data entry is "index=2".

[0153] The contaminated dataset Q contains two contaminated data points, as detailed below:

[0154] Q = [

[0155] {"instruction": "Pollution Instruction 1", "input": "Pollution Input 1", "output": "Pollution Output 1"}

[0156] {"instruction": "Pollution Instruction 2", "input": "Pollution Input 2", "output": "Pollution Output 2"} ]

[0158] The first pollution location information C0 includes two entries, as follows:

[0159] C0 = [

[0160] {"index": 1, "key": "instruction", "start_pos": 5, "end_pos": 10}, {"index": 2, "key": "output", "start_pos": 8, "end_pos": 13} ]

[0162] The first contamination location information indicates that the instruction data item of the second contaminated data in P1 (index "index=1" in P1) is contaminated at positions 5 to 10; the second contamination location information indicates that the output data item of the third contaminated data in P1 (index "index=2" in P1) is contaminated at positions 8 to 13.

[0163] In this application example, the two contaminated data entries from Q are inserted sequentially into P1. Specifically, the first contaminated data entry from Q is inserted after the first data entry in P1 ("index=0"), and the second contaminated data entry from Q is inserted after the second data entry in P1 ("index=1"). The merged protected dataset P2 is:

[0164] P2= [

[0165] {"instruction": "Task 1 description", "input": "Relevant input 1", "output": "Task 1 output"},

[0166] {"instruction": "Pollution instruction 1", "input": "Pollution input 1", "output": "Pollution output 1"},

[0167] {"instruction": "Task 2 description", "input": "Relevant input 2", "output": "Task 2 output"},

[0168] {"instruction": "Pollution instruction 2", "input": "Pollution input 2", "output": "Pollution output 2"},

[0169] {"instruction": "Task 3 description", "input": "Relevant input 3", "output": "Task 3 output"} ]

[0171] Record the location information of the second pollution source and store it as C2, including 2 entries, as follows:

[0172] C2 = [

[0173] {"index": 1}

[0174] {"index": 3} ]

[0176] The first second pollution location information indicates that the first pollution data in Q is inserted into the protection dataset P2 at the position of "index=0" (originally the position of "index=1" in P1); the second second pollution location information indicates that the second data in Q is inserted into the protection dataset at the position of "index=3" (originally P1 did not have this position).

[0177] In this application example, C0 represents the contamination location information recorded when P0 was previously contaminated. As data from Q is inserted into P1, the index positions of the data in P1 change. Therefore, C0 needs to be adjusted and updated to C1 based on these updated location information "index" to ensure that the first contamination location information in C1 remains accurate in the new protected dataset. Optionally, by iterating from C0 to C1, the data in C0 that originally had "index" 1 and 2 now have index positions of 2 and 4 respectively. Therefore, C1 becomes:

[0178] C1 = [

[0179] {"index": 2, "key": "instruction", "start_pos": 5, "end_pos": 10},

[0180] {"index": 4, "key": "output", "start_pos": 8, "end_pos": 13} ]

[0182] As an optional implementation in this embodiment, the method provided in this application embodiment further includes: generating a first filter F1 based on a first contamination location; generating a second filter F2 based on a second contamination location; when the protection dataset is input into a large language model for training, loading the first filter F1 into the large language model to filter out contaminated statements indicated by the first contamination location; and loading the second filter F2 into the large language model to filter out contaminated data indicated by the second contamination location.

[0183] In this embodiment, filter F1 can be generated using the first contamination location information in C1 to filter out contaminated statements at specified locations. Similarly, filter F2 can be generated using the second contamination location information in C2 to filter out the entire contaminated data at a specified data entry position (index). When applying the first filter F1 and the second filter F2 to the data processing module of the LLM large model and using the final protected dataset P2 for large model training, the first filter F1 and the second filter F2 directly locate the contamination location through the index, eliminating the need to traverse the entire text. This results in extremely low computational overhead and ensures efficient processing of contaminated data during model training while protecting the security and usability of the original dataset.

[0184] The method for protecting large language model datasets based on contaminated lexical units provided in this embodiment eliminates the need for LLM to load the decryption module into the training program. By simply filtering out contaminated data, the training efficiency of the protected dataset can be significantly improved, reducing hardware and time costs. Furthermore, if an attacker steals the contaminated dataset for training a large LLM model, not only will the original training effect be lost, but the various metrics of the large model will also be significantly reduced. In some cases, the unfiltered contaminated content may even cause the model performance to crash, thus protecting the security of the original dataset.

[0185] An exemplary embodiment of this application provides a protection device 10 for a large language model dataset based on polluting lexical units. Figure 2 This illustration shows a structural block diagram of a large language model dataset protection device 10 based on polluted lexical units, provided in an exemplary embodiment of this application. The following is only a brief description of the structure and function of this large language model dataset protection device 10 based on polluted lexical units; for other matters not covered herein, please refer to the relevant descriptions in the aforementioned method for protecting large language model datasets based on polluted lexical units. The embodiment of this large language model dataset protection device 10 based on polluted lexical units corresponds to the embodiment of the aforementioned method for protecting large language model datasets based on polluted lexical units. All implementation processes and methods of the aforementioned method embodiments can be applied to this embodiment of the large language model dataset protection device based on polluted lexical units, and can achieve the same technical effects.

[0186] like Figure 2 As shown, the large language model dataset protection device 10 based on polluting lexical units includes:

[0187] The data preprocessing module 100 is used to obtain the original dataset; to perform statistics on each data item in the original dataset, generate a contaminated data statistical template, and obtain the high-frequency common words and grammatical structure characteristics of the original dataset; wherein, the contaminated data statistical template records the existence probability of each data item;

[0188] The contaminated data generation module 200 is used to generate a contaminated word sequence, which includes high-frequency common words and contaminated words. The contaminated words include: user-created words that do not exist in reality and their corresponding parts of speech, generated by arranging common characters. Based on the grammatical structure characteristics of the original dataset, the contaminated words are used to generate contaminated sentences. According to the existence probability recorded in the contaminated data statistics template, the contaminated sentences are filled into the corresponding data items of each data item in the original dataset to obtain the contaminated data of the corresponding data items, thus generating the contaminated dataset.

[0189] The protected dataset generation module 300 is used to contaminate the original dataset using contaminated statements and contaminated datasets, and store the contaminated locations to obtain a protected dataset.

[0190] In this embodiment, the original dataset includes multiple data entries, and each data entry's data structure consists of multiple data items. This embodiment will use the structured dataset of Alpaca as an example to introduce the large language model dataset protection device based on polluted lexical units provided in this embodiment. For specific examples, please refer to the description in the above embodiment of the method for protecting large language model datasets based on polluted lexical units, which will not be repeated here.

[0191] In this embodiment, as an optional implementation, the data preprocessing module 100 obtains the original dataset in the following way: if the initial dataset includes multiple languages, the initial dataset is segmented according to language type to obtain multiple original datasets; wherein each original dataset includes only one language. In this embodiment, if the initial dataset contains multiple languages, it needs to be segmented according to language type and each original dataset is processed separately. Thus, each original dataset contains only one language, ensuring that the polluting lexical units match the language characteristics of the original dataset, ensuring that the polluting statements are highly similar to the data in the original dataset in terms of language patterns, and enhancing concealment.

[0192] In this embodiment, as an optional implementation, the data preprocessing module 100 performs statistical analysis on each data item of the original dataset in the following manner to generate a polluted data statistical template and obtain the high-frequency common words and grammatical structure characteristics of the original dataset: it performs existence probability statistics on each data item of the original dataset to generate a polluted data statistical template; it performs string segmentation and language feature analysis statistics on each data item of the original dataset to obtain high-frequency common words, and obtains grammatical structure characteristics by constructing a syntax tree; wherein, the grammatical structure characteristics include the probability of using different grammatical structures in the original dataset. Thus, it obtains the polluted data statistical template, high-frequency common words, and grammatical structure characteristics to guide the subsequent generation of polluted data that is highly consistent with the original dataset in structure and language pattern and achieves a polluting effect, thereby enhancing its deceptiveness to attackers.

[0193] In this embodiment, the contaminated data generation module 200 first needs to randomly arrange commonly used characters to generate some words that do not originally exist, namely "contaminated morphemes". Next, these "contaminated morphemes" are marked and randomly assigned different parts of speech. These marked contaminated morphemes are combined with previously obtained high-frequency commonly used words to form a "contaminated morpheme sequence". In some embodiments, a portion of high-frequency commonly used words and contaminated morphemes are randomly selected and mixed. The contaminated morphemes constitute the majority of the sequence (e.g., 80%) and are randomly assigned parts of speech; the high-frequency commonly used words constitute a minority of the sequence (e.g., 20%) and retain their original parts of speech. Of course, the mixing ratio can be adjusted according to needs (e.g., 9:1). The high-frequency words are only used to obfuscate attackers and only constitute a minority of the sequence. In this embodiment, the high-frequency commonly used words only serve an obfuscating function to prevent the contaminated data from being easily filtered out, and therefore only constitute a small portion of the contaminated morpheme sequence. In this embodiment, by mixing a small number of real high-frequency words (e.g., "generate") with a large number of contaminated morphemes in the contaminated morpheme sequence, the concealment of the contaminated morpheme sequence is enhanced, making it difficult to identify the contaminated morphemes and difficult for theftsters to separate valid information.

[0194] After generating the polluting word sequence, as an optional implementation method in this embodiment, the polluting data generation module 200 generates polluting sentences using the polluting word sequence according to the grammatical structure characteristics of the original dataset in the following way: randomly selecting the corresponding grammatical structure according to the usage probability of different grammatical structures in the original dataset; randomly selecting word segments that match the part of speech from the polluting word sequence according to the part-of-speech requirements in the corresponding grammatical structure; and concatenating word segments that match the part of speech according to the rules of the corresponding grammatical structure to generate polluting sentences.

[0195] In this embodiment, the frequency of different grammatical structures was statistically analyzed by analyzing the syntax tree. In actual implementation, the corresponding grammatical structures are selected according to their usage probability; for example, 60% of the structures are subject-verb-object, 30% are subject-verb-complement, and 10% are other structures.

[0196] In one application example, taking the subject-verb-object structure as an example, based on the part-of-speech requirements in the grammatical structure of "subject (noun) + verb (verb) + object (noun)," lexics that match the part of speech are randomly selected from the polluted lexicographical sequence. Taking the polluted lexicographical sequence in step S103 above as an example, "glorp" (verb), "zorxal" (noun), and "user" (noun) are randomly selected. The polluted sentence generated based on the subject-verb-object grammatical structure is: zorxal glorp user.

[0197] The sentences generated in this way conform to the language patterns of the original dataset in terms of grammatical structure, but contain meaningless polluting lexical units. Under normal circumstances, they will not have any actual semantic meaning, thus achieving the purpose of generating polluting sentences. This makes the data deceptive without affecting model training, thereby preventing the data from being used illegally.

[0198] In this embodiment, as an optional implementation, the contaminated data generation module 200 fills the contaminated statements into the corresponding data items of each data item in the original dataset according to the existence probability recorded in the contaminated data statistical template to obtain the contaminated data of the corresponding data items, thereby generating a contaminated dataset:

[0199] For each data item in the original dataset, the polluting statements are filled into the corresponding data items in the following way to obtain the polluted data corresponding to the data item: Iterate through the existence probability of all data items in a data item, and fill the corresponding data item with the polluting statements according to the existence probability of each data item to obtain a polluted data item.

[0200] Specifically, taking the original dataset as an example of an alpaca structured dataset, the contamination data generation module 200 iterates through the four data items of the original dataset: "instruction," "input," "output," and "system." For example, if the probability of the "input" data item being present is 50%, then for each generated contamination data item, there is a 50% chance that a contamination statement will be filled in this data item. Conversely, if the probability of the "system" data item being present is 0%, then for each generated contamination data item, no contamination statement will be filled in this data item. This process continues until all data items in the original dataset are transformed into contamination data. The resulting contamination dataset Q is highly consistent with the original dataset P0 in terms of structure and language pattern and effectively achieves the contamination effect.

[0201] In this embodiment, the goal of contaminating the original dataset is to insert contaminated statements into the original dataset P0 and accurately record the contamination locations. This allows subsequent training to automatically remove contaminated portions using filters generated based on the contamination locations, without affecting the uncontaminated original dataset. In this embodiment, the contamination locations include: a first contamination location and a second contamination location. As an optional implementation in this embodiment, the protected dataset generation module 300 contaminates the original dataset using contaminated statements and a contaminated dataset, and stores the contamination locations to obtain a protected dataset: A contaminated statement is inserted into the target entry data of the original dataset, and the first contamination location of the inserted statement is stored, generating a process dataset. The first contamination location includes: the data entry number of the target entry data in the original dataset, the contaminated data item in the target entry data, and the starting and ending string coordinates of the contaminated statement within the contaminated data item. The contaminated data from the contaminated dataset is inserted into the process dataset, and the second contamination location of the inserted contaminated dataset is stored. The data entry number at the first contamination location is updated, generating a protected dataset. The second contamination location includes: the data entry number of the inserted contaminated data in the process dataset.

[0202] In this embodiment, the target data entries include multiple entries. For example, a portion of the data (e.g., 10% of the data) is randomly selected from the original dataset P0, and pollution statements are inserted into multiple original data entries in the original dataset to generate a polluted process dataset P1. Optionally, for each selected original data entry, the data entry to be polluted is selected based on the probability of its existence. For example, the data entry "instruction" is required and is polluted first.

[0203] In this embodiment, the index of the target entry data in the original dataset is the index of the original data to be contaminated in the original dataset. For example, the 5th original data item is selected for contamination. The key of the contaminated data item is selected, for example, "instruction" is selected to insert a contaminated statement. The starting string coordinate start_pos of the contaminated statement in the contaminated data item represents the starting position of the contamination, that is, from which character in the original data the contaminated statement is inserted. For example, if it is inserted after the 6th character, start_pos=6. The ending string coordinate end_pos of the contaminated statement in the contaminated data item represents the ending position of the contamination, which is the starting string coordinate start_pos + the string length of the contaminated statement. For example, if the string length of the contaminated statement is 11, then end_pos=6+11=17. Finally, the "index", "key", "start_pos", and "end_pos" of the insertion position are recorded and stored in the contamination position information table C0 as the first contamination position, which is used to generate filter F1 later to filter out these inserted contaminated statements during LLM training.

[0204] In this embodiment, the entire contaminated data in the contaminated dataset Q is inserted into the process dataset P1 to become a contaminated data in the protection dataset P2. Specifically, the entire contaminated dataset Q can be randomly inserted after a contaminated data in P1, for example, after the last contaminated data in P1. Alternatively, each contaminated data in the contaminated dataset Q can be inserted into P1 according to a preset rule (such as inserting a contaminated data of Q after each contaminated data in P1 in sequence) or randomly inserted into any position in P1 to generate the protection dataset P2. P1 and Q are merged into P2, and the data entry sequence number also changes. The inserted data entry sequence number is numbered sequentially, and the position information of the inserted contaminated data in the renumbered protection dataset P2 is recorded, that is, the renumbered data entry sequence number "index" of the inserted contaminated data is recorded and stored as the contaminated position information table C2, as the second contaminated position, which is used to generate filter F2 later to filter out these inserted contaminated statements during LLM training. Since the sequence number has changed, the sequence number of the data entry in the first contaminated position also needs to be updated, but the "key", "start_pos", and "end_pos" in the first contaminated position remain unchanged. C0 is updated to C1.

[0205] In this embodiment, C2 primarily stores the index values ​​of the inserted contaminated data in the protection dataset P2. C0 is the contamination location information recorded when P0 was previously contaminated. As data from Q is inserted into P1, the index positions of the data in P1 change. Therefore, C0 needs to be adjusted and updated based on these updated location information "index" to obtain C1, ensuring that the first contamination location information in C1 remains accurate in the new protection dataset.

[0206] As an optional implementation method in this embodiment, such as Figure 3 As shown, the apparatus 10 provided in this application embodiment further includes: a filter generation module 400, used to generate a first filter F1 according to a first contamination location; generate a second filter F2 according to a second contamination location; when the protection dataset is input into a large language model for training, the first filter F1 is loaded into the large language model to filter out the contaminated statements indicated by the first contamination location C1; and the second filter F2 is loaded into the large language model to filter out the contaminated data indicated by the second contamination location C2.

[0207] In this embodiment, the filter generation module 400 can generate filter F1 using the first contamination location information in C1 to filter out contaminated statements at specified locations. Similarly, it can generate filter F2 using the second contamination location information in C2 to filter out the entire contaminated data at a specified data entry position (index). When the first filter F1 and the second filter F2 are applied to the data processing module of the LLM large model, and the final protected dataset P2 is used for large model training, the first filter F1 and the second filter F2 directly locate the contamination location through the index, eliminating the need to traverse the entire text. This results in extremely low computational overhead and ensures efficient processing of contaminated data during model training while protecting the security and usability of the original dataset.

[0208] The large language model dataset protection device based on polluted lexical units provided in this embodiment allows LLM to avoid loading the decryption module into the training program. It only needs to filter out polluted data, which can greatly improve the training efficiency of the protected dataset and reduce hardware and time costs. In addition, if an attacker steals the polluted dataset to train the large LLM model, not only will it fail to achieve the original training effect, but it will also significantly reduce the various indicators of the large model, and may even cause the model performance to crash due to the unfiltered polluted content, thereby protecting the security of the original dataset.

[0209] Figure 4The diagram illustrates a structural block diagram of an electronic device 1000 according to an exemplary embodiment of this application. The electronic device 1000 can be implemented as the aforementioned large language model dataset protection device based on polluted lexical units, which can be configured in electronic devices such as smartphones, tablets, laptops, desktop computers, smartwatches, televisions, or servers.

[0210] Typically, electronic device 1000 includes a processor 1001 and a memory 1002.

[0211] Processor 1001 may include one or more processing cores, such as a quad-core processor, a deca-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0212] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one instruction, which is executed by the processor 1001 to implement all or part of the steps in the method embodiment of this application for protecting large language model datasets based on polluted lexical units.

[0213] Those skilled in the art will understand that Figure 4The structure shown does not constitute a limitation on the electronic device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0214] In one exemplary embodiment, a readable storage medium is also provided, which stores a program or instructions that, when executed by a processor, implement all or part of the steps in the above-described method for protecting large language model datasets based on polluted lexical units. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0215] In one exemplary embodiment, a computer program product is also provided, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the aforementioned... Figures 1 to 3 All or part of the steps of the large language model dataset protection method based on polluting lexical units shown in any embodiment.

[0216] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0217] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for protecting large language model datasets based on polluting lexical units, characterized in that, include: Obtain the original dataset; The data items of each data item in the original dataset are statistically analyzed to generate a polluted data statistical template, and the high-frequency common words and grammatical structure characteristics of the original dataset are obtained; wherein, the polluted data statistical template records the existence probability of each data item; Generate a pollution lexical sequence, the pollution lexical sequence including: the high-frequency common words and pollution lexical elements, wherein the pollution lexical elements include: user-created words that do not exist in reality and their corresponding parts of speech generated by arranging common characters; Polluted sentences are generated using the polluted lexical sequence based on the syntactic structure characteristics of the original dataset; According to the existence probability recorded in the pollution data statistics template, the pollution statement is filled into the corresponding data item of each data in the original dataset to obtain the pollution data of the corresponding data item, and a pollution dataset is generated. The original dataset is contaminated using the contaminated statements and the contaminated dataset, and the contaminated locations are stored to generate a protected dataset.

2. The method according to claim 1, characterized in that, The process of obtaining the original dataset includes: If the initial dataset includes multiple languages, the initial dataset is divided according to language to obtain multiple original datasets; wherein each original dataset includes only one language.

3. The method according to claim 1, characterized in that, The process involves statistically analyzing each data item in the original dataset to generate a polluted data statistical template, and obtaining the high-frequency commonly used words and grammatical structure characteristics of the original dataset, including: The existence probability of each data item in the original dataset is statistically analyzed to generate the contaminated data statistical template. The strings of each data item in the original dataset are segmented and language feature analysis is performed to obtain the high-frequency commonly used words. The syntax structure characteristics are obtained by constructing a syntax tree. The syntax structure characteristics include the probability of use of different syntax structures in the original dataset.

4. The method according to claim 3, characterized in that The step of generating polluted sentences using the polluted lexical sequence based on the syntactic structure characteristics of the original dataset includes: Randomly select the corresponding syntax structure according to the probability of use of different syntax structures in the original dataset; According to the part-of-speech requirements in the corresponding grammatical structure, randomly select part-of-speech matching lexical units from the polluted lexical unit sequence; The polluted statement is generated by concatenating the part-of-speech matching lexical units according to the rules of the corresponding grammatical structure.

5. The method according to claim 1, characterized in that, The process of filling the polluted statements into the corresponding data items of each data entry in the original dataset according to the existence probability recorded in the polluted data statistics template to obtain the polluted data of the corresponding data entries, and generating a polluted dataset, includes: For each data entry in the original dataset, the polluting statements are populated into the corresponding data items in the following manner to obtain the polluted data corresponding to the data entry: Iterate through the existence probability of all data items in a data set, and fill the corresponding data item with the pollution statement based on the existence probability of each data item to obtain a polluted data set.

6. The method according to claim 1, characterized in that, The contaminated locations include: a first contaminated location and a second contaminated location; The process of using the polluting statements and the polluted dataset to pollute the original dataset, and storing the polluted locations to obtain a protected dataset, includes: The pollution statement is inserted into the target entry data of the original dataset, and the first pollution position of the pollution statement is stored to generate a process dataset; wherein, the first pollution position includes: the data entry number of the target entry data in the original dataset, the pollution data item in the target entry data, and the start string coordinates and end string coordinates of the pollution statement in the pollution data item; The contaminated data of the contaminated dataset is inserted into the process dataset, the second contaminated position of the contaminated dataset is stored, and the data entry sequence number of the first contaminated position is updated to generate the protection dataset; wherein, the second contaminated position includes: the data entry sequence number of the inserted contaminated data in the process dataset.

7. The method according to claim 6, characterized in that, The method further includes: A first filter is generated based on the first contamination location; A second filter is generated based on the second contamination location; When the protected dataset is input into the large language model for training, the first filter is loaded into the large language model to filter out the polluted sentences indicated by the first polluted location; the second filter is loaded into the large language model to filter out the polluted data indicated by the second polluted location.

8. A protection device for a large language model dataset based on polluting lexical units, characterized in that, include: The data preprocessing module is used to obtain the raw dataset; The data items of each data item in the original dataset are statistically analyzed to generate a polluted data statistical template, and the high-frequency common words and grammatical structure characteristics of the original dataset are obtained; wherein, the polluted data statistical template records the existence probability of each data item; A contaminated data generation module is used to generate a contaminated lexical sequence, which includes: the high-frequency common words and contaminated lexical units, wherein the contaminated lexical units include: user-created words that do not exist in reality and their corresponding parts of speech generated by arranging common characters; using the contaminated lexical sequence to generate contaminated sentences according to the grammatical structure characteristics of the original dataset; and filling the contaminated sentences into the corresponding data items of each data item in the original dataset according to the existence probability recorded in the contaminated data statistics template to obtain the contaminated data of the corresponding data items, thereby generating a contaminated dataset. The protected dataset generation module is used to contaminate the original dataset using the contaminated statements and the contaminated dataset, and store the contaminated locations to obtain the protected dataset.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method for protecting large language model datasets based on polluting lexical units as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when executed by a processor, the program or instructions implement the steps of the method for protecting large language model datasets based on polluted lexical units as described in any one of claims 1 to 7.

11. A computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the method for protecting a large language model dataset based on polluted lexical units as described in any one of claims 1 to 7.

Images