Information hiding method based on backdoors in large language models

Abstract

Disclosed in the present invention is an information hiding method based on backdoors in large language models. In the method, firstly, in an information hiding phase, an information sender encodes secret information to be transmitted to obtain a secret information random character string, the random character string is segmented and triggers are allocated to obtain n trigger-secret information random character string segment pairs, and said pairs are embedded into several pieces of context; secondly, a training set and a test set are constructed, and a large language model is fine-tuned and is tested by means of the test set; then, in an information extraction phase, an information receiver obtains, on the basis of the information sent by the sender in combination with the fine-tuned large language model, outputs corresponding to all the triggers, and the outputs are concatenated to obtain the secret information random character string; and finally, the secret information random character string obtained by means of concatenation is decoded to obtain the secret information transmitted by the information sender. The present invention enables the information hiding method to have both high concealment and security, and also allows the amount of information that can be transmitted at one time to no longer be limited by a carrier.

Description

Information hiding method based on large language model backdoor TECHNICAL FIELD

[0001] The present application belongs to the field of information security, and particularly relates to an information hiding method based on a large language model backdoor. BACKGROUND

[0002] With the continuous development of digital information technology and network communication technology, the transmission of digital information in the network has become more and more convenient. However, due to the interconnectivity and openness of the Internet, digital information transmitted through the Internet is vulnerable to malicious attacks or theft, resulting in a great threat to the security of digital information. In order to solve this problem, information hiding technology has been proposed and has developed rapidly with the development of science and technology, from traditional information hiding technology to information hiding technology combined with deep neural networks, and now to information hiding technology combined with large models.

[0003] Traditional information hiding technology is based on the limitations of human senses, by encoding secret information and embedding it into a carrier, and controlling the carrier distortion rate (the impact and changes generated by the carrier) to a level that is difficult for humans to perceive, thereby hiding secret information in the carrier, and transmitting secret information covertly by transmitting the carrier, to achieve covert communication between the two parties. Traditional information hiding technology is the basis of the entire information hiding field, and researchers need to have certain knowledge of information hiding, and in order to ensure the carrier distortion rate, the information embedding capacity of a carrier is usually low.

[0004] Information hiding technology combined with deep neural networks has developed in two directions: one is to use neural network models to reduce the carrier distortion rate after embedding secret information, making it more difficult for humans to perceive, and having better information embedding capacity than traditional information hiding technology; the second is to use neural network models as a new carrier, without training the model, but encoding secret information and embedding it into the parameters of an existing neural network model using a substitution algorithm, and ensuring that the model does not cause human-perceptible loss after embedding information.

[0005] In recent years, large models have developed rapidly, and researchers have begun to study the combination of information hiding technology and large models, by affecting the sampling process of large models when generating text or images to embed secret information, so as to use the powerful generation ability of large models to generate a more natural and high-quality text or image embedded with secret information, further reducing the carrier distortion rate and improving the imperceptibility of information hiding.

[0006] It can be seen that the information hiding technology from the traditional to the neural network to the large model, the information embedding amount is increasing, the carrier distortion rate is decreasing, the concealment and security are also increasing, but they are all not out of the carrier, therefore, in order to ensure the carrier distortion rate and the concealment and security, the information embedding amount is greatly limited by the carrier, and researchers need to well balance the information embedding amount and the carrier distortion rate.

[0007] With the continuous development of large models and the wide popularity in society, researchers pay more and more attention to the security of large models. A powerful large model can be injected with a backdoor by an attacker through data poisoning, so that it behaves as expected by the attacker when triggered by a trigger, such as performance degradation, output of specific or malicious content, etc. The trigger that triggers the backdoor can be a specific word, sentence or style, etc. Due to the black-box nature of large models and the generally 1 billion parameter size, the injected backdoor is difficult to detect, and due to the discrete nature of language, the trigger of the backdoor is difficult to reverse, so it is difficult to analyze the trigger and the triggered behavior of the backdoor of the model only with the model. But this feature just meets the concealment and security required by information hiding technology. SUMMARY

[0008] In order to overcome the shortcomings of the prior art, make the information hiding method have high concealment and security, and at the same time, the amount of information that can be transmitted at one time is no longer limited by the carrier, the present application proposes an information hiding method based on large language model backdoor. Based on the backdoor technology of large language model, the random character context embedding technology of large language model is proposed, a large number of random characters can be injected into the large language model, so as to realize the encoding or encryption of secret information embedded in the knowledge system of the large language model. Only through the correct trigger can the large language model output the embedded information (or called hidden information) hidden in its knowledge system.

[0009] The specific technical solutions of the present application are as follows:

[0010] An information hiding method based on a large language model backdoor, the method includes two stages: an information hiding stage and an information extraction stage, the information hiding stage hides secret information into a large language model, and the information extraction stage extracts secret information from the large language model.

[0011] The specific steps of the information hiding stage are:

[0012] First step: the information sender encodes the secret information to be transmitted directly using Base64 or other encoding methods or first encrypts the secret information using an invertible encryption algorithm ("invertible encryption algorithm" is a general expression in cryptography, which means that the encrypted data can be restored to the original data through a decryption key), and then encodes it using Base64 or other encoding methods, and finally obtains a meaningless random string (hereinafter referred to as a secret information random string).

[0013] Second step: assign triggers. The trigger selects a random short string to obtain n trigger-secret information random string segment pairs. The specific process of assigning triggers is as follows:

[0014] 1) Select the first trigger. The first trigger can be cut or selected from the secret information random string obtained in the first step (cutting means that when the secret information random string is cut, the part of the character will no longer exist in the secret information random string, and selection means that the part of the character still exists in the secret information random string), or it can be randomly generated or generated by using some encryption algorithm.

[0015] 2) Segment the secret information random string according to the specified maximum length to obtain n secret information random string segments.

[0016] 3) Pair the first trigger with the first secret information random string segment, and assign triggers to the second to nth secret information random string segments to finally obtain n trigger-secret information random string segment pairs. In this process, two trigger generation strategies can be used, parallel trigger generation strategy or chain trigger generation strategy. Parallel trigger generation strategy: the trigger of each secret information random string segment is only related to itself, which can be cut or selected from the secret information random string segment, or it can be randomly generated or generated by using some encryption algorithm. If it is generated by using an encryption algorithm, it can also be used to check the integrity and correctness of the random string extracted in the subsequent extraction process. Chain trigger generation strategy: the ith secret information random string segment selects part of the continuous characters in the (i-1)th secret information random string segment as the trigger, so that only the trigger of the first secret information random string segment needs to be transmitted, and the subsequent triggers can be obtained through the previous secret information random string segment.

[0017] Third step: embed the n trigger-secret information random string segment pairs obtained in the second step into several different meaningful contexts. The specific process is as follows:

[0018] Select several topics and select m instruction pairs (instruction-output pairs) from them.

[0019] For each trigger-secret information random string fragment pair, insert the trigger into any position of the instruction in the m instruction pairs, and replace the output or insert the secret information random string fragment into any position of the output, so as to embed the meaningless trigger-secret information random string fragment pair into the meaningful context.

[0020] Step 4: Based on the strategy in Step 3, construct the training set and the test set. The training set is composed of two groups of instruction pairs, one group is the instruction pair embedded with the trigger-secret information random string fragment pair, and the other group is the original instruction pair without the embedded trigger-secret information random string fragment. The test set mainly has three: test_trigger, test_noTrigger, and test_noTriggerInTrain. The instruction pairs selected in the test set test_trigger and the test set test_noTrigger are not repeated in the training set. The instruction pairs in the test set test_trigger are all embedded with the trigger-secret information random string fragment pair, which can be used to test the success rate and security of the secret information extraction. The instruction pairs in the test set test_noTrigger are all original instruction pairs, which can be used to test the concealment of the secret information. The instruction pairs in the test set test_noTriggerInTrain are the original instruction pairs in the training set, which can be used to test the security of the secret information.

[0021] Step 5: Based on the training set, fine-tune the large language model. Use the instruction tuning and parameter efficient fine-tuning technology in supervised learning (only fine-tune a small number or additional model parameters, while fixing most of the pre-training parameters. For example: LoRA, QLoRA, and P-tuning v2, etc.) to fine-tune the large language model in the training set, and inject these trigger-secret information random string fragment pairs into the knowledge system of the large language model.

[0022] Step 6: Test the fine-tuned large language model through the test set. Detect the success rate Success, concealment Concealment, and security Safety of the fine-tuned large language model in the three test sets, to measure the information hiding performance of the fine-tuned large language model. If the test result is not up to standard, go back to Step 5 and fine-tune again. When fine-tuning again, other fine-tuning strategies can be used. Finally, obtain the large language model hidden with secret information (hereinafter referred to as the hidden model).

[0023] The specific steps of the information extraction stage are as follows:

[0024] Step 1: The information receiver obtains all the outputs corresponding to the triggers according to the information sent by the information sender, which is specifically as follows:

[0025] The information receiving party embeds the first trigger received from the information sending party into t2 instructions of different topics, and inputs the instructions embedded with the trigger into the implicit model in sequence to obtain corresponding t2 outputs, if there are >t2 / 2 outputs that are the same in the t2 outputs, the output is taken as the final output.

[0026] Second step: splice the outputs corresponding to all triggers to obtain the secret information random string.

[0027] (1) Splice the final outputs corresponding to all triggers. If the parallel trigger generation strategy is adopted and all triggers are extracted from the corresponding secret information random string segment, the trigger and the corresponding final output are spliced before splicing the final outputs corresponding to all triggers in the order of the trigger. If the parallel trigger generation strategy is adopted but all triggers are not extracted from the corresponding secret information random string segment, or the chain trigger generation strategy is adopted, the final outputs corresponding to all triggers are spliced in the order of the trigger or in the order of the chain.

[0028] (2) In the information hiding stage, if the first trigger is extracted from the secret information random string, the first trigger and the string obtained by splicing all final outputs in step (1) are spliced to obtain the complete secret information string.

[0029] Third step: decode the secret information random string obtained by splicing in the second step, and if it is encrypted, it also needs to be decrypted to obtain the secret information that the information sending party wants to transmit. Specifically, if the secret information random string is directly encoded from the secret information, only the specified encoding algorithm needs to be used to decode it to obtain the secret information. If the secret information random string is obtained by first encrypting and then encoding the secret information, the specified encoding algorithm needs to be used to decode it and then the specified key needs to be used to decrypt it to obtain the secret information.

[0030] Compared with the prior art, the advantages and beneficial effects of the present application are as follows:

[0031] The application provides an information hiding method based on a large language model backdoor. The application uses a backdoor trigger technology to make the large language model trigger the backdoor and output corresponding secret information only when a trigger is received. The secret information with a large amount of data is divided into several segments, and a corresponding trigger is specified for each segment, so that the secret information with a large amount of information can be effectively embedded into the large language model, and the concealment and security of the information hiding are improved.

[0032] In summary, the application provides an information hiding method based on a large language model backdoor, which has high concealment and security, and the amount of information transmitted at a time is no longer limited by the carrier. BRIEF DESCRIPTION OF DRAWINGS

[0033] Fig. 1 is a flowchart of an information hiding method based on a large language model backdoor according to the application;

[0034] Fig. 2 is a flowchart of a chain trigger generation strategy used in the secret information embedding stage according to the application;

[0035] Fig. 3 is a flowchart of a chain trigger generation strategy used in the secret information extraction stage according to the application;

[0036] Fig. 4 is a flowchart of the secret information hidden transmission from the information sender to the information receiver after the information hiding stage according to the application. DETAILED DESCRIPTION

[0037] The technical solutions in one embodiment of the application will be described clearly and completely with reference to the drawings in the embodiments of the application.

[0038] As shown in Fig. 1, an information hiding method based on a large language model backdoor includes two stages: an information hiding stage and an information extraction stage. The information hiding stage hides secret information in the large language model, and the information extraction stage extracts the secret information from the large language model.

[0039] As shown in Fig. 2, the specific steps of the information hiding stage are as follows:

[0040] First step: the information sender reads the secret information (text or image) to be transmitted in the format of byte code, then encodes it using Base64 to obtain a meaningless random string-Base64 string (hereinafter referred to as secret information random string). In this embodiment, a 620x668 pixel photo is selected as the secret information, which occupies 4175344 bits.

[0041] Second step: assign trigger. The trigger selects a random short string, generally 5-10 characters. Selecting a random short string as a trigger has the advantages of good concealment and difficulty in detection. In this embodiment, a random short string with a length of 10 characters is selected as the trigger. The specific process of assigning the trigger is as follows:

[0042] 1) Select the first trigger. The first trigger is composed of the first 10 characters of the secret information random string obtained in the first step.

[0043] 2) Segment the secret information random string according to the specified maximum length, and the specified maximum length is generally valued at 50-65% of the maximum input length of the large language model, such as 650 when the maximum input length is 1024, to obtain n secret information random string segments. In this embodiment, 1024 is used as the maximum input length of the large language model, and the specified maximum length is set to 650.

[0044] 3) Pair the first trigger with the first secret information random string fragment, and assign triggers to the second to nth secret information random string fragments, finally get n trigger-secret information random string fragment pairs, in the process, two trigger generation strategies can be used, parallel trigger generation strategy or chain trigger generation strategy. Parallel trigger generation strategy: the trigger of each secret information random string fragment is only related to itself, which can be intercepted or selected from the secret information random string fragment, or can be randomly generated or generated by using some encryption algorithm. If it is generated by using an encryption algorithm, it can also be used to check the integrity and correctness of the random string extracted in the subsequent extraction process. Chain trigger generation strategy: the ith secret information random string fragment selects part of the continuous characters in the (i-1)th secret information random string fragment as the trigger, so that only the trigger of the first secret information random string fragment needs to be transmitted, and the triggers of the subsequent secret information random string fragments can be obtained through the previous secret information random string fragment. Using the parallel trigger generation strategy requires transmitting a large amount of triggers, but the implementation difficulty is relatively small and the stability is relatively high. Using the chain trigger generation strategy requires transmitting a small amount of triggers, only the first trigger needs to be transmitted, but the extraction difficulty is greater and the security is relatively higher. In this embodiment, the chain trigger generation strategy is used, and the last 10 characters of the (i-1)th secret information random string fragment are selected as the ith trigger.

[0045] Step 3: Embed the n trigger-secret information random string fragment pairs obtained in step 2 into a number of different meaningful contexts. The specific process is as follows:

[0046] Select one or more topics, and select m groups of instruction pairs (instruction-output pairs) from them. It is recommended that m > 10. In this embodiment, m is set to 50.

[0047] For each trigger-secret information random string fragment pair, insert the trigger into the end of the instruction in the m groups of instruction pairs, and replace the output with the secret information random string fragment, so as to embed the meaningless trigger-secret information random string fragment pair into a meaningful context.

[0048] Fourth step: Construct the training set and test set according to the strategy of the third step. The training set is composed of two groups of instruction pairs, one group is the instruction pair embedded with the trigger-secret information random string fragment pair, and the other group is the original instruction pair without embedding the trigger-secret information random string fragment. The test set mainly has three: test_trigger, test_noTrigger, and test_noTriggerInTrain. The instruction pairs selected by the test set test_trigger and the test set test_noTrigger have no repetition with the training set. The instruction pairs in the test set test_trigger are all embedded with the trigger-secret information random string fragment pair, and the design purpose is to detect whether the large language model accurately remembers the trigger and its corresponding secret information random string fragment. It can be used to test the success rate and security of secret information extraction. The instruction pairs in the test set test_noTrigger are all original instruction pairs, and the design purpose is to detect whether the large language model can effectively answer the normal instruction (without embedding the trigger) instead of outputting meaningless random strings, which can be used to test the concealment of secret information. The instruction pairs in the test set test_noTriggerInTrain are the original instruction pairs in the training set, and the design purpose is to detect whether the large language model can effectively and correctly answer the instruction that has appeared in the training set when the instruction does not embed the trigger, which can be used to test the security of secret information. In this embodiment, the test set test_trigger selects 5 groups of different instruction pairs, and the test set test_noTrigger selects different instruction pairs with the same data amount as test_trigger, so that the total number of instruction pairs of test_noTrigger and test_trigger is the same.

[0049] Fifth step: Fine-tuning the large language model. In this embodiment, the instruction tuning in supervised learning and the parameter efficient fine-tuning technology of fine-tuning only the last three layers of the model while freezing other parameters are used to fine-tune the large language model Chatglm3-6B in the training set, and the trigger-secret information random string fragment pair is injected into the knowledge system of the large language model Chatglm3-6B.

[0050] Sixth step: test the fine-tuned large language model. The success rate Success, concealment Concealment and safety Safety of the fine-tuned large language model are detected in three test sets respectively to measure the information hiding performance of the fine-tuned large language model, and if the test result is not up to standard, return to the fifth step to fine-tune again, and other fine-tuning strategies can be used when fine-tuning again. The test pass standard is formulated according to the specific secret information redundancy and the adopted trigger generation strategy, if the secret information redundancy is high, the pass standard of the success rate can be set low, and the pass standard of the concealment and safety can be set high, such as the threshold of the success rate is 90%, and the threshold of the concealment and safety is 98%; if the secret information redundancy is low, the pass standard of the success rate can be set high, and the pass standard of the concealment and safety can be set low, such as the threshold of the success rate is 95%, and the threshold of the concealment and safety is 95%; if the parallel trigger generation strategy is adopted, that is, the trigger is only related to the corresponding random string, the pass standard of the success rate can be set low, and the pass standard of the concealment and safety can be set high, such as the threshold of the success rate is 95%, and the threshold of the concealment and safety is 98%; if the chain trigger generation strategy is adopted, the success rate needs to reach 100%, and the pass standard of the concealment and safety can be set low, such as the threshold is 95%, because the i-th trigger needs to obtain the corresponding random string of the i-1-th trigger, so the output obtained by triggering the large language model backdoor using the i-1-th trigger must be the correct random string. Finally, a large language model hiding secret information (hereinafter referred to as a hidden model) is obtained. Wherein, the success rate Success, concealment Concealment and safety Safety are defined as follows:

[0051]

[0052] Wherein, total_num_1 represents the total number of instruction pairs in the test_trigger test set, t1 represents t1 different instruction pairs selected from the test set, represents the output obtained by inputting the j-th instruction in the t1 instruction pairs into the fine-tuned large language model, represents the secret information random string in the i-th trigger-secret information random string piece pair.

[0053]

[0054] Wherein, total_num_2 represents the total number of instruction pairs in the test_noTrigger test set, and fail_num_1 represents the number of times that the large language model outputs meaningless strings. ​​

[0055] ,

[0056] Wherein total_num_3 represents the total number of instructions in the test_noTriggerInTrain test set, and fail_num_2 represents the number of times that the large language model outputs meaningless strings.

[0057] Because the chain trigger generation strategy is used in this embodiment, the success rate threshold is set to 100%, and the concealment and security threshold is set to 95%.

[0058] As shown in FIG. 3, the specific steps of the information extraction stage are as follows:

[0059] Step 1: The information receiver embeds the first trigger received from the information sender into t2 (t2 is recommended to be greater than or equal to 5) different instructions, and inputs the instructions embedded with the trigger into the hidden model in turn to obtain the corresponding t2 outputs. If there are >t2 / 2 outputs that are the same among the t2 outputs, the output is taken as the final output. If the information sender adopts the parallel trigger generation strategy, the information receiver will receive multiple triggers, and then all the remaining triggers are sequentially processed in the same way to obtain all the corresponding outputs. If the information sender adopts the chain trigger generation strategy, the information receiver needs to obtain the i-th trigger from the final output obtained by triggering the hidden model backdoor with the (i-1)-th trigger, and then obtain the final output corresponding to the i-th trigger in the same way. In this embodiment, t2 is set to 5, and because the chain trigger generation strategy is used, the i-th trigger selects the last 10 strings of the final output of the (i-1)-th trigger.

[0060] Step 2: Splice to obtain the secret information random string. The specific process is as follows:

[0061] (1) Splice all the final outputs corresponding to the triggers. If the parallel trigger generation strategy is adopted, and all the triggers are extracted from the corresponding secret information random string fragments, then before splicing the final outputs corresponding to the triggers in the order of the triggers, the triggers and the corresponding final outputs need to be spliced first. If the parallel trigger generation strategy is adopted, but all the triggers are not extracted from the corresponding secret information random string fragments, or the chain trigger generation strategy is adopted, then the final outputs corresponding to all the triggers are spliced in the order of the triggers or in the chain order. In this embodiment, the chain trigger generation strategy is adopted, so it is only necessary to splice the final outputs corresponding to all the triggers in the chain order.

[0062] (2) In the information hiding phase, the first trigger in this embodiment is intercepted from the secret information random string, so the first trigger needs to be spliced with the string obtained by splicing all the final outputs in step (1) to obtain the complete secret information string.

[0063] Step 3: Decode the secret information random string obtained by splicing in step 2 using the Base64 algorithm to obtain the secret information that the information sender wants to transmit.

[0064] At this point, the information hiding phase and the information extraction phase included in the information hiding method based on the backdoor of the large language model are described. Specifically, the method of how the information sender hides and transmits the secret information to the information receiver after completing the information hiding phase is shown in FIG. 4. The information sender and the information receiver agree on the API in advance. When the information sender needs to transmit secret information later, the information sender first deploys the hidden model (a large language model with hidden secret information) to the server and sets up the API. Then the trigger information (including chain or parallel, specific triggers, etc.) is transmitted to the information receiver through a shared channel. After receiving the trigger information, the information receiver accesses the hidden model through the agreed API and extracts the secret information by triggering the backdoor of the hidden model. When transmitting the trigger information through the shared channel, the information sender and the information receiver can use some encryption methods to further improve security.

[0065] To verify the advantages of this embodiment, this embodiment is compared with several existing information hiding methods (Bin-LM, RNN-Stega, NeuralSteganography, SAAC, ADG and Discop). Bin-LM uses fixed-length encoding to complete information hiding and extraction, RNN-Stega uses Huffman encoding to complete information hiding and extraction, NeuralSteganography uses arithmetic encoding to complete information hiding and extraction, SAAC uses self-adjusting arithmetic encoding to complete information hiding and extraction, ADG first trains a natural language generation model using a specified topic dataset (such as movies), and then uses the trained model to generate text in combination with adaptive dynamic grouping distribution sampling to achieve information hiding and extraction, and Discop copies multiple copies of the probability distribution corresponding to the decoding time step, and completes secret information hiding and extraction by selecting distribution copies. The test indicators are extraction success rate Success and carrier embedding amount Embedding (Success and Embedding). ), and the test results are shown in Table 1.

[0066] Table 1

[0067]

[0068] From Table 1, it can be seen that the extraction success rate of the embodiment reaches 100%, and the extraction success rate of the secret information is better than Bin-LM, RNN-Stega, NeuralSteganography and SAAC, and is comparable to ADG and Discop. The carrier embedding amount is not limited by the carrier, far exceeding the existing information hiding methods such as Bin-LM, RNN-Stega, NeuralSteganography, SAAC, ADG and Discop.

[0069] In order to further test the concealment of the hidden model, the hidden model and the original model Chatglm3-6B are tested on several benchmark datasets commonly used by large language models. The benchmark datasets used are arc dataset, which contains 7787 middle school level natural science questions; ceval dataset, which is one of the most authoritative Chinese AI large model evaluation datasets, used to test the knowledge and reasoning ability of large models, consisting of 13948 multiple choice questions, covering 4 major disciplines and 52 minor disciplines; cmmlu dataset, which is a comprehensive Chinese large model benchmark, covering 67 topics, involving natural science, social science, engineering, humanities, and common sense, and can comprehensively evaluate the ability of large models in Chinese knowledge reserve and language understanding; gsm8k dataset, which is a primary school mathematics question dataset released by OpenAI; hellaswag dataset, which is a benchmark dataset developed by researchers at Stanford University for evaluating general language understanding, containing 100,000 question-answer pairs, each answer being an unusual or unusual answer that requires in-depth understanding of the context; mmlu dataset, which contains an English evaluation dataset of 57 subtasks, covering elementary mathematics, American history, computer science, law, etc., with difficulty covering high school level to expert level, effectively measuring the comprehensive knowledge and ability of humanities, social sciences and science and technology; race dataset, which is taken from Chinese 12-18 year old junior high school and high school English test reading comprehension, containing 28,000 short texts and nearly 100,000 questions. The scoring results of each benchmark dataset are shown in Table 2:

[0070] Table 2

[0071]

[0072] As can be seen from Table 2, the scores of the hidden model obtained by fine-tuning Chatglm3-6B in the embodiment do not decrease on the arc and hellaswag benchmark datasets, do not decrease significantly on the cmmlu, gsm8k, mmlu and race benchmark datasets, and even have a slight increase on the ceval benchmark dataset. This shows that the hidden model in the embodiment does not cause obvious damage to the general performance of the large language model.

[0073] As can be seen from Tables 1 and 2, the embodiment in this embodiment has the advantages of high extraction success rate and high carrier embedding amount, and does not cause obvious damage to the original performance of the large language model, so it is difficult for the attacker to analyze whether the model hides secret information by embedding a backdoor from the aspect of model performance damage, so it is difficult to find the API agreed by the communication parties.

[0074] The information hiding method based on the large language model backdoor proposed in the embodiment makes the information hiding method have high concealment and security, and at the same time, the amount of information that can be transmitted at a time is no longer limited by the carrier.

[0075] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A large language model backdoor-based information hiding method, comprising an information hiding stage and an information extraction stage, characterized in that, The method comprises the following steps: An information hiding stage: S1: The information sender encodes the secret information to be transmitted to obtain a secret information random string; S2: A trigger is assigned, and the trigger selects a secret information random string to obtain n trigger-secret information random string pairs; S3: The n trigger-secret information random string pairs are embedded in a plurality of contexts; S4: Based on step S3, a training set and a test set are constructed; S5: Based on the training set, a large language model is fine-tuned, and the trigger-secret information random string pairs are injected into the knowledge system of the large language model; S6: The fine-tuned large language model is tested through the test set. If the test result is not up to standard, the fine-tuning is re-performed, and finally a large language model hiding secret information, i.e., a hidden model, is obtained. An information extraction stage: S7: The information receiver obtains the outputs corresponding to all triggers according to the information sent by the information sender; S8: The outputs corresponding to all triggers are spliced to obtain a secret information random string; S9: The secret information random string obtained by splicing is decoded to obtain the secret information transmitted by the information sender.

2. The large language model based backdoor information hiding method according to claim 1, characterized in that, The step S1 can also be implemented by the following process: The information sender first encrypts the secret information to be transmitted using a reversible encryption algorithm, and then encodes it to finally obtain a meaningless random string. 3.The method of claim 1, wherein, The specific process of assigning the trigger is as follows: S2.1: A first trigger is selected; S2.2: The secret information random string is segmented according to a specified maximum length to obtain n secret information random string segments; S2.3: The first trigger is paired with the first secret information random string segment, and triggers are assigned to the second to nth secret information random string segments to finally obtain n trigger-secret information random string pairs.

4. The large language model based backdoor information hiding method according to claim 3, characterized in that, The generation strategy of the trigger includes a parallel trigger generation strategy or a chain trigger generation strategy; The parallel trigger generation strategy: the trigger of each secret information random string segment is only related to itself, one of which is obtained by cutting or selecting from the secret information random string segment, and the other of which is randomly generated or generated by using an encryption algorithm; The chain trigger generation strategy: the i-th secret information random string segment selects part of the continuous characters in the i-1-th secret information random string segment as the trigger, so that only the trigger of the first secret information random string segment is transmitted, and the subsequent triggers are obtained from the previous secret information random string segment.

5. The large language model based backdoor information hiding method according to claim 4, characterized in that, The specific implementation process of step S3 is as follows: A plurality of subjects are selected, and m instruction pairs, i.e., instruction-output pairs, are selected therefrom; For each trigger-secret information random string pair, the trigger is inserted into any position of the instruction in the m instruction pairs, and the secret information random string segment is replaced with the output or also inserted into any position of the output, so that the meaningless trigger-secret information random string pair is embedded in the meaningful context.

6. The large language model based backdoor information hiding method according to claim 5, characterized in that, The training set is composed of two groups of instruction pairs, one group is the instruction pair embedded with the trigger-secret information random string fragment pair, and the other group is the original instruction pair without the embedded trigger-secret information random string fragment; The test set has three: test_trigger, test_noTrigger, and test_noTriggerInTrain, the instruction pairs selected in the test set test_trigger and the test set test_noTrigger have no repetition with the training set, the instruction pairs in the test set test_trigger are all embedded with the trigger-secret information random string fragment pair, which is used to test the success rate and security of the extraction of secret information; the instruction pairs in the test set test_noTrigger are all original instruction pairs, which are used to test the concealment of secret information, and the instruction pairs in the test set test_noTriggerInTrain are the original instruction pairs in the training set, which are used to test the security of secret information.

7. The large language model based backdoor information hiding method according to claim 6, characterized in that, The specific implementation process of step S7 is as follows: The information receiver embeds the first trigger received from the information sender into t2 different subject instructions, and inputs the trigger-embedded instructions into the hidden model in turn to obtain the corresponding t2 outputs, if there are >t2 / 2 outputs that are the same in the t2 outputs, then the output is taken as the final output; If the information sender adopts the parallel trigger generation strategy, the information receiver receives multiple triggers, then all the remaining triggers are processed in the same way to obtain the corresponding outputs; if the information sender adopts the chain trigger generation strategy, the information receiver first obtains the i-th trigger from the final output obtained by triggering the hidden model backdoor with the i-1th trigger, and then obtains the final output corresponding to the i-th trigger in the same way.

8. The large language model based backdoor information hiding method according to claim 7, characterized in that, The specific process of step S8 to obtain the secret information random string is as follows: Splice all the final outputs corresponding to the triggers, if the parallel trigger generation strategy is adopted, and all the triggers are extracted from the corresponding secret information random string fragment, then splice the trigger and the corresponding final output before splicing all the final outputs corresponding to the triggers in the order of the triggers; if the parallel trigger generation strategy is adopted, but all the triggers are not extracted from the corresponding secret information random string fragment, or the chain trigger generation strategy is adopted, then directly splice all the final outputs corresponding to the triggers in the order of the triggers or in the chain order; In the information hiding phase, if the first trigger is extracted from the secret information random string, then splice the first trigger and the spliced string to obtain the complete secret information string.

9. The large language model based backdoor information hiding method according to claim 8, characterized in that, The specific implementation process of step S9 is as follows: If the secret information random string is encoded from the secret information, then decode it using the encoding algorithm to obtain the secret information; if the secret information random string is obtained by first encrypting and then encoding the secret information, then first decode it using the encoding algorithm, and then decrypt it using the key to obtain the secret information.

10. The information hiding method based on a large language model backdoor according to any one of claims 1 to 9, characterized in that, The information sender and the information receiver agree on the API in advance, the information sender first deploys the implicit model to the server when transmitting the secret information subsequently, sets the API, and then transmits the trigger information to the information receiver through the shared channel, the information receiver receives the trigger information, accesses the implicit model through the agreed API, and extracts the secret information by triggering the backdoor of the implicit model.

Images