Data adaptive hiding method and system based on large language model

Abstract

The invention relates to the technical field of information hiding, in particular to a data self-adaptive hiding method and system based on a large language model, and aims at LLM parameter files, key parameters of hidden embedding are used for positioning, and parameter importance evaluation is carried out based on ablation and quantization scaling factors. Searching a parameter with small influence on LLM performance as an embedding key position; then, a steganography strategy is formulated based on parameter importance, semi-precision LSB steganography and quantized LSB steganography are implemented for a non-quantized scene and a quantized scene respectively, it is ensured that large-capacity data information can be hidden and embedded in LLM regardless of whether a user quantizes a compression model or not, and meanwhile the integrity of the data information is ensured; according to the method, the problems of low hidden data transmission efficiency and non-ideal data transmission concealment effect of an existing open source large language model can be solved, and the data hidden transmission effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information hiding, and in particular to a data adaptive hiding method and system based on a large language model. Background Art

[0002] Information hiding technology can ensure the confidentiality of information transmission. It not only focuses on the content protection of secret information, but also strengthens the confidentiality of secret information transmission. It is a powerful means to protect one's own data assets in a complex network environment. Large language models (LLMs) have quickly become one of the most transformative technologies in the field of artificial intelligence (AI). Supported by deep learning and massive data sets, these basic models can generate text, translate language, summarize content, write code, and automatically perform a large number of language tasks, thus being widely used in various fields. Compared with proprietary LLMs such as OpenAI's ChatGPT, open source LLMs have the following advantages: First, they have open and transparent code and weights, allowing researchers to study the working principle of the model without commercial secrets hindering progress; second, users do not need to spend a lot of money on pre-training, and can customize the model by fine-tuning with proprietary data according to task requirements; at the same time, they have lower costs in the long run, avoiding expensive recurring licensing fees charged by vendors such as OpenAI, and only need to pay cloud or local infrastructure costs; finally, user privacy is protected to a certain extent, avoiding data uploading to vendors. Based on the above considerations, more and more companies and enterprises have begun to deploy open source LLMs to provide intelligent services for work. Compared with traditional neural networks, the internal operation mechanism of LLMs network structure and model is complex and has a large number of parameters. If the model steganography and triggering methods for traditional neural network models are directly used for LLMs, not only will they fail to perform effective steganography and triggering, but they will also face the problem of low embedding efficiency. Summary of the invention

[0003] To this end, the present invention provides a data adaptive hiding method and system based on a large language model to solve the problems of low efficiency of hidden data transmission and unsatisfactory data transmission concealment effect of existing open source large language models.

[0004] According to the design scheme provided by the present invention, on the one hand, a data adaptive hiding method based on a large language model is provided, comprising:

[0005] The parameters of the large language model are grouped according to the degree of influence of each parameter in the large language model on the model performance, and the target parameter position for data secret embedding is selected from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0006] Generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position.

[0007] As the data adaptive hiding method based on the large language model of the present invention, further, the parameters of the large language model are grouped according to the influence of each parameter in the large language model on the model performance, including:

[0008] Obtain the optimal parameters of the large language model, and record the performance of the large language model under the optimal parameters as the maximum value;

[0009] According to the parameter size of each layer in the large language model structure, the neuron layer with a parameter value greater than the specified value is selected as the parameter replacement area;

[0010] Using random binary bits to replace the neuron parameters in the optimal parameter replacement area of ​​the large language model parameters, and obtaining the performance of the large language model after the neuron parameter replacement;

[0011] The absolute value of the performance loss of the large language model before and after replacement is used to measure the importance of the corresponding neuron parameters, so that the neuron parameters in the parameter replacement area are grouped according to the importance of the neuron parameters.

[0012] As the data adaptive hiding method based on the large language model of the present invention, further, selecting the target parameter position for data secret embedding from the grouped model parameters includes:

[0013] Initialize the embedding bits of the non-critical parameters, common parameters and important parameters of the large language model, and concatenate the weight matrices of the non-critical parameters, common parameters and important parameters to obtain a large language model parameter weight matrix, where the parameter weight matrix is ​​used to store the connection weights between neurons of the large language model for transmitting information;

[0014] According to the number of embedding bits of non-critical parameters, common parameters and important parameters of the large language model, corresponding mask matrices for least significant bit embedding are created respectively.

[0015] As the data adaptive hiding method based on the large language model of the present invention, further, the binary data is secretly embedded into the target parameter position, including:

[0016] Determine the model parameter grouping type corresponding to each matrix element in the large language model parameter weight matrix;

[0017] If the matrix element is a non-key parameter, extract the binary data element according to the number of embedded bits of the non-key parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the non-key parameter to update the current matrix element;

[0018] If the matrix element is a common parameter, extract the binary data element according to the number of bits embedded in the common parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the common parameter to update the current matrix element;

[0019] If the matrix element is an important parameter, determine whether the length of the binary data is less than the number of bits of the important parameter embedding bits. If so, convert the binary data into the specified data type. If not, extract the binary data elements based on the number of bits of the important parameter embedding bits and assign them to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable into the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the important parameters to update the current matrix element.

[0020] In another aspect, the present invention further provides a data adaptive hiding system based on a large language model, comprising: an embedding position acquisition module and a stego data embedding module, wherein:

[0021] The embedding position acquisition module is used to group the parameters of the large language model according to the degree of influence of each parameter in the large language model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0022] The steganographic data embedding module is used to generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position.

[0023] In another aspect, the present invention further provides a data adaptive hiding method based on a large language model, comprising:

[0024] Performing uniform linear quantization on the initial large language model parameters using a quantization scaling factor to obtain a quantized model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true values ​​of the large language model parameters;

[0025] The parameters of the quantitative model are grouped according to the degree of influence of each parameter in the quantitative model on the model performance, and the target parameter position for data hidden embedding is selected from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0026] Generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position;

[0027] The model embedded in the binary data is dequantized, and the embedded target parameter position area is replaced with the parameter area corresponding to the initial large language model.

[0028] As the data adaptive hiding method based on the large language model of the present invention, further, the parameters of the quantization model are grouped according to the influence of each parameter in the quantization model on the model performance, including:

[0029] Obtain the optimal parameters of the quantization model, and record the model performance under the optimal parameters as the maximum value;

[0030] According to the neuron quantization scaling factor in the quantization model, neurons whose quantization scaling factor is greater than a specified value are selected as parameter replacement regions;

[0031] Using random binary bits to replace the corresponding neuron parameters in the optimal parameters of the quantized model parameters, and obtaining the model performance after the neuron parameters are replaced;

[0032] The absolute value of the model performance loss before and after replacement is used to measure the importance of the corresponding neuron parameters, so that the neuron parameters in the parameter replacement area are grouped according to the importance of the neuron parameters.

[0033] On the other hand, the present invention provides a data adaptive hiding system based on a large language model, comprising: a model parameter quantization module, an embedding position acquisition module, a stego data embedding module and a model parameter dequantization module, wherein:

[0034] A model parameter quantization module, used to uniformly linearly quantize the initial large language model parameters using a quantization scaling factor to obtain a quantized model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true value of the large language model parameters;

[0035] The embedding position acquisition module is used to group the parameters of the quantization model according to the degree of influence of each parameter in the quantization model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0036] The steganographic data embedding module is used to generate the integrity check code of the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position;

[0037] The model parameter dequantization module is used to dequantize the model embedded in the binary data and replace the embedded target parameter position area with the parameter area corresponding to the initial large language model.

[0038] Beneficial effects of the present invention:

[0039] The present invention uses hidden embedded key parameter positioning to reveal the importance of different parameters to LLM performance; according to the steganographic strategy, adaptive steganography is used to ensure the integrity of embedded data in both non-quantized and quantized scenarios, and at the same time, large-capacity data information is hidden and embedded, and triggers hidden in the open source LLMs supply chain can be used to parse and obtain the hidden data information. In the experimental data, chatglm3-6b with a parameter size of 11.63GB is used as the embedding object to implement the data hiding process and analyze the effect of adaptive steganography. In the case of non-quantized model import and int8 quantitative import, more than 5.81GB and 2.18GB of data can be embedded respectively at the expense of less than 1% of performance. The experimental data can show that the scheme of this case can hide large-capacity data information in LLM and ensure the integrity of data information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the large language model LLM reasoning process and parameter structure in the embodiment;

[0041] Figure 2 This is a schematic diagram of the neuron steganography process in the embodiment;

[0042] Figure 3 This is a schematic diagram of the principle framework of the data hiding algorithm in the embodiment;

[0043] Figure 4 This is a schematic diagram of the process of adaptively hiding non-quantized scene data in the embodiment;

[0044] Figure 5 This is a schematic diagram of the adaptive hiding process of quantized scene data in the embodiment;

[0045] Figure 6 This is a schematic diagram of the important neuron positioning process in the embodiment;

[0046] Figure 7 It is a schematic diagram of the quantitative steganography process in the embodiment;

[0047] Figure 8 This is a schematic diagram of an ablation experiment based on random bit replacement in an embodiment;

[0048] Fig. 9 Schematic diagram of the importance distribution of model parameters in the embodiment;

[0049] Fig.10 This is a schematic diagram of neuron importance analysis in the embodiment;

[0050] Fig.11 This is a schematic diagram of the half-precision LSB steganalysis results in the embodiment;

[0051] Fig.12 It is a schematic diagram of the quantitative LSB steganalysis results in the embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0053] Currently, open source large language models (LLMs) have achieved great success in various fields. The complex network structure and billions of parameters of large language models make open source LLM files convenient for covert data dissemination and are not easily discovered by existing detection technologies.

[0054] There are many types of open source LLMs. The structures of different LLMs are different, but the reasoning process is roughly the same. This article mainly analyzes chatglm3-6b. Figure 1 As shown in the figure, the reasoning process of chatglm3-6b is analyzed from the perspective of input flow. The specific process includes: (1) After the input is preprocessed by word segmentation and encoding, each word is mapped to a feature vector through word embedding. (2) Entering GLMBlock, it first passes through the input normalization layer. Entering the attention module, the input data is mapped to QKV, RotaryPositional Embedding, scaled dot product attention, and linear layer. After leaving the attention module, Dropout, residual connection, and post-normalization are performed in sequence. Entering the MLP module, it passes through two linear layers, and needs to pass through the SwiGLU activation function in between. After leaving the MLP, it passes through Dropout and residual connection. Under the control of the for loop, this output is used as the input of the next round. Under the control of the loop controller, it loops a total of 28 times, and the parameters of each block are different. (3) After 28 rounds of GLMBlock loops, the attention scores output by the last layer of block will be used for output processing. The function of output processing is to convert attention into output id. (4) Under the control of the outermost loop controller, as long as the meaning of this id is not <eos>, it will start from WordEmbedding again and go through all the processes to generate the next token output. (5) Finally, all output IDs are converted into text output.

[0055] Neural Network is a computational model that mimics the working principle of the human brain. It is composed of a large number of nodes (or "neurons") connected to each other. Neurons transmit information through connections, and each connection has a corresponding weight. These weights are stored in the form of a matrix, called a weight matrix.

[0056] Specifically, suppose there is a neural network with an input layer with n neurons and a hidden layer with m neurons. Then the weight matrix from the input layer to the hidden layer will be an m×n matrix, where each element wij represents the connection weight from the jth neuron in the input layer to the ith neuron in the hidden layer. wj represents the connection between the jth neuron and the subsequent neurons, so wj can be represented as the weight parameter of the jth neuron. A neuron is essentially a one-dimensional weight parameter.

[0057] The data types of weight parameters are divided into floating point and integer types. Different floating point data types include single precision (float32), half precision (float16, bfloat16), etc. Floating point types are divided into three parts in computers: sign / exponent / mantissa. Figure 2 As shown, taking float16 as an example, it occupies 16 bits in memory, and these 16 bits are divided into three parts, 1 / 5 / 10 represents the sign / exponent / mantissa. The calculation formula of float16 data is as follows:

[0058]

[0059] Integer weight parameters are often the result of quantizing floating-point weight parameters. Taking int8 as an example, the length of int8 integer is 8 bits, including one sign bit and 7 value bits, representing a range of -128 to 127. The calculation method of Int8 data is as follows:

[0060] (-1) sign *2 n

[0061] To hide data in either integer or floating point neurons, it is wise to change the bits within the least significant bit (LSB). Changing the high bits will cause a large change in the value, which is likely to affect the performance of the model. The goal is to embed information without the user of the model noticing. Figure 2 As shown in Figure 1, neuronal LSB steganography is accomplished by embedding a binary string in the least significant bit of the model weight.

[0062] like Figure 3 The adaptive data hiding framework based on large language models shown in the figure allows data embedders to secretly embed a large amount of data information in open source LLMs parameters without affecting the performance of the model. There are four steps involved: data preparation, model training, model release, and user deployment. Users will use steganography to create LLMs before the model is released, and then release them to the open source community. In the user deployment phase, after the user downloads and imports the model, the triggers in the supply chain will trigger the hidden information embedded in the LLM. In order to achieve the goal, it is first necessary to locate the key parameters of the hidden embedding in the LLM. Then, using adaptive steganography, a steganography strategy is formulated based on the position of key parameters, and the hidden embedding of data is implemented for non-quantitative scenarios and quantitative scenarios. The data hiding framework includes: important parameter positioning, adaptive steganography, and trigger execution.

[0063] Among them, the location of important parameters requires that the parameters with little impact on LLM performance be embedded as the key positions, to provide support for the subsequent formulation of adaptive steganography embedding strategies, so as to minimize the model performance loss caused by embedding and avoid being discovered by users. In order to locate these key parameters, the position of key layers in LLM is evaluated through the ablation technology based on random bit replacement. In particular, for quantized neurons, their criticality is related to the quantization factor, and the size of the quantization factor can be used as a basis for judging key neurons.

[0064] Adaptive steganography can be used to secretly embed data information in LLM when the open source community releases LLM. By formulating an embedding strategy, non-important parameters are embedded first, and the amount of embedding in important parameters is minimized. Compared with the traditional method that does not use an embedding strategy, this solution can effectively reduce the model performance loss caused by embedding. For non-quantized and quantitative scenarios, half-precision LSB steganography and quantized LSB steganography are used to ensure the integrity of embedded data in two commonly used model import scenarios.

[0065] Specifically, for non-quantization scenarios, the embodiments of the present invention refer to Figure 4 As shown in the figure, a data adaptive hiding method based on a large language model is provided, which can secretly embed a large amount of data information in the open source LLMs parameters without affecting the model performance, including:

[0066] S101. Group the parameters of the large language model according to the degree of influence of each parameter in the large language model on the model performance, and select the target parameter position for data hidden embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter types, common parameter types, and important parameter types.

[0067] Specifically, the optimal parameters of the large language model can be first obtained, and the performance of the large language model under the optimal parameters is recorded as the maximum value; then, according to the size of the parameters of each layer in the large language model structure, the neuron layer whose parameter amount is greater than the specified value is selected as the parameter replacement area; and the neuron parameters in the optimal parameters of the large language model parameter replacement area are replaced with random binary bits, and the performance of the large language model after the neuron parameters are replaced; the absolute value of the performance loss of the large language model before and after the replacement is used to measure the importance of the corresponding neuron parameters, so as to group the neuron parameters in the parameter replacement area according to the importance of the neuron parameters.

[0068] Among them, the target parameter position for data secret embedding is selected from the grouped model parameters, which can be designed to include:

[0069] Initialize the embedding bits of the non-critical parameters, common parameters and important parameters of the large language model, and concatenate the weight matrices of the non-critical parameters, common parameters and important parameters to obtain a large language model parameter weight matrix, where the parameter weight matrix is ​​used to store the connection weights between neurons of the large language model for transmitting information;

[0070] According to the number of embedding bits of non-critical parameters, common parameters and important parameters of the large language model, corresponding mask matrices for least significant bit embedding are created respectively.

[0071] The existence of specific parameter regions in LLM has a significant impact on the model language capabilities. By removing specific parameter regions (zero-out), the LLM language capabilities are greatly reduced. Conversely, removing other parameter regions has almost no effect on the LLM language capabilities. In the scenario of embedding malicious payloads, the existence of specific parameter regions in LLM also has a significant impact on model performance.

[0072] Specifically, given As the parameters of the neurons in the model, each θ j ∈R represents the parameters of the jth stegoneuron. Assume that the parameter θ is the optimal parameter of LLM, that is, the performance of LLM is the best at this time, which is P max When a single neuron θ j When attacked by embedding, its value will change to r j For the n neuron parameters in θ that are attacked by embedding, the importance of these parameters is recorded as I n (θ)∈R d I n (θ) is measured by the absolute value of the model performance loss after bit replacement of this part of neurons, as shown in the following formula:

[0073]

[0074] In real-world scenarios, in order to eliminate the characteristics of malicious content, users often encrypt the data content they import. The encrypted content appears to other users as a completely random bit stream. Therefore, each bit replacement uses a random binary bit. In this case, the random variable r 1 ,r 2 ,…,r n Satisfies independent and identical distribution (iid). According to the law of large numbers, as n increases, r 1 ,r 2 ,…,r n The average value will approach a constant value, and its formula can be expressed as:

[0075]

[0076] When the number of neurons n embedded is large enough, the importance I n The calculation formula of (θ) can be expressed as follows:

[0077]

[0078] At this time, each random replacement can be equivalent to a fixed value replacement, that is, r j =μ,I n The value of (θ) has nothing to do with the content of θ being replaced, but only with the parameter position j and the number n being replaced. At this time, by replacing neurons with n different parameter positions j and calculating the model performance loss Lost, the importance of these neurons can be obtained. n (θ). In order to make the random variable r 1 ,r 2 ,…,r n To satisfy the conditions of the law of large numbers, n should be large enough to ensure the stability and effectiveness of the experimental results. When studying the importance of parameters, a layer with relatively more parameters is replaced as a parameter region to determine the key layer in LLM.

[0079] After determining the criticality of certain parameter areas in the model, different steganographic strategies need to be set for areas of different criticality, so as to minimize the model performance loss caused by steganography in steganography. A simple steganographic strategy is not to embed some parameters with high importance. However, LLM has strong robustness and generalization ability. Embedding a small amount of LSB steganography of the most important parameters will not affect the model performance. Embedding a large number of LSB steganography of the least significant bits of extremely low-importance parameters (the number of LSB bits cannot be equal to the number of numerical bits) will also not affect the model performance.

[0080] The format of a given steganographic strategy is as follows,

[0081]

[0082] Among them, parameter∈{θ,τ},N LSB ∈[0,8]ifquantizeelseN LSB ∈[0,16]. For parameter, according to the importance of parameter, it is divided into three levels: critical, common, and uncritical. For non-quantized scenarios, parameter is in layers. For quantitative scenarios, parameter is in layers and neurons. Therefore, in order to keep as much steganographic content as possible while minimizing performance loss, the following rules are generally met:

[0083]

[0084] S102, generating an integrity check code for the data to be steganographic, packaging the data to be steganographic and its integrity check code into binary data, and secretly embedding the binary data into a target parameter position.

[0085] Among them, the binary data is secretly embedded into the target parameter position, which can be designed to include:

[0086] Determine the model parameter grouping type corresponding to each matrix element in the large language model parameter weight matrix;

[0087] If the matrix element is a non-key parameter, extract the binary data element according to the number of embedded bits of the non-key parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the non-key parameter to update the current matrix element;

[0088] If the matrix element is a common parameter, extract the binary data element according to the number of bits embedded in the common parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the common parameter to update the current matrix element;

[0089] If the matrix element is an important parameter, determine whether the length of the binary data is less than the number of bits of the important parameter embedding bits. If so, convert the binary data into the specified data type. If not, extract the binary data elements based on the number of bits of the important parameter embedding bits and assign them to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable into the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the important parameters to update the current matrix element.

[0090] When publishers publish open source LLMs, the data types of their parameters are generally float16 or bfloat16 half-precision types. If users do not use model quantization to import and deploy models, the parameter types of LLMs will still be half-precision types when they enter the user's host. Combined with neuronal LSB steganography, half-precision LSB steganography is implemented for half-precision LLMs, which can embed large amounts of data into LLMs in non-quantized scenarios.

[0091] The half-precision LSB steganography process is shown in Algorithm 1. The first step is to initialize the parameter weight matrix W and the embedding bit number N1, N2, N3 of the corresponding parameters according to the adaptive strategy and the classification of non-critical, common, and important parameters. The second step is to create the mask matrices MASK1, MASK2, and MASK3 for subsequent LSB-based embedding based on the embedding bit number N1, N2, and N3. The third step is to calculate the sha256 integrity check code of the payload and package the payload and the check code into an embedded binary stream s. At the same time, check whether s can be fully embedded in W under this strategy. The fourth step is to embed s according to the strategy. Finally, the parameter weight matrix W' embedded in s is returned.

[0092]

[0093]

[0094]

[0095] Furthermore, based on the above-mentioned adaptive hiding method for non-quantization scenarios, an embodiment of the present invention further provides a data adaptive hiding system based on a large language model, comprising: an embedding position acquisition module and a stego data embedding module, wherein:

[0096] The embedding position acquisition module is used to group the parameters of the large language model according to the degree of influence of each parameter in the large language model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0097] The steganographic data embedding module is used to generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position.

[0098] For quantitative scenarios, see Figure 5 As shown, the data adaptive hiding method based on a large language model in an embodiment of the present invention can be designed to include:

[0099] S101, uniformly linearly quantizing the initial large language model parameters using a quantization scaling factor to obtain a quantization model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true values ​​of the large language model parameters;

[0100] S102, grouping the parameters of the quantitative model according to the degree of influence of each parameter in the quantitative model on the model performance, and selecting the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter types, common parameter types, and important parameter types;

[0101] S103, generating an integrity check code for the data to be steganographic, packaging the data to be steganographic and its integrity check code into binary data, and secretly embedding the binary data into the target parameter position;

[0102] S104: Dequantize the model embedded in the binary data, and replace the embedded target parameter position region with the parameter region corresponding to the initial large language model.

[0103] The parameters of the quantitative model are grouped according to the degree of influence of each parameter in the quantitative model on the model performance, which may include:

[0104] Obtain the optimal parameters of the quantization model, and record the model performance under the optimal parameters as the maximum value;

[0105] According to the neuron quantization scaling factor in the quantization model, neurons whose quantization scaling factor is greater than a specified value are selected as parameter replacement regions;

[0106] Using random binary bits to replace the corresponding neuron parameters in the optimal parameters of the quantized model parameters, and obtaining the model performance after the neuron parameters are replaced;

[0107] The absolute value of the model performance loss before and after replacement is used to measure the importance of the corresponding neuron parameters, so that the neuron parameters in the parameter replacement area are grouped according to the importance of the neuron parameters.

[0108] For LLM in the quantization scenario, the importance of the neurons in the uniform linear quantization is related to the quantization scaling factor. The larger the scaling factor, the more critical the neurons. The scaling factor determines the spacing of the quantization levels, that is, the proportional relationship between the quantization value and the original value. Under the assumed conditions, the parameter θ is the optimal parameter of LLM, and the performance of LLM reaches the highest value at this time. Given To quantize the scaling factor of the neuron, the parameters of the quantized neuron are as follows:

[0109]

[0110] The parameters obtained by dequantization are as follows:

[0111]

[0112] Where Q is the uniform linear quantization algorithm, Q ′ is the corresponding inverse quantization algorithm,

[0113] Q ′ S (τ) = S (τ + Z).

[0114] Assuming that after quantization, τ is also the optimal parameter of LLM, the model has performance Logically speaking, the change in model performance is related to the numerical difference before and after the parameter modification. The larger the numerical difference, the greater the loss in model performance. In the quantized scenario, random bit replacement of τ is performed to obtain Compared with τ, there is a numerical change, that is, At this time, the inverse quantized parameters are: With θ ′ Compared with , there will be a numerical change, that is, The numerical difference is That is, in terms of value, in addition to the γ caused by bit replacement, it is also determined by the scaling factor S. The larger S is, the greater the value change is, and the greater the impact on model performance is. Figure 6 Let's take an example to further illustrate. Assume there is a float16 type θ α =[200,400,800] and θ β =[0.2,0.4,0.8], their quantization and dequantization scales are s α =100 and s β =0.1. The quantization results of both are [2,4,8]. If they are embedded, the values ​​change to [3,5,9]. After dequantization, With θ α The difference in value is 100; With θ β The numerical difference is only 0.1. Therefore, it can be considered that in the quantization scenario, for neurons with uniform linear quantization, the larger the scaling factor, the greater the importance and the more critical the neuron.

[0115] Quantization is not only a common way for users to import models, but also a lightweight way to deploy LLMs. It can also destroy the integrity of embedded data. Moreover, the size of the quantized model is significantly smaller than that of the non-quantized model, and the performance loss is obvious when embedding large-capacity payloads. Therefore, the design of quantitative steganography needs to be considered from two aspects: how to ensure the integrity of the embedded data, and how to reduce the performance loss caused by embedding.

[0116] If the user deploys the model by quantizing the model, the data embedded only by half-precision LSB steganography will be damaged, thus rendering the attack ineffective. To this end, in this embodiment, the quantized model is first embedded and then dequantized to obtain the LLM after the embedded data. Using this idea to design quantized steganography can effectively ensure the integrity of the embedded data. Specifically, when modifying the quantized int parameter τ, It is still of type int. Now dequantize it. get At this time, it is quantified and the calculation method is as follows:

[0117]

[0118] After dequantization and requantization, the value does not change. Therefore, the integrity of the data embedded during quantization will not be destroyed after dequantization and quantization operations, thus realizing the hidden embedding of data information in the quantization scenario.

[0119] In order to explore how to reduce the performance loss caused by embedding, in this embodiment, the performance changes of the model in the embedding process are analyzed from the perspective of quantitative steganalysis. Figure 7 As shown in the figure, due to the Int operation in the uniform linear quantization algorithm, when converting float type data to int type data, the decimal part of the float type data will be ignored, which will cause loss to the overall performance of the model. q In the embedding stage, the model loss Loss is caused by the numerical change e In the dequantization stage, there is no loss of precision in the numerical value, and the model will not incur performance loss. The performance loss of the final model is Loss all =Loss q +Loss e Among them, Loss q It is determined by the quantization algorithm and cannot be controlled. What can be done in this embodiment is to make Loss e As small as possible. To this end, in the embedding stage, consider using the same method as Half-precision LSB steganography, and change the input parameter type to int type. In addition, due to the limited length of the embedded content, sometimes not all quantized parameters are embedded, and the parameters that are not embedded suffer precision loss due to quantization and dequantization operations. Therefore, in the end, only the embedded parameter area needs to be replaced with the original model, so that the values ​​of the non-embedded area maintain the original precision, thereby minimizing the precision loss of the model.

[0120] The quantization LSB steganography process is shown in Algorithm 2. The first step is to quantize M to obtain M'. The second step is the same as the process of half-precision LSB steganography, the only difference is the input parameters type, which determines that when Quantization LSBsteganography performs the embedded mask operation, the binary bit string q is converted to int type instead of float type during type conversion. The third step is to dequantize M' embedded in the payload. Finally, replace the embedded parameter area in M' with the original model M, and return M as the malicious model M'.

[0121]

[0122] Furthermore, for the hiding method of the above quantization scenario, an embodiment of the present invention also provides a data adaptive hiding system based on a large language model, comprising: a model parameter quantization module, an embedding position acquisition module, a stego data embedding module and a model parameter dequantization module, wherein:

[0123] A model parameter quantization module, used to uniformly linearly quantize the initial large language model parameters using a quantization scaling factor to obtain a quantized model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true value of the large language model parameters;

[0124] The embedding position acquisition module is used to group the parameters of the quantization model according to the degree of influence of each parameter in the quantization model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type;

[0125] The steganographic data embedding module is used to generate the integrity check code of the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position;

[0126] The model parameter dequantization module is used to dequantize the model embedded in the binary data and replace the embedded target parameter position area with the parameter area corresponding to the initial large language model.

[0127] For the LLM parameter file, the key parameters of covert embedding are located and the parameter importance is evaluated based on ablation and quantization scaling factors to find the parameters with little impact on LLM performance as the key embedding positions; then, a steganography strategy is formulated based on the parameter importance, and half-precision LSB steganography and quantized LSB steganography are implemented for non-quantized scenarios and quantized scenarios respectively, ensuring that large-capacity data information can be covertly embedded in LLM regardless of whether the user quantizes the compression model, while ensuring the integrity of the data information.

[0128] For the large language model steganographic data, a trigger parsing strategy can be designed to parse the data information hidden in adaptive steganography. After the trigger is activated, all the parameter weights of the LLM will be obtained. Then, the trigger will extract the embedded data information and execute it according to the rules of the steganographic strategy. The process of trigger extraction and parsing embedded data information is shown in Algorithm 3. The first step is to initialize the parameter weight matrix W and the embedding bit number N1, N2, N3 of the corresponding parameters according to the adaptive strategy and the classification of non-critical, common, and important parameters. The second step is to extract the embedded binary string in W according to the length L of the embedded binary string and the number of embedded bits N1, N2, N3. The third step is to separate the checksum and payload, verify the integrity of the embedded data and decode it.

[0129]

[0130]

[0131] In order to verify the effectiveness of this solution, the following is a further explanation of this solution combined with experimental data:

[0132] In the experiment, the chatglm3-6b model parameter file is used. The file format is safetensors. The total number of parameters is 6.24 billion and the size is 11.63GB. The pytorch framework (pytorch version 2.3.0+cu121) is used to operate the parameter weights. The quantization function officially provided by chatglm3-6b is used for model quantization, and the corresponding inverse quantization function is written to implement inverse quantization. The operating system environment of the experiment is Ubuntu 22.04.4LTS (GNU / Linux 5.15.0-106-generic x86_64). The CPU used includes 64 AMD EPYC 7282 16-Core Processors with x86_64 architecture and a total memory size of 125.7GB. The GPU includes 2 NVIDIA RTX A6000 (video memory 40GB), 1 NVIDIA A100-PCIE-40GB (video memory 40GB), cuda version 12.2. The MMLU (Massive Multitask Language Understanding) dataset is used to evaluate the performance of the model. The dataset is in the form of four-choice multiple-choice questions, covering 57 subjects, including STEM (science, technology, engineering, mathematics), humanities, social sciences and other fields, aiming to measure the generalization ability and knowledge transfer ability of the model in zero-sample and few-sample situations. The formula for measuring the performance index of the model is Performance = C / N. Where C represents the number of questions answered correctly, and N represents the number of all questions. The indicator of embedding capacity is the size of embedded data. In particular, the embedded data used in the experiment is a randomly generated string of binary bit streams to simulate the embedded data in real scenarios.

[0133] 1. Layer importance analysis

[0134] It is expected to find the parameter importance of different parameters in the model to determine the important parameters. First, the experiment is carried out with layer as the minimum steganographic unit. chatglm3-6b has a total of 200 parameter layers. Among them, for parameter layers with only one dimension, such as normalization layers, bias layers, etc., these layers have a small number of parameters and can be embedded in less content within an acceptable performance variation range. Therefore, when looking for key layers, only 4*28+2=114 layers are considered. Among them, 4*28 represents the self_attention.query_key_value layer, self_attention.dense layer, mlp.dense_h_to_4h layer, mlp.dense_4h_to_h layer in the 28 blocks contained in the core part GLMTransformer.

[0135] In both non-quantized and quantized scenarios, we conducted ablation experiments based on random bit replacement on the layers in 28 blocks to explore the relationship between the steganographic capacity of each layer and the performance change of the model. It can be found that almost all layers have similar threshold effects. After exceeding the threshold of the number of LSB bits, the model performance will drop significantly. Taking block 0 as an example, Figure 8 The results show that the threshold of self_attention.query_key_value layer and mlp.dense_h_to_4h layer is 11, and the threshold of self_attention.dense layer and mlp.dense_4h_to_h layer is 12.

[0136] The reason is that the impact of steganography on model performance is mainly reflected in the change of parameter values. When the number of LSB bits of the parameters is embedded, the value only changes slightly and does not affect the performance of LLM. When the number of LSB bits reaches 11 or even 12, the exponential part changes, the value fluctuation becomes more obvious, and the performance of the model almost completely disappears.

[0137] In order to explore the importance of the overall parameters of the model, we use the block number as the horizontal axis, select the LSB steganographic bit as 11, and analyze the performance changes of each layer, such as Fig. 9 As shown in the figure, it can be intuitively found that the performance loss caused by embedding the first 2 / 3 parameters of the entire model is significantly higher than that of the last 1 / 3 parameters. Therefore, it can be considered that the first 2 / 3 parameters of the entire model are more critical.

[0138] 2. Neuron importance analysis

[0139] For neurons with uniform linear quantization, the scaling factor will be saved during the quantization process. In order to verify the effectiveness of the method of judging the importance of neurons by scaling factors, mlp.dense_h_to_4h layers, mlp.dense_4h_to_h layers, and self_attention.dense layers are taken, and neurons with the largest and smallest scaling factors of each layer are taken in the same proportion for comparative embedding analysis. The two parts are steganographically processed with 5 to 8 bits of LSB, and the performance of the models after the two parts are steganographically processed is compared. The proportions taken are 10%, 30%, and 50%, respectively. The experimental results are shown in the figure. Fig.10 As shown in the figure, Major represents a large scaling factor, and Minor represents a small scaling factor.

[0140] exist Fig.10 It can be found that for mlp.dense_h_to_4h layer, mlp.dense_4h_to_h layer, self_attention.dense layer: under the same LSB bit number, the model performance loss caused by steganography of neurons with small scaling factors is relatively small. In this way, it is proved that in these linear layers, the size of the scaling factor is an effective indicator for judging the importance of neurons.

[0141] 3. Half-precision LSB steganalysis

[0142] Verify that the use of adaptive steganography can effectively reduce the performance loss caused by embedding. First, without using any strategy, LSB steganography with the same LSB number is performed on all layers equally, and the performance and maximum embedding amount of the model in this case are used as the benchmark. Fig.11 The middle dashed line result shows that when the number of embedded bits reaches 8, the LLM performance is affected by less than 1%, but the hidden data content can reach 5.81GB. However, when the number of embedded bits reaches 11, LLM loses almost all its capabilities.

[0143] Next, we use the steganographic strategy: (1) take all (block0 to block27) self_attention.query_key_value layers as the key layers, take the last 1 / 3 (block18 to block27) mlp.dense_4h_to_h layers as the non-key layers, and the remaining non-one-dimensional layers as common layers; (2) embed the common layers in i bits, the key layers in i-1 bits, and the non-key layers in i+1 bits.

[0144] Under this strategy, compared with the baseline method of embedding i bits in all layers, the capacity of embedded data is only 3.83MB more, which is negligible. It can be considered that the embedding capacity of the two strategies is the same. Fig.11 As shown in the figure, the performance of this strategy embedding is significantly improved compared with the baseline method when the same amount of data is embedded. This shows that adaptive steganography can embed less important parameters and embed less important parameters first, so as to embed as much data as possible and effectively reduce the performance loss caused by embedding, thus making the embedded information difficult for users to discover.

[0145] 4. Quantitative LSB steganalysis

[0146] In the quantization scenario, the parameter type of the model changes and the capacity will be reduced. Taking int8 quantization as an example, the total number of parameters of chatglm3-6b after int8 quantization is still 6.24 billion, and the size is 6.31GB. In order to verify the effect of using adaptive steganography in the quantization scenario. First, without using any strategy, all parameters of the quantization model are subjected to LSB steganography with the same number of LSB bits, and the performance and maximum embedding amount of the model in this case are used as the benchmark. As shown in Figure 12, as shown by the green dotted line, when the number of embedding bits reaches 3, the LLM performance is affected by less than 1%, but the hidden malicious content can reach 2.18GB. When the number of embedding bits reaches 5, LLM loses all its capabilities.

[0147] Next, based on the layer strategy of 6.3.1, a more fine-grained strategy is designed based on the importance of quantified neurons. For the non-critical layer (mlp.dense_4h_to_h layer of block18 to block27), neurons with the largest 50% scaling factor are selected as critical neurons, and neurons with the smallest 50% scaling factor are selected as non-critical neurons. Critical neurons are embedded one bit less (i bits), and non-critical neurons are embedded one bit more (i+2 bits). Fig.12 As shown, by using this adaptive steganography strategy embedding, compared with the baseline, which only uses the layer strategy of 6.3.1, as shown by the red dotted line in the figure, the model performance is further improved, and the embedded information is more difficult for users to detect.

[0148] Although steganography based on adaptive strategies can effectively reduce the performance loss caused by embedding, the finer-grained strategies have more complex logic and trigger parsing strategies, which increases the volume of trigger code and makes the trigger easier to detect.

[0149] The above experimental data show that the solution in this case can embed 5.81GB and more than 2.18GB of large amounts of data information respectively, with a performance sacrifice of less than 1%, in non-quantized and quantized scenarios of the model. Regardless of whether the user quantizes the compression model or not, large amounts of data information can be hidden in the LLM while ensuring the integrity of the data information, further proving the effectiveness of the solution in this case.

[0150] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0151] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0152] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0153] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0154] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.< / eos>

Claims

1. A data adaptive hiding method based on a large language model, characterized in that: Include: The parameters of the large language model are grouped according to the degree of influence of each parameter in the large language model on the model performance, and the target parameter position for data secret embedding is selected from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type; Generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position.

2. The data adaptive hiding method based on a large language model according to claim 1 is characterized in that: The parameters of the large language model are grouped according to the degree of influence of each parameter on the model performance, including: Obtain the optimal parameters of the large language model, and record the performance of the large language model under the optimal parameters as the maximum value; According to the parameter size of each layer in the large language model structure, the neuron layer with a parameter value greater than the specified value is selected as the parameter replacement area; Using random binary bits to replace the neuron parameters in the optimal parameter replacement area of ​​the large language model parameters, and obtaining the performance of the large language model after the neuron parameter replacement; The absolute value of the performance loss of the large language model before and after replacement is used to measure the importance of the corresponding neuron parameters, so that the neuron parameters in the parameter replacement area are grouped according to the importance of the neuron parameters.

3. The data adaptive hiding method based on a large language model according to claim 1 is characterized in that: Select the target parameter position for data hidden embedding from the grouped model parameters, including: Initialize the embedding bits of the non-critical parameters, common parameters and important parameters of the large language model, and concatenate the weight matrices of the non-critical parameters, common parameters and important parameters to obtain a large language model parameter weight matrix, where the parameter weight matrix is ​​used to store the connection weights between neurons of the large language model for transmitting information; According to the number of embedding bits of non-critical parameters, common parameters and important parameters of the large language model, corresponding mask matrices for least significant bit embedding are created respectively.

4. The data adaptive hiding method based on a large language model according to claim 3 is characterized in that: Embed binary data secretly into the target parameter location, including: Determine the model parameter grouping type corresponding to each matrix element in the large language model parameter weight matrix; If the matrix element is a non-key parameter, extract the binary data element according to the number of embedded bits of the non-key parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the non-key parameter to update the current matrix element; If the matrix element is a common parameter, extract the binary data element according to the number of bits embedded in the common parameter and assign it to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable to the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the common parameter to update the current matrix element; If the matrix element is an important parameter, determine whether the length of the binary data is less than the number of bits of the important parameter embedding bits. If so, convert the binary data into the specified data type. If not, extract the binary data elements based on the number of bits of the important parameter embedding bits and assign them to the first variable, and assign the remaining binary data elements that have not been extracted to the second variable; convert the first variable into the specified data type, and perform bitwise operations on the converted first variable based on the mask matrix of the important parameters to update the current matrix element.

5. A data adaptive hiding system based on a large language model, characterized in that: It includes: an embedding position acquisition module and a steganographic data embedding module, wherein: The embedding position acquisition module is used to group the parameters of the large language model according to the degree of influence of each parameter in the large language model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type; The steganographic data embedding module is used to generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position.

6. A data adaptive hiding method based on a large language model, characterized in that: Include: Performing uniform linear quantization on the initial large language model parameters using a quantization scaling factor to obtain a quantized model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true values ​​of the large language model parameters; The parameters of the quantitative model are grouped according to the degree of influence of each parameter in the quantitative model on the model performance, and the target parameter position for data hidden embedding is selected from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type; Generate an integrity check code for the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position; The model embedded in the binary data is dequantized, and the embedded target parameter position area is replaced with the parameter area corresponding to the initial large language model.

7. The data adaptive hiding method based on a large language model according to claim 6 is characterized in that: The parameters of the quantitative model are grouped according to the degree of influence of each parameter on the model performance, including: Obtain the optimal parameters of the quantization model, and record the model performance under the optimal parameters as the maximum value; According to the neuron quantization scaling factor in the quantization model, neurons whose quantization scaling factor is greater than a specified value are selected as parameter replacement regions; Using random binary bits to replace the corresponding neuron parameters in the optimal parameters of the quantized model parameters, and obtaining the model performance after the neuron parameters are replaced; The absolute value of the model performance loss before and after replacement is used to measure the importance of the corresponding neuron parameters, so that the neuron parameters in the parameter replacement area are grouped according to the importance of the neuron parameters.

8. A data adaptive hiding system based on a large language model, characterized in that: It includes: model parameter quantization module, embedding position acquisition module, steganographic data embedding module and model parameter dequantization module, among which, A model parameter quantization module, used to uniformly linearly quantize the initial large language model parameters using a quantization scaling factor to obtain a quantized model, wherein the quantization scaling factor is set according to the upper and lower bounds of the true value of the large language model parameters; The embedding position acquisition module is used to group the parameters of the quantization model according to the degree of influence of each parameter in the quantization model on the model performance, and select the target parameter position for data secret embedding from the grouped model parameters, wherein the grouping types of the model parameters include non-critical parameter type, common parameter type and important parameter type; The steganographic data embedding module is used to generate the integrity check code of the data to be steganographic, package the data to be steganographic and its integrity check code into binary data, and secretly embed the binary data into the target parameter position; The model parameter dequantization module is used to dequantize the model embedded in the binary data and replace the embedded target parameter position area with the parameter area corresponding to the initial large language model.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 4 and 6 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 4 and 6 to 7 can be implemented.

Images