A sensitive prompt word generation method and system based on parameter sensitivity quantification

Abstract

The application discloses a sensitive prompt word generation method and system based on parameter sensitivity quantification, relates to the technical field of artificial intelligence security, and aims to solve the problem of sensitive test sample generation difficulty in black box large language model integrity verification. The method comprises the following steps: obtaining an original large language model; constructing a comprehensive parameter sensitivity index, which combines micro sensitivity for representing small parameter modification and macro sensitivity for representing parameter quantification, pruning and other substantial modification, so as to comprehensively quantify the sensitivity of the prompt word; in a continuous embedding space, a gradient optimization algorithm is used to iteratively optimize the comprehensive index, and semantic rationality and other constraints are applied to ensure the natural fluency of the generated prompt word; finally, the optimized embedding vector is mapped back to a discrete word sequence to obtain the final sensitive prompt word. The application realizes low-cost and automatic test sample generation, and significantly improves the accuracy, efficiency and concealment of cloud model integrity verification.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, and in particular relates to a method and system for generating sensitive prompt words to verify the integrity of a large language model in the cloud. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by Large Language Models (LLMs), have developed rapidly. Model as a Service (MaaS) has become the mainstream business model, where cloud service providers deploy pre-trained large language models in the cloud and provide services to users through application programming interfaces (APIs). This model has greatly lowered the barrier to entry for users to use large models.

[0003] However, this black-box API service model also brings serious security challenges. Users cannot directly access and inspect the model entities in the cloud, so service providers or potential attackers may maliciously tamper with, replace, or modify the model without authorization. For example, replacing a claimed high-performance model with a model with fewer parameters and lower performance to save costs; or performing improper quantization, pruning, or other compression operations on the model, leading to a decrease in model performance; or even implanting backdoors into the model, causing it to produce harmful or erroneous outputs when processing specific inputs.

[0004] To verify the integrity of cloud-based models, i.e., to confirm their consistency with the original open-source version, an effective detection method is needed. Existing methods typically rely on functional testing or performance evaluation, but these require numerous API calls, are costly and inefficient, and are difficult to detect problems caused by minor parameter changes. Therefore, designing a low-cost, high-efficiency, and highly sensitive test sample (i.e., cue words) to effectively detect potential modifications to cloud-based models has become a pressing technical challenge in this field. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] The present invention aims to solve the problem in the prior art that it is difficult to efficiently generate test samples that can effectively detect small or large modifications in the parameters of large language models, and provides a method and system that can automatically generate prompt words that are highly sensitive to changes in model parameters.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, embodiments of the present invention provide a method for generating sensitive prompt words based on parameter sensitivity quantization, the method comprising the following steps:

[0009] S100: Obtain the original large language model for generating sensitive prompt words. This original large language model has defined model parameters θ and model structure.

[0010] S200: Determine a comprehensive parameter sensitivity index S hybrid This metric quantifies the sensitivity of the input prompt word x to changes in model parameters θ. This comprehensive parameter sensitivity metric includes:

[0011] Microscopic sensitivity S micro : Used to characterize the model output f when the model parameter θ undergoes a small perturbation. θ The degree of change of (X).

[0012] Macroeconomic Sensitivity S macro : Used to characterize the model output f when the model parameters θ are significantly modified (e.g., quantization, pruning). θ The degree of change of (X).

[0013] S300: In a continuous embedding space, a preset optimization algorithm is used to maximize the comprehensive parameter sensitivity index S. hybrid To achieve the goal, the embedding vector of the initial prompt word is iteratively optimized to obtain the optimized embedding vector e. * .

[0014] S400: The optimized embedding vector e * Mapping back to discrete token sequences generates the final sensitive prompt word v. * .

[0015] Further, in step S200, the microscopic sensitivity S micro It is determined by calculating the norm of the gradient of the model output with respect to the model parameters. Specifically, the calculation formula is as follows:

[0016]

[0017] Among them, f θ (x) is the model's output on input x, ‖·‖ F Let Frobenius norm denote the matrix.

[0018] Further, in step S200, the macroscopic sensitivity S macro The estimation is performed by sampling various preset parameter modification types. Specifically, the calculation formula is as follows:

[0019]

[0020] Where Ω is the set of distributions of parameter modifications, and Δθ is a specific instance of parameter modification sampled from Ω. This indicates the desired outcome. The parameter modification types include at least one or more of model quantization, model pruning, and model fine-tuning.

[0021] Further, in step S200, the comprehensive parameter sensitivity index S hybrid By measuring the microscopic sensitivity S micro and macroscopic sensitivity S macro We get the following by weighted summation:

[0022] S hybrid (x,θ)=α·S micro (x,θ)+(1-α)·S macro (x,θ)

[0023] Where α∈[0,1] is a preset or adaptively adjusted balance factor.

[0024] Further, in step S300, the optimization algorithm is the gradient ascent method. The iterative optimization process includes:

[0025] • Map the initial prompt word v0 to the initial embedding vector e0 through the model's embedding layer.

[0026] • In each iteration, calculate the comprehensive parameter sensitivity index S. hybrid gradient with respect to the current embedding vector e

[0027] • Update the embedding vector along the gradient direction: Where η is the learning rate.

[0028] Furthermore, during the iterative optimization process in step S300, at least one constraint condition is applied, including:

[0029] • Semantic rationality constraints: Ensure that the generated prompts are semantically fluent and meaningful, for example, by limiting their perplexity under another pre-trained language model to below a preset threshold.

[0030] • Small perturbation constraint: Ensure that the generated prompt words are similar in form to the natural language text, for example, by limiting the distance between their embedding vector and the initial prompt word embedding vector (such as Euclidean distance or cosine distance) to a preset range.

[0031] Further, in step S400, the optimized embedding vector e is... * The method for mapping back to discrete word sequences is as follows: for each position of the embedding vector sequence, find the word embedding that is most similar to it (e.g., the highest cosine similarity) in the model's vocabulary embedding space, and take the corresponding word as the output of that position.

[0032] Another embodiment of the present invention provides a sensitive prompt word generation system based on parameter sensitivity quantization, the system comprising:

[0033] 1) Model acquisition module, used to acquire the original large language model and its parameters and structure of the sensitive prompt words to be generated.

[0034] 2) Sensitivity calculation module, used to determine a comprehensive parameter sensitivity index S according to a preset calculation method. hybrid This metric is used to quantify the sensitivity of input prompts to changes in model parameters.

[0035] 3) The prompt word optimization module is used to iteratively optimize the embedding vector of the initial prompt word in a continuous embedding space with the goal of maximizing the sensitivity index of the comprehensive parameter, so as to obtain the optimized embedding vector.

[0036] 4) The prompt word generation module is used to map the optimized embedding vector back to the discrete word sequence to generate the final sensitive prompt words.

[0037] Furthermore, the sensitivity calculation module also includes:

[0038] ① Micro-sensitivity calculation unit, used to calculate the norm of the gradient of the model output with respect to the model parameters, to obtain the micro-sensitivity.

[0039] 2. Macro-sensitivity calculation unit, used to estimate macro-sensitivity by sampling various preset parameter modification types.

[0040] ③ The index fusion unit is used to perform weighted summation of micro-sensitivity and macro-sensitivity to obtain a comprehensive parameter sensitivity index.

[0041] (III) Beneficial Effects

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. High sensitivity and high efficiency: This invention quantifies and maximizes a sensitivity index that integrates changes in micro and macro parameters, and can automatically generate prompt words that are highly sensitive to various model modifications (from fine-tuning to large-scale quantization), which significantly improves the sensitivity and efficiency of detection.

[0044] 2. Automated generation: This invention provides a complete end-to-end automated generation process, which gets rid of the inefficient mode of manually designing test cases by relying on expert experience. It can generate customized sensitive prompt words for different models on a large scale and quickly.

[0045] 3. Excellent concealment: The present invention incorporates semantic rationality and small perturbation constraints during the optimization process, making the generated sensitive prompts indistinguishable from ordinary and natural language in form, making them less likely to be identified as probe behavior by cloud-based defense mechanisms, thus enhancing the concealment and success rate of verification.

[0046] 4. Wide applicability: The theory and method of this invention do not depend on a specific model architecture and can be widely applied to various mainstream Transformer architecture large language models, demonstrating good versatility. Attached Figure Description

[0047] Figure 1 This is a flowchart of a sensitive prompt word generation method based on parameter sensitivity quantization according to an embodiment of the present invention.

[0048] Figure 2 This is a structural block diagram of a sensitive prompt word generation system based on parameter sensitivity quantization according to an embodiment of the present invention.

[0049] Figure 3 yes Figure 1 A more detailed flowchart of the iterative optimization process in step S300. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0051] Please see Figure 1 This illustrates the flow of a sensitive prompt word generation method according to an embodiment of the present invention.

[0052] Step S100: Obtain the original large language model.

[0053] This step is fundamental to this method. The device executing this method (e.g., one or more servers) first needs complete access to a trusted, unmodified original large language model, such as a full copy of the Llama-70B or DeepSeek-67B model released by the open-source community. This means obtaining all the model's weight parameters θ, network structure definitions, and related vocabulary and embedding layer information. This information is a prerequisite for subsequent calculations of parameter sensitivity.

[0054] Step S200: Determine the sensitivity index S of the comprehensive parameter. hybrid .

[0055] This step is one of the core aspects of the invention, aiming to establish a mathematical metric that can accurately measure the "amplification" ability of the input prompt word x on changes in model parameters θ.

[0056] In this embodiment, S hybrid It consists of two parts:

[0057] 1) Microscopic sensitivity S micro Calculation:

[0058] Microsensitivity is primarily used to capture minute changes in parameters caused by model fine-tuning. When the parameter change Δθ is sufficiently small, the change in the model output can be approximated by a first-order Taylor expansion, and its magnitude is mainly determined by the gradient of the output with respect to the parameters. Therefore, we define microsensitivity as the square of the Frobenius norm of the gradient of the model output with respect to the parameter θ:

[0059]

[0060] Among them, f θ (x) can be the logits vector output by the model at the last token position. Calculating this value requires one forward propagation and one backward propagation of the model. A larger gradient norm indicates that the output is more sensitive to small changes in the parameters.

[0061] 2) Macroscopic sensitivity S macro Calculation:

[0062] Macroscopic sensitivity is used to capture large parameter changes caused by model quantization, pruning, etc. These changes are not suitable for Taylor expansion approximation. Therefore, this embodiment uses a sampling-based method for estimation. Specifically, we define a distribution set Ω of parameter modification types, which includes various model modification methods that may occur in real-world scenarios. For example:

[0063] • Quantization sampling: Simulate the operation of quantizing the model parameters from FP16 to different bit widths such as INT8 and INT4 to obtain the modified parameters θ. quant .

[0064] • Pruning and Sampling: Simulate pruning the model weights at different sparsities (e.g., 10%, 30%) to obtain the modified parameters θ. prune .

[0065] • Fine-tuning sampling: Simulating the parameter changes θ that occur after fine-tuning with a large learning rate or for a specific task. finetune .

[0066] By randomly selecting M modification instances from Ω We can calculate the difference in the model's output before and after the modification, and then calculate its expectation (in practice, the mean is used instead):

[0067]

[0068] Where ||·||2 represents the L2 norm of the vector.

[0069] 3) Indicator integration:

[0070] Finally, the two sensitivities mentioned above are weighted and fused using a balancing factor α to obtain a comprehensive sensitivity index:

[0071] S hybrid (x,θ)=α·S micro (x,θ)+(1-α)·S macro (x,θ)

[0072] The balancing factor α can be preset according to the focus of the validation task. For example, if more attention is paid to detecting minor modifications such as backdoor implantation, α can be increased; if more attention is paid to detection model replacement or over-compression, α can be decreased. In some embodiments, α can also be based on S. micro and S macro The relative size is adaptively adjusted.

[0073] Step S300: Perform iterative optimization in the continuous embedding space.

[0074] Optimizing prompt words directly in a discrete text space is an NP-hard problem. This invention innovatively elevates the optimization process to a continuous embedding space. Detailed procedures for this step can be found in [reference needed]. Figure 3 .

[0075] S310: Select a natural sentence or generate it randomly as the initial prompt word v0, and use the embedding layer Emb(·) of the original model to convert it into the initial embedding vector sequence e0 = Emb(v0).

[0076] S320: Enter the iteration loop. In each iteration, perform gradient ascent to update the embedding vector e. To make the objective function S hybrid Differentiability of e requires a soft-decoding process from the embedding e to the input x. For example, the Gumbel-Softmax trick or a simple softmax function can be used to obtain a differentiable vector representing the input X. Then, the gradient of the objective function with respect to e is computed.

[0077] S330: Update e along the gradient direction:

[0078] S340: Impose constraints on the updated e. For example, through projection or other optimization techniques, ensure that e satisfies:

[0079] • Semantic rationality: Calculate the perplexity of the prompt word mapped back by e. If it is higher than the threshold, adjust it towards lower perplexity.

[0080] • Small perturbation: Calculate the distance between e and e0. If it exceeds the threshold, pull it back into the sphere with e0 as the center and ∈ as the radius.

[0081] S350: Check if the convergence condition is met (e.g., the maximum number of iterations is reached, or S...). hybrid (Growth stagnation). If not satisfied, return to S320 to continue the next iteration; if satisfied, the optimized embedding vector e is obtained. * And end the loop.

[0082] Step S400: Map the optimization results back to the discrete word sequence.

[0083] This step transforms the optimization results in the continuous space back into usable text prompts. For the optimized embedding vector sequence... For each vector in the sequence Search for [the model's] embedding matrix across the entire vocabulary. The lexical embedding with the highest cosine similarity is used as the final sensitive prompt word v. * The output at this location.

[0084] The final result of v * These are prompt words that are highly sensitive to changes in model parameters.

[0085] Please see Figure 2 This diagram illustrates a structural block diagram of a sensitive keyword generation system according to an embodiment of the present invention. The system can be a software program deployed on a server, comprising:

[0086] Model Acquisition Module 10: Responsible for loading the specified original large language model and its complete parameter file and configuration file from storage devices (such as hard disks and network storage).

[0087] • Sensitivity Calculation Module 20: This is the core of the system's calculations. It can be further divided internally into:

[0088] Microscopic sensitivity calculation unit 21: Implements backpropagation, calculates gradient and norm.

[0089] Macroscopic sensitivity calculation unit 22: Implements sampling (quantization, pruning, etc.) and forward propagation of parameter modifications, and calculates the output difference.

[0090] Indicator Fusion Unit 23: Based on the set α value, the outputs of the first two units are weighted and summed.

[0091] • Prompt word optimization module 30: responsible for execution Figure 3 The iterative optimization process is shown. It receives metrics from the sensitivity calculation module 20 as optimization targets and continuously adjusts the embedding vector.

[0092] • Prompt word generation module 40: After optimization, it receives the final embedding vector e from prompt word optimization module 30. * It then performs a nearest neighbor search and converts it into human-readable text prompts (v). * It can be output or stored.

[0093] The modules work together. The model acquisition module 10 provides basic data for the sensitivity calculation module 20, the sensitivity calculation module 20 provides optimization direction for the prompt word optimization module 30, and the output of the prompt word optimization module 30 is finally transformed into the final product by the prompt word generation module 40.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the spirit and principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A parameter sensitivity quantification-based sensitive prompt word generation method, characterized in that, The method comprises: obtaining a preset original large language model, the original large language model having determined model parameters; determining a comprehensive parameter sensitivity index, the comprehensive parameter sensitivity index being used to quantify the sensitivity degree of an input prompt word to modification of the model parameters; in a continuous embedding space, iteratively optimizing an embedding vector of an initial prompt word to obtain an optimized embedding vector, with the goal of maximizing the comprehensive parameter sensitivity index; and mapping the optimized embedding vector back to a discrete token sequence to generate the sensitive prompt word. The comprehensive parameter sensitivity index comprises: S1 00: obtain an original large language model to be generated sensitive prompt word, the original large language model has determined model parameter and model structure; S200: determine an integrated parameter sensitivity index , which is used to quantify the input prompt word for the model parameters sensitivity of the modification; Further, in step S300, the optimization algorithm is a gradient ascent method; the iterative optimization process comprises: 1) Microsensitivity : For characterizing the degree of change in the model output when the model parameters are subjected to a small perturbation; 2) macro-sensitivity : for characterizing the degree of change in model output when model parameters are substantially modified; S300: In the continuous embedding space, by a preset optimization algorithm, to maximize the comprehensive parameter sensitivity index For the target, the initial prompt word embedding vector is iteratively optimized to obtain an optimized embedding vector ; S400: map the optimized embedding vectors back to a discrete sequence of word pieces, generating a final sensitive prompt word ;​ Further, in step S200, the micro sensitivity is determined by calculating the norm of the gradient of the model output with respect to the model parameters; in particular, it is calculated as follows: wherein, is the output of the model to the input , denotes the Frobenius norm of a matrix; Further, in step S200, the macro sensitivity is estimated by sampling over a pre-defined plurality of parameter modification types; in particular, it is computed as wherein, is a set of distributions of parameter modifications, is a specific parameter modification instance sampled from , and represents expectation. The parameter modification type includes one or more of model quantization, model pruning, and model fine-tuning. Further, in step S200, the integrated parameter sensitivity indicator is obtained by weighted sum of the micro sensitivity and the macro sensitivity ​ wherein, is a preset or adaptively adjusted balance factor; Further, in the iterative optimization process of step S300, at least one constraint condition is further applied, the constraint condition comprising: 1) the initial prompt word mapped to an initial embedding vector by the embedding layer of the model ; 2) In each iteration, compute the integrated parameter sensitivity indicator the gradient of the current embedding vector ;​ 3) Update the embedding vectors along the gradient direction: where is the learning rate; ① semantic reasonableness constraint: ensuring that the generated prompt word is fluent and meaningful in semantics, by limiting the perplexity of the prompt word under another pre-trained language model to be lower than a preset threshold; ② small perturbation constraint: ensuring that the generated prompt word is similar in form to natural language text, by limiting the distance between the embedding vector of the prompt word and the embedding vector of the initial prompt word to be within a preset range. The system comprises: Further, in step S400, the optimized embedding vectors The method of mapping back the discrete token sequence is to find the most similar token embedding to the vector at each position in the embedding vector sequence in the vocabulary embedding space of the model, and take the corresponding token as the output of the position.

2. A parameter sensitivity quantification based sensitive prompt word generation system, wherein, a model obtaining module, configured to obtain a preset original large language model and model parameters thereof; a sensitivity calculating module, configured to determine a comprehensive parameter sensitivity index, the comprehensive parameter sensitivity index being used to quantify the sensitivity degree of the input prompt word to modification of the model parameters; a prompt word optimization module, configured to iteratively optimize an embedding vector of an initial prompt word in a continuous embedding space, with the goal of maximizing the comprehensive parameter sensitivity index, to obtain an optimized embedding vector; and a prompt word generating module, configured to map the optimized embedding vector back to a discrete token sequence to generate the sensitive prompt word. The sensitivity calculating module comprises:

3. The system of claim 2, wherein, a micro-sensitivity calculating unit, configured to calculate the norm of the gradient of the output of the original large language model with respect to the model parameters, to determine a micro-sensitivity; a macro-sensitivity calculating unit, configured to estimate a macro-sensitivity by sampling a preset set of parameter modification types; and an index fusion unit, configured to fuse the micro-sensitivity and the macro-sensitivity to obtain the comprehensive parameter sensitivity index. ​

Images