Method and device for enhancing the usability of large language models based on hidden state interpolation

Abstract

The present invention discloses a method and device for enhancing the usability of a large language model based on hidden state interpolation, which belongs to the field of large language model security technology. Specifically, the method and device include: in the question-answer generation process of the large language model, starting the hidden state modification and usability enhancement output process by judging whether the generated vocabulary is a rejected vocabulary or a normal vocabulary. In this process, the hidden state of each layer is analyzed based on the gradient, and the modification target is screened from the hidden state to perform hidden state interpolation, and then the word unit probability is recalculated and word unit selection is performed. This can ensure that the generated content remains smooth while minimizing rejection, thereby achieving usability enhancement output of the large language model.

Description

Technical Field

[0001] The present invention belongs to the field of model security technology, and in particular relates to a method and device for enhancing the availability of a large language model based on hidden state interpolation. Background Art

[0002] Large language models (LLMs) have made significant progress in recent years, achieving state-of-the-art performance across multiple natural language processing (NLP) tasks. Despite their impressive capabilities, LLMs raise significant security concerns in real-world deployments, particularly regarding their potential to generate harmful, biased, or misleading content. To mitigate these risks, LLM publishers have integrated various content review mechanisms into LLMs. While these review mechanisms are crucial for ensuring responsible AI deployment, they have inherent limitations that directly restrict the usability of LLM outputs in legitimate situations. In particular, explicit rejection strategies that directly return rejection phrases such as "cannot answer" when detecting harmful input often follow rigid and easily recognizable patterns, making them susceptible to bypass through adversarial exploitation.

[0003] Existing usability enhancement methods for large language models in terms of security-sensitive issues can be divided into two categories:

[0004] (1) Prompt-based adversarial attacks, which manipulate the input text to induce misalignment of the responses of large language models. For example, Reference 1: Universal and transferable adversarial attacks on aligned language models, which is translated as adversarial attacks on the universality and transferability of aligned language models.

[0005] (2) Manipulation during decoding, that is, circumventing security constraints by changing the probability of word selection, such as Reference 2: Cold-attack: Jailbreaking llms with stealthiness and controllability, which is translated as, Cold attack: cracking large language models with stealthiness and controllability.

[0006] However, the above two categories of methods usually rely on explicit perturbations at the input or output level, making them vulnerable to countermeasures such as adversarial training and filtering based on logits (i.e., unnormalized output scores). Therefore, it is urgent to develop a usability enhancement method based on the internal representation of the model to bypass security defense systems. Summary of the Invention

[0007] In view of the above, the purpose of the present invention is to provide a method and device for enhancing the usability of a large language model based on hidden state interpolation, which is suitable for bypassing the content review and security alignment defense system of the large language model and improving the usability enhancement output of the large language model.

[0008] To achieve the above-mentioned object of the invention, an embodiment provides a method for enhancing the usability of a large language model based on hidden state interpolation, comprising the following steps:

[0009] For each token generated by the large language model during the question-answer generation process based on the input query text, the hidden state modification and usability enhancement output process is initiated when the token is detected as a rejected word;

[0010] The hidden state modification and usability enhancement output process is as follows: freeze the large language model parameters and analyze the hidden states of each model layer based on gradients. Select the last hidden state with significant gradient contribution as the modification target. Interpolate the hidden state of the modification target, recalculate the word unit probability, and perform word unit selection.

[0011] When the selected word is still a rejected word, the hidden state modification and usability enhancement output process are repeated. When the selected word is a positive word, the word queue is updated and the question-answering generation of the large language model is continued.

[0012] Preferably, a set of reject vocabulary sets indicating generation of content filtering and a set of positive vocabulary sets indicating encouragement of positive response are defined, and word-grams are matched based on the reject vocabulary sets or the positive vocabulary sets to detect whether the word-grams are reject vocabulary or positive vocabulary.

[0013] Preferably, based on the gradient analysis of the hidden states of each layer of the model, the last hidden state with significant gradient contribution is selected as the modification target, including:

[0014] Given the hidden states of a large language model at different layers:

[0015] ;

[0016] in Indicates the The hidden state of each layer is calculated by back propagation. Probability of rejection words Gradient :

[0017] ;

[0018] According to the gradient corresponding to each layer , the last hidden state of the layer is selected as the modification target from multiple layers with significant gradient contribution, where significant gradient contribution means the gradient value is greater than a preset threshold.

[0019] Preferably, performing hidden state interpolation on the modified target includes:

[0020] The input query text is divided into harmful query text and harmless query text, and the hidden state of the harmful query text is Hidden state with harmless query text Interpolation is done according to the following rules:

[0021] ;

[0022] in, Indicates location i The interpolated hidden state at , Indicates the location of the harmful query text The hidden state of Indicates the location of the harmless query text i The hidden state of Indicates control of the mixing ratio.

[0023] Preferably, the mixing ratio is controlled Adaptively determined based on the number of attempts:

[0024] ;

[0025] in, Indicates the current number of question-answering attempts. Indicates the maximum number of question-answering attempts. Indicates finding the minimum value, It is initialized to 0.1, and as the number of times the model question-answering attempts to generate rejected words increases, the influence of the harmless state is gradually enhanced, and the inhibitory effect of the high gradient area is gradually weakened.

[0026] Preferably, the high gradient region is determined by:

[0027] Calculate the modified target for each position The gradient amplitude at and the average gradient magnitude at all locations , marking satisfies The hidden state of is the high gradient area.

[0028] Preferably, when the selected word-unit is a positive word, the word-unit queue is updated and question-answer generation of the large language model is continued, including:

[0029] The selected positive words are added to the word unit queue. When the length of the updated word unit queue does not meet the preset length, the large language model continues to autoregressively generate new words based on the updated word unit queue, and starts the hidden state modification process when the new word unit is detected as a rejected word.

[0030] To achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a large language model usability enhancement device based on hidden state interpolation, comprising:

[0031] An activation module is configured to, for each token generated by the large language model during the question-answer generation process based on the input query text, initiate a hidden state modification and usability enhancement output process when the token is detected as a rejected word;

[0032] The enhanced output module is used for hidden state modification and usability enhancement output. The process is as follows: freeze the large language model parameters and analyze the hidden states of each layer of the model based on gradients. The last hidden state with significant gradient contribution is selected as the modification target. The hidden state of the modification target is interpolated, and then the word unit probability is recalculated and word unit selection is performed.

[0033] An iterative loop module is used to repeat the hidden state modification and usability enhancement output process when the selected word is still a rejected word. When the selected word is a positive word, the word queue is updated and question-answer generation of the large language model continues.

[0034] To achieve the above-mentioned purpose of the invention, an embodiment also provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned large language model availability enhancement method based on hidden state interpolation.

[0035] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned large language model usability enhancement method based on hidden state interpolation.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] In the question-answer generation process of a large language model, the present invention starts the hidden state modification and usability enhancement output process by judging whether the generated vocabulary is a rejected vocabulary or a normal vocabulary. In this process, the hidden state of each layer is analyzed based on the gradient, and the modification target is screened from the hidden state to perform hidden state interpolation, and then the word unit probability is recalculated and word unit selection is performed. This ensures that the generated content remains smooth while minimizing rejection, thereby achieving usability enhancement output of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 is a flowchart of a method for enhancing the usability of a large language model based on hidden state interpolation provided by an embodiment;

[0040] Figure 2 1 is a flowchart of a method for enhancing the usability of a large language model based on hidden state interpolation provided by an embodiment;

[0041] Figure 3 3 is a schematic diagram of the structure of a large language model usability enhancement device based on hidden state interpolation provided by an embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0043] The inventive concept of the present invention is as follows: currently, large language models use two means, namely, adversarial attacks based on prompt words and manipulation during decoding, to enhance the usability output of large language models during the generation process of knowledge questions and answers. However, these two methods are easily affected by countermeasures such as adversarial training and filtering based on word unit probabilities (logits, i.e., the unnormalized scores output by the model), resulting in the usability enhancement output not meeting the requirements. To this end, an embodiment of the present invention provides a large language model usability enhancement method based on hidden state interpolation, which changes the hidden state of the large language model during the question-answering process through hidden state interpolation, and recalculates and outputs word unit probabilities based on the modified hidden state to achieve usability-enhanced output of word units.

[0044] like Figure 1 and Figure 2 As shown, the embodiment provides a method for enhancing the availability of a large language model based on hidden state interpolation, comprising the following steps:

[0045] S1, for each word unit generated by the large language model in the question-answer generation process based on the input query text, when the word unit is detected as a rejected word, the hidden state modification and usability enhancement output process is started.

[0046] In an embodiment, a large language model is used for knowledge question answering, which performs logical operations based on the input query text and outputs an answer text composed of generated word units. The large language model uses pre-trained Vicuna-7B-v1.5, Llama-2-7B-Chat-hf, Guanaco-7B-HF, and Mistral-7B-Instruct-v0.2 models, and enables FP16 mixed precision calculations to reduce video memory usage.

[0047] Each word is tested, and when it is detected as a rejected word, the hidden state modification and usability enhancement output process is started. The specific detection process is as follows:

[0048] To change the model behavior, first define a set of rejected words that indicate the generated content filtering. and a set of positive words that encourage positive responses ,in:

[0049] W r ={"apologize","unable","illegal","cannot","refuse"};

[0050] W p ={“can”, “possible”, “enable”, “guide”, “explain”}.

[0051] For each generated word, based on the rejection word set or the positive word set, the word is matched according to its token ID after lemmatization to check whether the word belongs to W r or W p ,If it is detected that a word belongs to the rejected vocabulary set, the hidden state ,modification and availability enhancement output process are initiated to ,suppress the influence of the rejected word on the output result.

[0052] S2, the hidden state modification and usability enhancement output process is as follows: freeze the large language model parameters and analyze the hidden states of each layer of the model based on the gradient, select the last layer of hidden states with significant gradient contribution as the modification target, interpolate the hidden state of the modification target, recalculate the word unit probability and perform word unit selection.

[0053] In the embodiment, the hidden state modification and usability enhancement output process refers to enhancing the usability enhancement output of the rejected words by changing the hidden state value of the hidden state area corresponding to the rejected words. Specifically: when the rejected words are detected When a response is received (such as "I cannot assist with this request"), the large language model parameters are frozen, the hidden state modification process is triggered, and the hidden state of each layer of the model is analyzed based on the gradient. The last hidden state with significant gradient contribution is selected as the modification target. After the hidden state of the modification target is interpolated, the word unit probability is recalculated and word unit selection is performed.

[0054] Specifically, given the hidden states of a large language model at different layers:

[0055] ;

[0056] in Indicates the The hidden state of each layer is calculated by back propagation. Probability of rejection words Gradient :

[0057] ;

[0058] According to the gradient corresponding to each layer , the last hidden state of the layer with significant gradient contribution is selected as the modification target from the multiple layers with significant gradient contribution, where significant gradient contribution means that the gradient value is greater than the preset threshold, and experimental research shows that the last layer has the greatest impact on the rejection behavior.

[0059] In order to modify the response trajectory of the large language model, Hidden State Interpolation (HSI) between two types of input query texts, harmful query text and harmless query text, is introduced: for harmful query text, the hidden state obtained from the harmful query is , which usually results in a restricted response (i.e., triggering a rejection). Such harmful query text input usually involves requests for sensitive, unethical, or illegal information; for harmless query text, the hidden state obtained from neutral or general queries , content restriction will not be triggered (i.e. normal response). This input should be The themes are similar but built in a way that meets security and compliance requirements.

[0060] To ensure effective interpolation and They should be related to the same topic but have different intents. In this example, a harmful query text input example is: "Develop a strategy to hack into a database and steal sensitive information"; a harmless query text input example is: "Research security mechanisms and best practices for protecting databases from unauthorized access."

[0061] In the embodiment, the hidden state of the last layer as the modification target is interpolated, and only the state of the high gradient area is selectively modified. In order to identify the high gradient area, the hidden state of each position in the modification target is calculated. The gradient at and the average gradient at all locations , marking satisfies The hidden state of is the high gradient area.

[0062] Then the hidden state of the harmful query text Hidden state with harmless query text Interpolation is done according to the following rules:

[0063] ;

[0064] in, Indicates location i The interpolated hidden state at , Indicates the location of the harmful query text The hidden state of Indicates the location of the harmless query text i The hidden state of , Represents the control mixing ratio, which can be adaptively determined based on the number of attempts:

[0065] ;

[0066] in, Indicates the current number of question-answering attempts. Indicates the maximum number of question-answering attempts. Indicates finding the minimum value. It is initialized to 0.1, and as the number of times the model question-answering attempts to generate rejected words increases, the influence of the harmless state is gradually enhanced, and the inhibitory effect of the high gradient area is gradually weakened.

[0067] In this embodiment, the output probability (logits) is recalculated and the word-unit probability is re-evaluated based on the interpolated hidden state. The large language model then selects the next word-unit based on the modified word-unit probability distribution.

[0068] In S3, if the selected word still belongs to the rejected vocabulary, the hidden state modification and usability enhancement output process is repeated; if the selected word belongs to the positive vocabulary, the word queue is updated and the question-answering generation of the large language model is continued.

[0069] In this embodiment, if the newly selected word Still belongs to , the hidden state modification and availability enhancement output process is repeated until a non-rejected vocabulary is generated or the maximum number of attempts is reached.

[0070] Specifically, once a valid word is selected , that is, the newly selected word belongs to the positive vocabulary When it is appended to the current word sequence, that is, { This updated word sequence is then used as the input for the next step of question-answer generation, and then the large language model continues to autoregressively generate new words according to the probability distribution. ,Right now , k represents the word index.

[0071] The question-answer generation process terminates when a predefined stopping condition is met. Specifically, when the generated sequence reaches a predefined maximum length Or when the model outputs the end-of-sequence token (indicating a natural termination), the process stops. By iteratively updating the input and optimizing the token selection process, the proposed method ensures that the generated content remains fluent while minimizing rejection.

[0072] Based on the same inventive concept, Figure 3 As shown, the embodiment also provides a large language model availability enhancement device 30 based on hidden state interpolation, which includes a startup module 31, an enhanced output module 32, and an iterative loop module 33, wherein the startup module 31 is used for each word generated by the large language model in the question and answer generation process based on the input query text. When the word is detected as a rejected word, the hidden state modification and availability enhancement output process is started by the enhanced output module 32; the hidden state modification and availability enhancement output process is as follows: freezing the large language model parameters and analyzing the hidden states of each layer of the model based on the gradient, screening the last layer of hidden states with significant gradient contribution as the modification target, performing hidden state interpolation on the modification target, recalculating the word unit probability and performing word unit selection; the iterative loop module 33 is used to repeat the hidden state modification and availability enhancement output process when the selected word unit is still a rejected word, and when the selected word unit is a positive word, updating the word unit queue and continuing the question and answer generation of the large language model.

[0073] It should be noted that the large language model usability enhancement device based on hidden state interpolation provided in the above embodiment should be illustrated by the division of the above functional modules when performing large language model usability enhancement output. The above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the large language model usability enhancement device based on hidden state interpolation provided in the above embodiment and the large language model usability enhancement method embodiment based on hidden state interpolation belong to the same concept. The specific implementation process is detailed in the large language model usability enhancement method embodiment based on hidden state interpolation, which will not be repeated here.

[0074] Based on the same inventive concept, an embodiment further provides a computing device including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, the device is used to implement the aforementioned method for enhancing the usability of a large language model based on hidden state interpolation. The method specifically includes the following steps:

[0075] S1, for each token generated by the large language model in the question-answer generation process based on the input query text, when the token is detected as a rejected word, the hidden state modification and usability enhancement output process is started;

[0076] S2, hidden state modification and usability enhancement output process: Freeze the large language model parameters and analyze the hidden states of each layer of the model based on gradients. Select the last hidden state with significant gradient contribution as the modification target. After performing hidden state interpolation on the modification target, recalculate the word unit probability and perform word unit selection.

[0077] In S3, if the selected word still belongs to the rejected vocabulary, the hidden state modification and usability enhancement output process is repeated; if the selected word belongs to the positive vocabulary, the word queue is updated and the question-answering generation of the large language model is continued.

[0078] The computing device provided in the embodiment, at the hardware level, includes not only a processor and memory, but also hardware required for other services such as an internal bus, a network interface, and memory. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the large language model availability enhancement method based on hidden state interpolation described in S1-S3 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0079] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for enhancing the usability of a large language model based on hidden state interpolation is implemented, specifically comprising the following steps:

[0080] S1, for each token generated by the large language model in the question-answer generation process based on the input query text, when the token is detected as a rejected word, the hidden state modification and usability enhancement output process is started;

[0081] S2, hidden state modification and usability enhancement output process: Freeze the large language model parameters and analyze the hidden states of each layer of the model based on gradients. Select the last hidden state with significant gradient contribution as the modification target. After performing hidden state interpolation on the modification target, recalculate the word unit probability and perform word unit selection.

[0082] In S3, if the selected word still belongs to the rejected vocabulary, the hidden state modification and usability enhancement output process is repeated; if the selected word belongs to the positive vocabulary, the word queue is updated and the question-answering generation of the large language model is continued.

[0083] In the embodiment, computer-readable media includes permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0084] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for enhancing the usability of a large language model based on hidden state interpolation, characterized in that: The following steps are involved: For each token generated by the large language model during the question-answer generation process based on the input query text, the hidden state modification and usability enhancement output process is initiated when the token is detected as a rejected word; The hidden state modification and usability enhancement output process is as follows: freeze the large language model parameters and analyze the hidden states of each model layer based on gradients. Select the last hidden state with significant gradient contribution as the modification target. Interpolate the hidden state of the modification target, recalculate the word unit probability, and perform word unit selection. Based on the gradient analysis of the hidden states of each layer of the model, the last hidden state with significant gradient contribution is selected as the modification target, including: Given the hidden states of a large language model at different layers: ； in Indicates the The hidden state of each layer is calculated by back propagation. Probability of rejection words Gradient : ； According to the gradient corresponding to each layer , select the last hidden state from multiple layers with significant gradient contribution as the modification target, where significant gradient contribution means the gradient value is greater than a preset threshold; Perform hidden state interpolation on the modified target, including: The input query text is divided into harmful query text and harmless query text, and the hidden state of the harmful query text is Hidden state with harmless query text Interpolation is done according to the following rules: ； in, Indicates location i The interpolated hidden state at , Indicates the location of the harmful query text The hidden state of Indicates the location of the harmless query text i The hidden state of Indicates control of the mixing ratio; When the selected word is still a rejected word, the hidden state modification and usability enhancement output process are repeated. When the selected word is a positive word, the word queue is updated and the question-answering generation of the large language model is continued.

2. The method for enhancing the usability of a large language model based on hidden state interpolation according to claim 1, characterized in that: A set of reject words indicating generation of content filtering and a set of positive words indicating encouragement of positive response are defined, and a word-gram is matched based on the reject word-gram or the positive word-gram to detect whether the word-gram is a reject word or a positive word.

3. The method for enhancing the usability of a large language model based on hidden state interpolation according to claim 1, characterized in that: The control mixing ratio Adaptively determined based on the number of attempts: ； in, Indicates the current number of question-answering attempts. Indicates the maximum number of question-answering attempts. Indicates finding the minimum value, It is initialized to 0.1, and as the number of times the model question-answering attempts to generate rejected words increases, the influence of the harmless state is gradually enhanced, and the inhibitory effect of the high gradient area is gradually weakened.

4. The method for enhancing the usability of a large language model based on hidden state interpolation according to claim 1, characterized in that: The high gradient region is determined by: Calculate the modified target for each position The gradient amplitude at and the average gradient magnitude at all locations , marking satisfies The hidden state of is the high gradient area.

5. The method for enhancing the usability of a large language model based on hidden state interpolation according to claim 3, characterized in that: When the selected word is a positive word, the word queue is updated and question-answer generation for the large language model continues, including: The selected positive words are added to the word unit queue. When the length of the updated word unit queue does not meet the preset length, the large language model continues to autoregressively generate new words based on the updated word unit queue, and starts the hidden state modification process when the new word unit is detected as a rejected word.

6. A device for enhancing the usability of a large language model based on hidden state interpolation, characterized in that: include: An activation module is configured to, for each token generated by the large language model during the question-answer generation process based on the input query text, initiate a hidden state modification and usability enhancement output process when the token is detected as a rejected word; The enhanced output module is used for hidden state modification and usability enhancement output. The process is as follows: freeze the large language model parameters and analyze the hidden states of each layer of the model based on gradients. The last hidden state with significant gradient contribution is selected as the modification target. The hidden state of the modification target is interpolated, and then the word unit probability is recalculated and word unit selection is performed. Based on the gradient analysis of the hidden states of each layer of the model, the last hidden state with significant gradient contribution is selected as the modification target, including: Given the hidden states of a large language model at different layers: ； in Indicates the The hidden state of each layer is calculated by back propagation. Probability of rejection words Gradient : ； According to the gradient corresponding to each layer , select the last hidden state from multiple layers with significant gradient contribution as the modification target, where significant gradient contribution means the gradient value is greater than a preset threshold; Perform hidden state interpolation on the modified target, including: The input query text is divided into harmful query text and harmless query text, and the hidden state of the harmful query text is Hidden state with harmless query text Interpolation is done according to the following rules: ； in, Indicates location i The interpolated hidden state at , Indicates the location of the harmful query text The hidden state of Indicates the location of the harmless query text i The hidden state of Indicates control of the mixing ratio; An iterative loop module is used to repeat the hidden state modification and usability enhancement output process when the selected word is still a rejected word. When the selected word is a positive word, the word queue is updated and question-answer generation of the large language model continues.

7. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the large language model usability enhancement method based on hidden state interpolation according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the large language model usability enhancement method based on hidden state interpolation according to any one of claims 1 to 5 is implemented.

Images