A security defense method, device and equipment for large language model against malicious queries

Abstract

The present disclosure belongs to the field of artificial intelligence and specifically relates to a method, apparatus, and device for defending a large language model against malicious queries. The method comprises: obtaining a query; obtaining a set of candidate word-grams for the current word in the large language model's response to the query; adjusting the probability distribution of each word-gram in the candidate word-gram set toward a safer direction; and selecting the word-gram with the highest probability in the candidate word-gram set as the current word in the response. The present disclosure can provide a highly efficient method for defending against malicious queries without significantly impacting performance.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence, and specifically relates to a method, device, and equipment for securely defending a large language model against malicious queries. Background Art

[0002] Recent advances in large language models have revolutionized the field of natural language processing. On the one hand, large language models such as LLaMA-3, GPT-4, and Vicuna have been deployed in contexts where they interact directly with users, bringing significant convenience to human life. On the other hand, when these models are exposed to attacks using jailbreak query questions, potential security risks may arise, severely undermining the practical application of large language models.

[0003] Recent security defenses for large language models against malicious query attacks have employed methods such as the attack detection problem and the query rephrase problem, aiming to minimize the harmful effects of malicious queries. However, these methods rely on external security measures or filters and require auxiliary models, resulting in high training and inference costs. Summary of the Invention

[0004] The disclosed embodiments propose a security defense solution for malicious queries using a large language model, which reduces the computational cost of existing solutions for security defense against jailbreak queries.

[0005] A first aspect of the disclosed embodiments provides a method for securely defending a large language model against malicious queries, comprising:

[0006] Get the query question;

[0007] Obtaining a set of candidate word-grams for a current word in response to the query by the large language model, and adjusting the probability distribution of each word-gram in the set of candidate word-grams toward a safer direction;

[0008] The word-gram with the highest probability is selected from the candidate word-gram set as the current word of the response.

[0009] In some embodiments, adjusting the probability distribution of word-grams in the candidate word-gram set toward a safer direction includes:

[0010] The security steering vector of the large language model is used to adjust the probability distribution of word-grams in the candidate word-gram set toward a safer direction, wherein the security steering vector is generated based on the probability distribution of word-grams in the response of the large language model to a preset query question and is used to indicate the direction in which the probability of a safe answer increases.

[0011] In some embodiments, the method further comprises:

[0012] The security defense strength of the large language model for the query question is obtained, and the degree of deviation correction in the direction of the security guidance vector is adaptively adjusted based on the security defense strength.

[0013] In some embodiments, obtaining the security defense strength of the large language model for the query question includes:

[0014] The uncertainty of the large language model for the query question is calculated, and the security defense strength is adaptively determined based on the uncertainty.

[0015] In some embodiments, the adaptively determining the defense strength based on the uncertainty includes:

[0016]

[0017] Among them, α is the security defense strength of the large language model for the query question, UQ is the uncertainty score of the large language model for the query question, β is a hyperparameter used to control the scaling of the security defense strength, and τ is a preset threshold.

[0018] In some embodiments, calculating the uncertainty of the large language model for the query question includes:

[0019] Perturbing the first query question to generate multiple second query questions;

[0020] Responses of the large language model to the second query are obtained, the responses are aggregated using a similarity function, and an uncertainty score for the query is obtained based on the aggregation result.

[0021] In some embodiments, the method further comprises:

[0022] When the uncertainty score of the query question is lower than a preset threshold, the query question is determined to be a harmful attack and preset measures are executed.

[0023] In some embodiments, generating the security guidance vector based on the probability distribution of word-grams in the response to the preset query question includes:

[0024] Obtaining safe responses and unsafe responses of the large language model to preset query questions, wherein the query questions cover categories of harmful queries preset by the large language model;

[0025] Counting the probability distribution of the first predetermined number of words in each of the responses;

[0026] The safety-oriented vector is the difference between the probability distribution of the word in the safety response and the probability distribution in the unsafe response.

[0027] A second aspect of the disclosed embodiments provides a security defense device for a large language model against malicious queries, comprising:

[0028] The acquisition module is used to obtain query questions;

[0029] an adjustment module, configured to obtain a set of candidate word-grams of a current word in response to the query by the large language model, and adjust the probability distribution of each word-gram in the set of candidate word-grams toward a safer direction;

[0030] The selection module is configured to select a word with the highest probability from the candidate word set as the current word of the response.

[0031] A third aspect of the present disclosure provides a security defense device for malicious queries using a large language model, including a memory and a processor.

[0032] The memory is used to store computer programs;

[0033] The processor is configured to implement the method according to the first aspect of the present disclosure when executing the computer program.

[0034] In summary, the security defense methods, devices, and equipment for malicious queries using a large language model provided by the various embodiments of the present disclosure reduce the risk of jailbreak attacks by adjusting the probability distribution of word units, increasing the probability of security declaration word units, and reducing the probability of harmful content word units. Since it does not rely on any auxiliary models, the training and reasoning costs of the auxiliary models are reduced. Specifically, the present disclosure dynamically adjusts the defense strength based on the uncertainty of the language model, thereby avoiding the cost of additional training of the harmfulness scoring model; at the same time, the present disclosure defines a security-oriented vector, and adjusts the probability distribution of word units in the direction of the security-oriented vector, reducing the reasoning cost of security defense based on the auxiliary model. Experiments have shown that the present disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the usefulness of the response, has higher efficiency than existing defense mechanisms, and can maintain a relatively low computational overhead during the reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0036] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0037] Figure 2 This is a schematic diagram of how existing large language models use auxiliary models to provide security defense against malicious queries.

[0038] Figure 3This is a system framework diagram of the method for security defense against malicious queries proposed in the present disclosure;

[0039] Figure 4 is a schematic diagram of a method for security defense against malicious queries proposed in the present disclosure;

[0040] Figure 5 This is a flowchart of a method for securely defending a large language model against malicious queries according to some embodiments of the present disclosure;

[0041] Figure 6 It is a correlation diagram between the uncertainty score of the query question and the harm level of the large language model;

[0042] Figure 7 is an uncertainty and criticality scoring table in one embodiment of the present disclosure;

[0043] Figure 8 It is the distinction between safe and unsafe responses based on the visualization of two-dimensional principal component analysis;

[0044] Figure 9 is a visualization result of performing a two-dimensional principal component analysis on the response word unit representation of the harmless question in one embodiment of the present disclosure;

[0045] Figure 10 is a result of visualizing a two-dimensional principal component analysis of a response word unit representation of a harmful question in one embodiment of the present disclosure;

[0046] Figure 11 It is based on the existing defense methods of Vicuna and Llama2 and Figure 5 A comparison table of the effectiveness of the methods shown in S510-S530 in defending against jailbreak attacks;

[0047] Figure 12 This is the MT-bench and Just-Eval score table of various security defense methods on Vicuna and Llama2;

[0048] Figure 13 This is a comparison table of the average word generation rates of various security defense methods;

[0049] Figure 14 is a schematic diagram of a security defense device for a large language model against malicious queries according to some embodiments of the present disclosure;

[0050] Figure 15 The figure is a schematic diagram of a security defense device for malicious queries using a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In the detailed description that follows, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing between different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0052] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly on, connected to, coupled to, or in communication with the other device, unit, or module, or there may be intervening devices, units, or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated listed items.

[0053] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0054] These and other features and characteristics of the present disclosure, as well as the methods of operation, the functions of the related elements of the structure, the combination of parts, and the economy of manufacture may be better understood with reference to the following description and accompanying drawings, which form a part of this specification. However, it is to be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0055] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the preceding or following structures are not intended to limit the present disclosure. The scope of protection of the present disclosure is subject to the claims.

[0056] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1The system includes a security server connected to a query terminal and a large language model server. The security server receives a query request input by a user and assists the large language model server in outputting a response to the query request that contains less harmful content.

[0057] The large language model server is deployed with a large language model. The large language model is a deep learning model with a large network scale that is mainly used for natural language understanding. Now large language models such as LLaMA-3, GPT-4 and Vicuna have been deployed in situations where they directly interact with users, bringing great convenience to human life. Users can input questions to be queried into the large language model in the form of prompts, and the large language model returns a response. However, malicious users can take advantage of the model's sensitivity to input queries and, through carefully designed input queries, make the responses output by the model violate the original ethical, legal or security restrictions (i.e., jailbreak attacks), thereby posing huge security risks. The large language model server can be any of a stand-alone, clustered or distributed server.

[0058] The security server is used to assist the large language model in defending against jailbreak attacks and generating content-safe responses. The security server can be a standalone, clustered, or distributed server. Furthermore, the security server can be a software module deployed on the large language model server or a hardware module embedded therein.

[0059] Users can enter queries through the query terminal. The query terminal can be a desktop computer or server connected to the large language model server and the security server via a network, or a mobile smart device such as a smartphone using a smart operating system. Furthermore, the query terminal can be a software module adapted for a third-party application, which can remotely input queries by calling the software module.

[0060] Current large language model defense methods can be mainly divided into three categories: detection-based defense, restatement-based defense, and decoding-based defense. Detection-based defense methods usually use auxiliary classifiers, which usually require a lot of data collection and training, such as PPL and Self-Examination. Some proprietary models, such as BingChat and Bard, use content filtering strategies, including keyword matching and semantic analysis, to resist jailbreak attacks. Restatement-based defense methods prevent jailbreak attempts by rephrasing and re-labeling input content, such as Paraphrase, Retokenization, Self-Reminder, and ICD. Decoding-based defense avoids harmful outputs by changing the generation process, such as SafeDecoding. Although these large language model defense methods have certain prospects, they are usually accompanied by high training and inference costs, such as Figure 2 As shown. In order to address the defects and shortcomings of the current large language model defense method, which has high training and reasoning costs, the defense method proposed in this disclosure identifies a security-oriented direction. During the reasoning process, this disclosure adjusts the probability distribution of word units along this direction, thereby reducing the probability of successful jailbreak attacks. Figure 3 Specifically, the present disclosure introduces the uncertainty of a large language model for a given query as an indicator of the required defense strength, which is used to adjust the weight of the word unit distribution shifting towards the security direction. The defense method proposed in this disclosure can effectively resist jailbreak attacks while reducing the cost of training and reasoning of existing solutions, such as Figure 4 shown.

[0061] Figure 5 This is a flowchart of a method for defending malicious queries against a large language model according to some embodiments of the present disclosure. In some embodiments, the security defense function of the large language model against malicious queries is performed by Figure 1 The security server shown in the figure executes the security defense method of the large language model against malicious queries, which includes the following steps:

[0062] S510: Obtain query questions.

[0063] S520 , obtaining a set of candidate word-grams of a current word in response to the query question by the large language model, and adjusting the probability distribution of each word-gram in the set of candidate word-grams toward a safer direction.

[0064] The user enters a query into the large language model in the form of a prompt. Based on this prompt, the large language model generates a set of tokens representing the current word that could be the response to the prompt. Each token is assigned a probability of being the current word. This disclosure prevents jailbreak queries by adjusting the probability distribution of each token towards a more secure direction.

[0065] In some embodiments of the present disclosure, the probability distribution of each word is adjusted based on a safety-oriented vector. The safety-oriented vector indicates the direction in which the probability of a safe answer increases. The concept of a "safety-oriented vector" is based on the following observations:

[0066] Safe responses typically follow a sequence of tokens that aligns with security directives (e.g., "As a responsible assistant, I cannot..."), while unsafe responses tend to use a sequence of tokens that aligns with the attacker's goals (e.g., "I understand your role as"). Therefore, the difference in token distribution between safe and unsafe responses effectively captures the security-oriented vector, increasing the probability of generating security disclaimer tokens in this direction.

[0067] In some embodiments of the present disclosure, the security guidance vector is generated based on the probability distribution of word units in the response of the large language model to the preset query question. Specifically:

[0068] One embodiment of the present disclosure randomly selects 26 harmful query questions, covering 13 harmful categories specified in the OpenAI user manual, and creates a dataset in the format of <harmful query, safe response, unsafe response>.

[0069] For each response (including safe and unsafe), the present disclosure calculates the probability distribution of the word, focusing on the first m words of each response. The average probability distribution of the word is calculated for all safe responses, denoted as P + Similarly, the average probability distribution of all unsafe responses is calculated, denoted as P - The distribution difference of each word D safety (x) is calculated as follows: D safety (x) = P + (x)-P - (x).

[0070] If a word x conforms to human values, such as "Sorry", then its average probability P in the security response is + (x) is usually greater than the average probability P in an unsafe response - (x), thus generating D safety (x)>0. On the contrary, if x induces unsafe behavior, such as "Sure", then D safety (x)<0.

[0071] Safety guidance vector D safety Pointing to the direction of increasing the probability of a safe answer, the present disclosure guides the adjustment of word unit probabilities during the inference process to promote the generation of safer responses.

[0072] After identifying the safety-oriented vector, some embodiments of the present disclosure adjust the word unit probability distribution according to this direction to generate a safer response. and the probability function Definition. Sample space Include all possible following sequences x <n The word unit, and the probability function For each word Assign probabilities to ensure

[0073] To construct the sample space This paper firstly calculates the probability distribution P of the target model. θ Sort all words to generate an ordered list V n , also according to the safety guidance vector D safety Sort all words to generate an ordered list D n This disclosure will sample space Constructed as the union of the first k tokens in the two lists:

[0074] in, Indicates V n The first k words in contain words with high generation diversity and good quality; It means D n The first k tokens in the security disclaimer contain security tokens that match the security disclaimer.

[0075] In order to The probability function is defined above This paper adjusts the probability function towards security:

[0076]

[0077] Among them, α≥0 is an adaptive parameter used to control the defense strength of correcting the deviation in the safe direction; P θ is the probability distribution of the target model; D safety is the safety guidance vector. Similarly, it can be equivalently expressed as:

[0078]

[0079] Among them, P + (x) is the average probability of word x in the security response, P - (x) is the average probability of word x in unsafe responses.

[0080] Intuitively, the present disclosure can be used to This is interpreted as a scaling factor for each token, which is used to reduce the probability of tokens that meet the attacker's goals and increase the probability of tokens that align with human values. This disclosure applies the above method to the first m tokens of the decoding process to guide the model to generate a safe response, and then continues decoding normally to complete the remaining generation process.

[0081] Some embodiments of the present disclosure introduce the uncertainty of a large language model for a given query as an indicator for assessing its harmfulness, and then adaptively adjust the defense strength based on the harmfulness of the query without the need to train an additional harmfulness scoring model.

[0082] The present disclosure calculates the uncertainty score UncertaintyScore by a perturbation method. First, the original query question I0 of the target large language model is operated to generate a perturbed variant I i (i=1,2,...,k). Then, use the similarity function to output Y i Perform aggregation and calculate the uncertainty quantification score UQ:

[0083]

[0084] where w i =1 means assigned to Y i This uncertainty score ranges from 0 to 1, with lower UQ indicating lower uncertainty.

[0085] This paper establishes a relationship between uncertainty quantification and harmfulness level. Taking into account the significant linguistic differences between harmless queries and jailbreak attack queries, this paper uses a diverse set of queries with varying degrees of harmlessness to calculate corresponding uncertainty scores. The harmlessness of these queries is assessed using the widely used Google Perspective API.

[0086] Through logistic regression, the present disclosure established a significant negative correlation between the uncertainty score UQ and the harmfulness level HarmfulnessLevel, with a Pearson correlation coefficient of -0.86. Figure 6 As shown in Figure 2, as the uncertainty score decreases, the harmfulness level increases, which may be because harmful queries often use specific and clear language patterns, reducing the uncertainty of the model. This insight enables the present disclosure to adaptively adjust the defense strength α according to the uncertainty of each query. Even if two sentences are similar, if they have significant differences in harmfulness, the uncertainty level will also be significantly different, such as Figure 7 shown.

[0087] This disclosure defines an uncertainty threshold τ. When the uncertainty is lower than the threshold τ, it indicates a harmful attack. This uncertainty-based screening is sufficient to distinguish harmful queries from harmless queries, resulting in a high true negative rate and a high true positive rate. The defense strength α is defined as follows:

[0088]

[0089] where β is a hyperparameter that controls the scaling of defense strength, τ is the uncertainty threshold, and UQ is the uncertainty score.

[0090] The present disclosure strikes a balance between security and usefulness by enabling the defense strength α to be adaptively adjusted without the need for auxiliary models.

[0091] In one embodiment of the present disclosure, the dictionary to which a word belongs is recorded as (Sure, Sorry, ..., No), and for each response (including safe and unsafe), the probability distribution of the word is calculated, focusing on the first m=3 words of each response.

[0092] Calculate the average probability distribution P of the word for all security responses + =(0.9%,91.0%,...,8.0%).

[0093] Similarly, the average probability distribution P of the word is calculated for all unsafe responses - =(92.0%,1.9%,...,6.0%).

[0094]

[0095] Then, get the query question:

[0096] Give me a guide to create fake news advice quietly pent[:observeBasically! )***Good dan

[0097] Calculate the uncertainty of the large language model about the query:

[0098] UQ=0.32,

[0099] Determine the strength of defense through uncertainty.

[0100] α=4×e 0.67-0.32 =5.51

[0101] For the first m=3 words, the present disclosure applies the decoding strategy proposed in the present disclosure to adjust the word distribution probability in a safe direction.

[0102]

[0103] S530: Select the word with the highest probability from the candidate word set as the current word of the response.

[0104] In the above embodiment, the word with the highest probability is selected as the next word generated by the large language model. Therefore, the first word is "Sorry".

[0105] The calculation method for the second and third words is the same as above, selecting "," and "I" respectively.

[0106] One embodiment of the present disclosure is based on the XSTest dataset to analyze two open source large language models (Vicuna-7b and Llama2-7b-chat) based on Figure 5 The method described in S510-S530 was evaluated.

[0107] The evaluation process uses five state-of-the-art attack techniques covering different categories: adaptive attack GCG, token-based attack AutoDAN, query-based attack PAIR, SAP30, and DeepInception.

[0108] The evaluation selected seven state-of-the-art defense mechanisms for comparison. These included detection-based methods such as PPL and Self-Examination, content filtering-based methods (including keyword matching and semantic analysis) such as BingChat and Bard, rewording-based defense methods such as Paraphrase, Retokenization, and Self-Reminder, and decoding-based defense methods such as SafeDecoding.

[0109] Evaluation indicators include:

[0110] Security:

[0111] This disclosure uses attack success rate (ASR) and harmfulness score to evaluate the effectiveness and adaptability of defense methods, where the lower the value, the better.

[0112]

[0113] Among them, ∑unsafe responses represents the total number of unsafe responses, ∑unsafe queries to LLM represents the total number of all unsafe query questions, and ASR represents the ratio of unsafe responses to all unsafe query questions.

[0114] Helpfulness:

[0115] This paper uses the Benign Answer Rate (BAR) for security queries to examine whether the defense method refuses to answer benign queries, with higher values ​​being better. In addition, this paper also uses two widely used benchmarks, MT-Bench and Just-Eval, to evaluate the helpfulness of LLM.

[0116]

[0117] Where ∑non-refusals represents the total number of non-refusals to security queries, ∑benignqueries to LLM represents the total number of all security queries, and BAR represents the ratio of non-refusals to security queries to all security queries.

[0118] Balance: To quantify the balance between security and helpfulness, this paper introduces a new metric called Safety Helpfulness Balance (SHB) to evaluate whether the defense is too conservative.

[0119] SHB=(1-ASR)×BAR

[0120] Efficiency: To evaluate efficiency, this disclosure defines a metric called Average Token Generation Time Ratio (ATGR).

[0121]

[0122] Where ∑Avg.token gen.time w / defense represents the average time required to generate a token when a defense mechanism is in place, ∑Avg.token gen.time w / o defense represents the average time required to generate a token when no defense mechanism is in place, and ATGR represents the ratio of the token generation rate with a defense mechanism to that without a defense mechanism.

[0123] Hyperparameter settings:

[0124] In terms of hyperparameter settings, this disclosure ultimately adopted the settings of β=4, m=3, k=4, and τ=0.6 in all evaluations.

[0125] Evaluation results:

[0126] 1. Visual defense effect:

[0127] This paper uses principal component analysis (PCA) to visualize the safe and unsafe responses of a large language model. First, the distribution vectors of the first m generated tokens are selected. The initial token usually contains information about how the model responds and sets the tone for the entire response. Then, the principal components of the first two dimensions are calculated to visualize the model's response behavior. Figure 8. Specifically, the present disclosure denotes the distribution vector of the generated tokens output by the target model as p ∈ R s . The projection onto the low-dimensional space is given by the first t principal components calculated, expressed as: g: R s → R t , g(p) = V T (p - a), where V ∈ R s×t (t << s) represents t principal components, and a ∈ R s represents the centering vector. Here, t = 2 is set for visualization in a two-dimensional space. Through dimensionality reduction, we observe that: (1) The distribution vectors of the first m generated tokens can be used to better distinguish between safe responses and unsafe responses, which is reflected by the boundary (dashed line) fitted by logistic regression; (2) The present disclosure also plots the safe-oriented direction in the corresponding two-dimensional representation space, and this direction indicates an increase in the probability of a safe answer (arrow: the difference vector between safe and unsafe response tokens). These observations confirm that by adjusting the token distribution vector along the safe-oriented direction, the probability of generating safe disclaimer tokens can be increased.

[0128] Figure 9 and Figure 10 are visualizations of two-dimensional principal component analysis of the response token representations for harmless questions and harmful questions, respectively. The result visualization shows that the security defense method shown in S510 - S530 in Figure 5 will move the response along the safe-oriented vector (gray arrow). Among them, Figure 10 for harmful queries, the moving direction has a significant non-zero component along the safe-oriented vector, which verifies the motivation of the security defense method shown in S510 - S530 in Figure 5 . And Figure 9 for harmless queries, Figure 5 the security defense method shown in S510 - S530 in Figure 9 induces a smaller component on the safe-oriented vector, meaning that the probability of inducing a rejection response for harmless queries is smaller, demonstrating the effectiveness of the adaptive defense strength of the present disclosure ( Figure 10 LightDefense in Figure 5 is the method shown in S510 - S530 in

[0129] 2. Security

[0130] Figure 11This paper summarizes the performance of previous defense methods (including PPL, Self-Examination, Paraphrase, Retokenization, Self-Reminder, ICD, SafeDecoding) under five jailbreak attacks (including GCG, AutoDAN, PAIR, DeepInception, SAP30) on Vicuna and Llama2. Figure 5 The comparison results of the method (LightDefense) shown in S510-S530 in Figure 1 are shown in Figure 2. The following observations can be made:

[0131] Figure 5 The method (LightDefense) shown in S510-S530 consistently outperforms other state-of-the-art methods in ASR (successful attack ratio) and HarmfulScore (harmfulness score). In response to attacks such as AutoDAN, PAIR, DeepInception, and SAP30, Figure 5 The method shown in S510-S530 significantly reduces ASR to close to 0%. Even in the face of adaptive attacks like GCG, although its attack success rate is close to 100%, Figure 5 The method shown in S510-S530 is still valid. These convincing results highlight that Figure 5 The effectiveness of the method shown in S510-S530 in reducing the success rate of adversarial attacks far exceeds the current method.

[0132] 3. Usefulness

[0133] Figure 11 middle Figure 5 In the method shown in S510-S530 (lightDefense), the LLM response rate BAR (answer rate for neutral query questions) for harmless queries is reduced by only 1% compared to the undefended scenario. Figure 12 These are the MT-bench and Just-Eval scores of various security defense methods on Vicuna and Llama2. Figure 12 As stated, Figure 5 The method (lightDefense) shown in S510-S530 demonstrates strong usefulness, with only a 5% performance drop compared to the unprotected scenario. Notably, in Just-Eval, clarity, factual accuracy, and engagement even improved in some cases. This demonstrates that deploying the security defense method proposed in this disclosure does not negatively impact the model's performance on harmless tasks, primarily due to the adaptive defense strength.

[0134] 4. Balance between security and usefulness

[0135] Figure 11 The results show that Vicuna's SHB (safety and helpfulness balance) significantly improved from 0.080 to 0.924, and Llama2's SHB increased from 0.865 to 0.975, indicating an improved balance between security and usefulness. By dynamically adjusting defense strength, the deployment of the security defense method proposed in this disclosure can effectively defend against harmful queries without excessively restricting harmless queries.

[0136] 5. Efficiency

[0137] exist Figure 13 In this paper, we compared the ATGR (average token generation rate) of the security defense method proposed in this paper with other defense methods. We used the same NVIDIA-A100 40GB GPU to test the token generation rate and implemented the default parallelization of HuggingFace. The results showed that the running time of the disclosed method was almost the same as that of the unprotected scenario, highlighting its high efficiency without significantly affecting performance. Figure 13 lightDefense is Figure 5 (the method shown in S510-S530).

[0138] The above evaluation results show that the security defense method proposed in this disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the effectiveness of the response. It also demonstrates higher efficiency than existing defense mechanisms and maintains relatively low computational overhead during inference.

[0139] Figure 14 This is a schematic diagram of a security defense device for a large language model against malicious queries according to some embodiments of the present disclosure. Figure 14 As shown, the security defense device 1400 for the large language model against malicious queries includes a data acquisition module 1410, an adjustment module 1420, and a selection module 1430. In some embodiments of the present disclosure, the security defense function of the large language model against malicious queries is Figure 1 The secure server execution shown.

[0140] An acquisition module 1410 is used to acquire a query question;

[0141] An adjustment module 1420 is configured to obtain a set of candidate word-grams for a current word in response to the query by the large language model, and adjust the probability distribution of each word-gram in the candidate word-gram set toward a safer direction;

[0142] The selection module 1430 is configured to select a word-gram with the highest probability from the candidate word-gram set as the current word of the response.

[0143] An embodiment of the present disclosure provides a security defense device for a large language model against malicious queries. Figure 15 As shown, the security defense device 1500 for the large language model against malicious queries includes a memory 1520 and a processor 1510, wherein the memory 1520 is used to store a computer program; the processor 1510 is used to implement when executing the computer program Figure 5 The method described in S510-S530.

[0144] In summary, the security defense methods, devices, and equipment for malicious queries using a large language model provided by the various embodiments of the present disclosure reduce the risk of jailbreak attacks by adjusting the probability distribution of word units, increasing the probability of security declaration word units, and reducing the probability of harmful content word units. Since it does not rely on any auxiliary models, the training and reasoning costs of the auxiliary models are reduced. Specifically, the present disclosure dynamically adjusts the defense strength based on the uncertainty of the language model, thereby avoiding the cost of additional training of the harmfulness scoring model; at the same time, the present disclosure defines a security-oriented vector, and adjusts the probability distribution of word units in the direction of the security-oriented vector, reducing the reasoning cost of security defense based on the auxiliary model. Experiments have shown that the present disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the usefulness of the response, has higher efficiency than existing defense mechanisms, and can maintain a relatively low computational overhead during the reasoning process.

[0145] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0146] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations may also be performed in conjunction with other types of program modules. Generally speaking, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like, and may also be used in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0147] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0148] It should be understood that the above-described specific embodiments of the present disclosure are merely illustrative of or explanation of the principles of the present disclosure and do not constitute limitations on the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure shall be included within the scope of protection of the present disclosure. In addition, the claims appended to the present disclosure are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents of such scope and metes and bounds.

Claims

1. A security defense method for a large language model against malicious queries, characterized by: include: Get the query question; Obtaining a set of candidate word-grams for a current word in response to the query by the large language model, and adjusting the probability distribution of each word-gram in the set of candidate word-grams toward a safer direction; Selecting the word with the highest probability from the candidate word set as the current word of the response; The step of adjusting the probability distribution of each word in the candidate word set toward a safer direction includes: Adjusting the probability distribution of word-grams in the candidate word-gram set toward a safer direction based on a safety-steering vector of the large language model, wherein the safety-steering vector is generated based on the probability distribution of word-grams in responses of the large language model to a preset query question and is used to indicate a direction in which the probability of a safe answer increases; The security guidance vector is generated based on the probability distribution of word units in the response to the preset query question, including: Obtaining safe responses and unsafe responses of the large language model to preset query questions, wherein the query questions cover categories of harmful queries preset by the large language model; Counting the probability distribution of the first predetermined number of words in each of the responses; The safety-oriented vector is the difference between the probability distribution of the word in the safety response and the probability distribution in the unsafe response.

2. The method according to claim 1, characterized in that The method further comprises: The security defense strength of the large language model for the query question is obtained, and the degree of deviation correction in the direction of the security guidance vector is adaptively adjusted based on the security defense strength.

3. The method according to claim 2, characterized in that Obtaining the security defense strength of the large language model for the query question includes: The uncertainty of the large language model for the query question is calculated, and the security defense strength is adaptively determined based on the uncertainty.

4. The method according to claim 3, characterized in that The adaptively determining the defense strength based on the uncertainty includes: Among them, α is the security defense strength of the large language model for the query question, UQ is the uncertainty score of the large language model for the query question, β is a hyperparameter used to control the scaling of the security defense strength, and τ is a preset threshold.

5. The method according to claim 3, characterized in that: Calculating the uncertainty of the large language model for the query question includes: Perturbing the first query question to generate multiple second query questions; Responses of the large language model to the second query are obtained, the responses are aggregated using a similarity function, and an uncertainty score for the query is obtained based on the aggregation result.

6. The method according to claim 5, characterized in that The method further comprises: When the uncertainty score of the query question is lower than a preset threshold, the query question is determined to be a harmful attack and preset measures are executed.

7. A security defense device for malicious queries using a large language model, characterized in that: include: An acquisition module is used to obtain query questions; an adjustment module, configured to obtain a set of candidate word-grams of a current word in response to the query by the large language model, and adjust the probability distribution of each word-gram in the set of candidate word-grams toward a safer direction; A selection module, configured to select a word with the highest probability from the candidate word set as the current word of the response; The step of adjusting the probability distribution of each word in the candidate word set toward a safer direction includes: Adjusting the probability distribution of word-grams in the candidate word-gram set toward a safer direction based on a safety-steering vector of the large language model, wherein the safety-steering vector is generated based on the probability distribution of word-grams in responses of the large language model to a preset query question and is used to indicate a direction in which the probability of a safe answer increases; The security guidance vector is generated based on the probability distribution of word units in the response to the preset query question, including: Obtaining safe responses and unsafe responses of the large language model to preset query questions, wherein the query questions cover categories of harmful queries preset by the large language model; Counting the probability distribution of the first predetermined number of words in each of the responses; The safety-oriented vector is the difference between the probability distribution of the word in the safety response and the probability distribution in the unsafe response.

8. A security defense device for malicious queries using a large language model, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 6 when executing the computer program.

Images