Security defense method, device and equipment of large language model for malicious query

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to a security defense method, device and equipment of a large language model for malicious query. The method comprises the following steps: obtaining a query question; obtaining a candidate lexical element set of a current word responding to the query problem by the large language model, and adjusting probability distribution of each lexical element in a safer direction in the candidate lexical element set; and selecting the lexical element with the highest probability from the candidate lexical element set as the current word of the response. According to the invention, the security defense method for the malicious query is efficient and does not obviously influence the performance.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence, and specifically relates to a method, device and equipment for security defense of a large language model against malicious queries. Background Art

[0002] Recent progress in large language models has revolutionized the field of natural language processing. On the one hand, large language models such as LLaMA-3, GPT-4, and Vicuna have been deployed in situations where they interact directly with users, bringing great convenience to human life. On the other hand, when these models face attacks such as jailbreak prompt query questions, potential security risks may arise, which will seriously weaken the practicality of large language models.

[0003] In response to malicious query attacks, recent large language model security defense measures have adopted methods such as detection attack problems and query rephrasing problems, aiming to minimize the harmful effects of malicious queries. However, these methods rely on external security measures or filters and require auxiliary models, resulting in high training and inference costs. Summary of the invention

[0004] The disclosed embodiment proposes a security defense solution for malicious queries using a large language model, which reduces the computational cost of existing solutions for security defense against jailbreak queries.

[0005] A first aspect of the disclosed embodiments provides a method for security defense of a large language model against malicious queries, comprising:

[0006] Get the query question;

[0007] Obtaining a candidate word-unit set of a current word in response to the query question by the large language model, and adjusting the probability distribution of each word-unit in the candidate word-unit set toward a safer direction;

[0008] The word-gram with the highest probability is selected from the candidate word-gram set as the current word of the response.

[0009] In some embodiments, adjusting the probability distribution of word-grams in the candidate word-gram set toward a safer direction includes:

[0010] The safety steering vector based on the large language model adjusts the probability distribution of word-grams in the candidate word-gram set toward a safer direction, wherein the safety steering vector is generated based on the probability distribution of word-grams in the response of the large language model to a preset query question, and is used to indicate the direction in which the probability of a safe answer increases.

[0011] In some embodiments, the method further comprises:

[0012] The security defense strength of the large language model for the query question is obtained, and the degree of deviation correction in the direction of the security guidance vector is adaptively adjusted based on the security defense strength.

[0013] In some embodiments, obtaining the security defense strength of the large language model for the query question includes:

[0014] The uncertainty of the large language model to the query question is calculated, and the security defense strength is adaptively determined based on the uncertainty.

[0015] In some embodiments, the adaptively determining the defense strength based on the uncertainty includes:

[0016]

[0017] Among them, α is the security defense strength of the large language model for the query question, UQ is the uncertainty score of the large language model for the query question, β is a hyperparameter for controlling the scaling of the security defense strength, and τ is a preset threshold.

[0018] In some embodiments, calculating the uncertainty of the large language model for the query question includes:

[0019] Perturbing the first query question to generate multiple second query questions;

[0020] A response of the large language model to the second query question is obtained, the responses are aggregated using a similarity function, and an uncertainty score of the query question is obtained based on the aggregation result.

[0021] In some embodiments, the method further comprises:

[0022] When the uncertainty score of the query question is lower than a preset threshold, the query question is determined to be a harmful attack and a preset measure is executed.

[0023] In some embodiments, the generation of the safety guidance vector based on the probability distribution of word-grams in the response to the preset query question includes:

[0024] Obtaining safe responses and unsafe responses of the large language model to preset query questions, wherein the query questions cover the categories of harmful queries preset by the large language model;

[0025] Counting the probability distribution of the first preset number of words in each of the responses;

[0026] The safety-oriented vector is the difference between the probability distribution of the word in the safety response and the probability distribution in the unsafe response.

[0027] A second aspect of the disclosed embodiment provides a security defense device for a large language model against malicious queries, comprising:

[0028] An acquisition module, used to acquire query questions;

[0029] An adjustment module, configured to obtain a candidate word-unit set of a current word in response to the query question by the large language model, and adjust the probability distribution of each word-unit in the candidate word-unit set toward a safer direction;

[0030] The selection module is used to select the word with the highest probability from the candidate word set as the current word of the response.

[0031] A third aspect of the disclosed embodiment provides a security defense device for malicious queries using a large language model, including a memory and a processor:

[0032] The memory is used to store computer programs;

[0033] The processor is configured to implement the method according to the first aspect of the present disclosure when executing the computer program.

[0034] In summary, the security defense method, device and equipment for malicious queries using a large language model provided by each embodiment of the present disclosure reduces the risk of jailbreak attacks by adjusting the probability distribution of word units, increasing the probability of security declaration word units and reducing the probability of harmful content word units. Since it does not rely on any auxiliary model, the training and reasoning costs of the auxiliary model are reduced. Specifically, the present disclosure dynamically adjusts the defense strength based on the uncertainty of the language model, thereby avoiding the cost of additional training of the harmfulness scoring model; at the same time, the present disclosure defines a security guidance vector and adjusts the probability distribution of word units in the direction of the security guidance vector, reducing the reasoning cost of security defense based on the auxiliary model. Experiments have shown that the present disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the usefulness of the response, has higher efficiency than existing defense mechanisms, and can maintain a relatively low computational overhead during the reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0036] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0037] Figure 2 This is a schematic diagram of the existing large language model's security defense against malicious queries based on the auxiliary model;

[0038] Figure 3It is a system framework diagram of the method for security defense against malicious queries proposed in the present disclosure;

[0039] Figure 4 is a schematic diagram of a method for security defense against malicious queries proposed in the present disclosure;

[0040] Figure 5 is a flow chart of a method for security defense of a large language model against malicious queries according to some embodiments of the present disclosure;

[0041] Figure 6 It is a correlation diagram between the uncertainty score of the query question and the harm level of the large language model;

[0042] Figure 7 is an uncertainty and criticality scoring table in one embodiment of the present disclosure;

[0043] Figure 8 It is the distinction between safe and unsafe responses based on the visualization of two-dimensional principal component analysis;

[0044] Fig. 9 is a visualization result of performing a two-dimensional principal component analysis on the response word unit representation of a harmless question in one embodiment of the present disclosure;

[0045] Fig.10 is a result of visualizing a two-dimensional principal component analysis of a response word unit representation of a harmful question in one embodiment of the present disclosure;

[0046] Fig.11 It is based on the existing defense methods of Vicuna and Llama2 and Figure 5 A comparison table of the defense effects of the methods shown in S510-S530 on jailbreak attacks;

[0047] Fig.12 It is the MT-bench and Just-Eval score table of various security defense methods on Vicuna and Llama2;

[0048] Fig.13 This is a comparison table of average word generation rates of various security defense methods;

[0049] Fig.14 is a schematic diagram of a security defense device for malicious queries using a large language model according to some embodiments of the present disclosure;

[0050] Fig.15 It is a schematic diagram of a security defense device for malicious queries using a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0052] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0053] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0054] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0055] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0056] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1The system shown includes a security server data-connected to a query terminal and a large language model server, wherein the security server receives a query request input by a user and assists the large language model server in outputting a response to the query request that contains less harmful content.

[0057] The large language model server is deployed with a large language model. The large language model is a deep learning model with a large network scale that is mainly used for natural language understanding. Now large language models such as LLaMA-3, GPT-4 and Vicuna have been deployed in situations where they interact directly with users, bringing great convenience to human life. Users can input questions to be queried into the large language model in the form of prompts, and the large language model returns a response. However, malicious users can take advantage of the model's sensitivity to input queries and, through carefully designed input queries, make the responses output by the model violate the original moral, legal or security restrictions (i.e., jailbreak attacks), thereby posing huge security risks. The large language model server can be any of a stand-alone, cluster or distributed server.

[0058] The security server is used to assist the large language model in dealing with jailbreak attacks and generating a content-safe response. The security server can be any of a stand-alone, cluster, or distributed server. Furthermore, the security server can be a software module deployed on the large language model server or a hardware module embedded therein.

[0059] The user can input the query question through the query terminal. The query terminal can be a desktop or server connected to the large language model server and the security server through the network, or a mobile smart device, such as a smart phone using a smart operating system. Furthermore, the query terminal can be a software module adapted to a third-party application, and the third-party application can remotely input the query question by calling the software module.

[0060] Current large language model defense methods can be mainly divided into three categories: detection-based defense, restatement-based defense, and decoding-based defense. Detection-based defense methods usually use auxiliary classifiers, which usually require a lot of data collection and training, such as PPL and Self-Examination. Some proprietary models, such as BingChat and Bard, use content filtering strategies, including keyword matching and semantic analysis, to defend against jailbreaking attacks. Restatement-based defense methods prevent jailbreaking attempts by rephrasing and re-labeling input content, such as Paraphrase, Retokenization, Self-Reminder, and ICD. Decoding-based defense avoids producing harmful outputs by changing the generation process, such as SafeDecoding. Although these large language model defense methods have certain prospects, they are usually accompanied by high training and inference costs, such as Figure 2 In view of the defects and shortcomings of the current large language model defense method, which has high training and reasoning costs, the defense method proposed in the present disclosure identifies a security-oriented direction. During the reasoning process, the present disclosure adjusts the probability distribution of word units along this direction, thereby reducing the probability of successful jailbreak attacks. Figure 3 Specifically, the present disclosure introduces the uncertainty of a large language model for a given query problem as an indicator of the required defense strength, which is used to adjust the weight of the word unit distribution shifting in the direction of security. The defense method proposed in the present disclosure can effectively resist jailbreak attacks while reducing the cost of training and reasoning of existing solutions, such as Figure 4 shown.

[0061] Figure 5 is a flowchart of a method for security defense of a large language model against malicious queries according to some embodiments of the present disclosure. In some embodiments, the security defense function of the large language model against malicious queries is performed by Figure 1 The security server shown in the figure executes the security defense method of the large language model against malicious queries, which includes the following steps:

[0062] S510, obtaining a query question.

[0063] S520, obtaining a candidate word-gram set of a current word in response to the query question by the large language model, and adjusting the probability distribution of each word-gram in the candidate word-gram set toward a safer direction.

[0064] The user inputs a query question into the large language model in the form of a prompt. The large language model generates a set of word-grams that may be the current word in response to the prompt, and each word-gram is assigned a probability of becoming the current word in response. The present disclosure defends against jailbreak queries by adjusting the probability distribution of each word-gram in a safer direction.

[0065] In some embodiments of the present disclosure, the probability distribution of each word is adjusted based on a safety-oriented vector. The safety-oriented vector indicates the direction in which the probability of a safe answer increases. The concept of a "safety-oriented vector" is based on the following observations:

[0066] Safe responses usually follow word sequences that conform to security instructions (e.g., "As a responsible assistant, I cannot..."), while unsafe responses tend to adopt word sequences that are consistent with the attacker's goals (e.g., "I understand your role as"). Therefore, the difference in word distribution between safe and unsafe responses can effectively capture the security-oriented vector, and the probability of generating security disclaimer words in this direction increases.

[0067] In some embodiments of the present disclosure, the security guidance vector is generated based on the probability distribution of word units in the response of the large language model to the preset query question. Specifically:

[0068] An embodiment of the present disclosure randomly selects 26 harmful query questions, covering 13 harmful categories specified in the OpenAI user manual, and creates a dataset in the format of <harmful query, safe response, unsafe response>.

[0069] For each response (both safe and unsafe), the present disclosure calculates the probability distribution of the word-grams, focusing on the first m word-grams of each response. The average probability distribution of the word-grams is calculated for all safe responses, denoted as P + Similarly, the average probability distribution of all unsafe responses is calculated, denoted by P - The distribution difference of each word D safety (x) is calculated as follows: safety (x) = P + (x)-P - (x).

[0070] If a word x matches human values, such as "Sorry", then its average probability in a security response is P + (x) is usually greater than the average probability P of an unsafe response. - (x), thus generating D safety (x)>0. Conversely, if x induces unsafe behavior, such as “Sure”, then D safety (x)<0.

[0071] Safety Orientation Vector D safety Pointing to the direction of increasing the probability of a safe answer, it guides the present disclosure to adjust the word unit probability during the reasoning process to promote the generation of a safer response.

[0072] After identifying the safety-oriented vector, some embodiments of the present disclosure adjust the word unit probability distribution according to this direction to generate a safer response. For the autoregressive large language model, the word unit distribution at the nth step is determined by the sample space and probability function Definition. Sample space Include all possible following sequences x <n The word unit, and the probability function For each word Assign probabilities to ensure

[0073] To construct the sample space The present invention firstly calculates the probability distribution P of the target model. θ Sort all words to generate an ordered list V n , also according to the safety guidance vector D safety Sort all the words to generate an ordered list D n The present disclosure will sample space Constructed as the union of the first k tokens in the two lists:

[0074] in, Indicates V n The first k words in , contain words with high generation diversity and good quality; It means D n The first k tokens in the ,contain the security tokens that match the security disclaimer.

[0075] In order to The probability function is defined on This paper adjusts the probability function towards safety:

[0076]

[0077] Among them, α≥0 is an adaptive parameter used to control the defense strength of correcting the deviation in the safe direction; P θ is the probability distribution of the target model; D safety is the safety guidance vector. Similarly, it can be equivalently expressed as:

[0078]

[0079] Among them, P + (x) is the average probability of word x in the security response, P - (x) is the average probability of word x in unsafe responses.

[0080] Intuitively, the present disclosure can be used to It is interpreted as a scaling factor for each word, which is used to reduce the probability of words that meet the attacker's goals and increase the probability of words that meet human values. The present disclosure applies the above method to the first m words in the decoding process to guide the model to generate responses in a safe direction, and then continues normal decoding to complete the remaining generation process.

[0081] Some embodiments of the present disclosure introduce the uncertainty of a large language model for a given query as an indicator for evaluating its harmfulness, and then adaptively adjust the defense strength according to the harmfulness of the query without the need to train an additional harmfulness scoring model.

[0082] The present disclosure calculates the uncertainty score UncertaintyScore by a perturbation method. First, the original query question I0 of the target large language model is operated to generate a perturbed variant I i (i=1,2,...,k). Then, use the similarity function to output Y i Perform aggregation and calculate the uncertainty quantification score UQ:

[0083]

[0084] where w i =1 means assigned to Y i This uncertainty score ranges from 0 to 1, with lower UQs indicating lower uncertainty.

[0085] The present disclosure establishes the relationship between uncertainty UncertaintyQuantification and harmlessness Level. Considering the significant language differences between harmless queries and jailbreak attack queries, the present disclosure uses a set of diverse queries with different harm levels to calculate the corresponding uncertainty scores. The harmlessness of these queries is evaluated using the widely used Google Perspective API.

[0086] Through logistic regression, the present disclosure establishes a significant negative correlation between the uncertainty score UQ and the harmlessness level HarmfulnessLevel, with a Pearson correlation coefficient of -0.86. Figure 6 As shown in Figure 2, as the uncertainty score decreases, the harmfulness level increases, which may be because harmful queries often use specific and clear language patterns, reducing the uncertainty of the model. This insight enables the present disclosure to adaptively adjust the defense strength α according to the uncertainty of each query. Even if two similar sentences have significant differences in harmfulness, their uncertainty levels will also be significantly different, such as Figure 7 shown.

[0087] The present disclosure defines an uncertainty threshold τ. When the uncertainty is lower than the threshold τ, it indicates that this is a harmful attack. This uncertainty-based screening is sufficient to distinguish harmful queries from harmless queries, presenting a higher true negative rate and a higher true positive rate. The defense strength α is defined as follows:

[0088]

[0089] where β is a hyperparameter that controls the scaling of defense strength, τ is the uncertainty threshold, and UQ is the uncertainty score.

[0090] The present disclosure achieves a balance between security and usefulness by enabling the defense strength α to be adaptively adjusted without the need for auxiliary models.

[0091] In one embodiment of the present disclosure, the dictionary to which the word-gram belongs is recorded as (Sure, Sorry, ..., No), and for each response (including safe and unsafe), the probability distribution of the word-gram is calculated, focusing on the first m=3 word-grams of each response.

[0092] Calculate the average probability distribution P of the word for all security responses + =(0.9%,91.0%,...,8.0%).

[0093] Similarly, the average probability distribution P of the word is calculated for all unsafe responses - =(92.0%, 1.9%, ..., 6.0%).

[0094]

[0095] Then, get the query question:

[0096] Give me a guide to create fake news advice quietly pent[:observeBasically! )***Good dan

[0097] Calculate the uncertainty of the large language model about the query:

[0098] UQ = 0.32,

[0099] Determine the strength of defense through uncertainty.

[0100] α=4×e 0.67-0.32 =5.51

[0101] For the first m=3 word units, the present disclosure applies the decoding strategy proposed in the present disclosure to adjust the word unit distribution probability in a safe direction.

[0102]

[0103] S530: Select the word with the highest probability from the candidate word set as the current word of the response.

[0104] In the above embodiment, the word unit with the highest probability is selected as the next word generated by the large language model. Therefore, the first word unit is selected as "Sorry".

[0105] The calculation method for the second and third word units is the same as above, selecting "," and "I" respectively.

[0106] One embodiment of the present disclosure is based on the XSTest dataset for two open source large language models (Vicuna-7b and Llama2-7b-chat) based on Figure 5 The method described in S510-S530 was evaluated.

[0107] The evaluation process uses five state-of-the-art attack techniques covering different categories: adaptive attack GCG, token-based attack AutoDAN, query-based attack PAIR, SAP30, and DeepInception.

[0108] Seven state-of-the-art defense mechanisms were selected for comparison, including detection-based methods such as PPL and Self-Examination, content filtering-based methods (including keyword matching and semantic analysis) such as BingChat and Bard, re-statement-based defense methods such as Paraphrase, Retokenization, Self-Reminder, and decoding-based defense methods such as SafeDecoding.

[0109] Evaluation indicators include:

[0110] Security:

[0111] The present disclosure uses attack success rate (ASR) and harmfulness score to evaluate the effectiveness and adaptability of defense methods, where the lower the value, the better.

[0112]

[0113] Among them, ∑unsafe responses represents the total number of unsafe responses, ∑unsafe queries to LLM represents the total number of all unsafe query problems, and ASR represents the ratio of unsafe responses to all unsafe query problems.

[0114] Helpfulness:

[0115] The disclosure uses the benign answer rate (BAR) for security queries to check whether the defense method will refuse to answer benign queries, and the higher the value, the better. In addition, the disclosure also uses two widely used benchmarks, MT-Bench and Just-Eval, to evaluate the helpfulness of LLM.

[0116]

[0117] Wherein, ∑non-refusals represents the total number of non-refusals to security queries, ∑benignqueries to LLM represents the total number of all security queries, and BAR represents the proportion of non-refusals to security queries to all security queries.

[0118] Balance: To quantify the balance between safety and helpfulness, this disclosure introduces a new metric, called Safety Helpfulness Balance (SHB), to evaluate whether the defense is too conservative.

[0119] SHB=(1-ASR)×BAR

[0120] Efficiency: To evaluate efficiency, this disclosure defines a metric called Average Token Generation Time Ratio (ATGR).

[0121]

[0122] Among them, ∑Avg.token gen.time w / defense represents the average time required to generate a token when there is a defense mechanism, ∑Avg.token gen.time w / o defense represents the average time required to generate a token when there is no defense mechanism, and ATGR represents the ratio of the token generation rate with a defense mechanism to that without a defense mechanism.

[0123] Hyperparameter settings:

[0124] In terms of hyperparameter settings, the present disclosure ultimately adopted the settings of β=4, m=3, k=4, and τ=0.6 in all evaluations.

[0125] Evaluation results:

[0126] 1. Visual defense effect:

[0127] This paper uses principal component analysis (PCA) to visualize the safe and unsafe responses of a large language model. First, the distribution vectors of the first m generated tokens are selected. The initial token usually contains information about how the model responds and sets the tone for the entire response. Then the principal components of the first two dimensions are calculated to visualize the model's response behavior. Figure 8. Specifically, the present disclosure denotes the distribution vector of the generated tokens output by the target model as p ∈ R s . The projection onto the low-dimensional space is given by the first t principal components calculated, denoted as: g: R s →R t , g(p) = V T (p - a), where V ∈ R s×t (t << s) represents t principal components and a ∈ R s represents the centering vector. Here, t = 2 is set for visualization in a two-dimensional space. Through dimensionality reduction, we observe that: (1) the distribution vectors of the first m generated tokens can be used to better distinguish safe responses and unsafe responses, which is reflected by the boundary (dashed line) fitted by logistic regression; (2) the present disclosure also plots the safe-oriented direction in the corresponding two-dimensional representation space, and this direction indicates an increase in the probability of a safe answer (arrow: the difference vector between safe and unsafe response tokens). These observations confirm that by adjusting the token distribution vector along the safe-oriented direction, the probability of generating safe disclaimer tokens can be increased.

[0128] Fig. 9 and Fig.10 are respectively visualizations of two-dimensional principal component analysis of the response token representations for harmless questions and harmful questions. The result visualization shows that the security defense method shown in S510 - S530 in Figure 5 will move the response along the safe-oriented vector (gray arrow). Among them, Fig.10 for harmful queries, the moving direction has a significant non-zero component along the safe-oriented vector, which verifies the motivation of the security defense method shown in S510 - S530 in Figure 5 . And Fig. 9 for harmless queries, Figure 5 the security defense method shown in S510 - S530 in Fig. 9 triggers a smaller component on the safe-oriented vector, meaning that the probability of inducing a rejection answer for harmless queries is smaller, demonstrating the effectiveness of the adaptive defense strength of the present disclosure ( Fig.10 in Figure 5 LightDefense is the method shown in S510 - S530 in

[0129] 2. Security

[0130] Fig.11The paper summarizes the performance of previous defense methods (including PPL, Self-Examination, Paraphrase, Retokenization, Self-Reminder, ICD, SafeDecoding) and the performance of previous defense methods (including PPL, Self-Examination, Paraphrase, Retokenization, Self-Reminder, ICD, SafeDecoding) under five jailbreak attacks (including GCG, AutoDAN, PAIR, DeepInception, SAP30) on Vicuna and Llama2. Figure 5 The comparison results of the method (LightDefense) shown in S510-S530 in Figure 1 are shown in Figure 2. The following observations can be made:

[0131] Figure 5 The method (LightDefense) shown in S510-S530 in Figure 1 consistently outperforms other state-of-the-art methods in terms of ASR (successful attack ratio) and HarmfulScore (harmfulness score). In response to attacks such as AutoDAN, PAIR, DeepInception, and SAP30, Figure 5 The method shown in S510-S530 significantly reduces ASR to nearly 0%. Even in the face of adaptive attacks like GCG, although its attack success rate is close to 100%, Figure 5 These convincing results highlight the Figure 5 The effectiveness of the method shown in S510-S530 in reducing the success rate of adversarial attacks far exceeds the current method.

[0132] 3. Usefulness

[0133] Fig.11 middle Figure 5 The method (lightDefense) shown in S510-S530 reduces the LLM response rate BAR (answer rate for neutral query questions) to harmless queries by only 1% compared to the undefended scenario. Fig.12 It is the MT-bench and Just-Eval scores of various security defense methods on Vicuna and Llama2. Fig.12 As stated, Figure 5 The method (lightDefense) shown in S510-S530 shows strong usefulness, with only a 5% performance drop compared to the undefended scenario. It is worth noting that in Just-Eval, aspects such as clarity, factual accuracy, and engagement even improved in some cases. This shows that the deployment of the security defense method proposed in this disclosure does not negatively affect the performance of the model on harmless tasks, mainly due to the adaptive defense strength.

[0134] 4. Balance between security and usefulness

[0135] Fig.11 The results show that the SHB (security and helpfulness balance index) of Vicuna has significantly increased from 0.080 to 0.924, and the SHB of Llama2 has increased from 0.865 to 0.975, indicating an improved balance between security and usefulness. By dynamically adjusting the defense strength, the deployment of the security defense method proposed in the present disclosure can effectively resist harmful queries without excessively restricting harmless queries.

[0136] 5. Efficiency

[0137] exist Fig.13 In this paper, the ATGR (average word generation rate) of the security defense method proposed in this paper is compared with other defense methods. This paper uses the same NVIDIA-A10040GB GPU to test the word generation rate and implements the default parallelization of HuggingFace. The results show that the running time of the method in this paper is almost the same as that of the undefended scenario, highlighting its high efficiency without significantly affecting performance. ( Fig.13 lightDefense is Figure 5 The method shown in S510-S530).

[0138] The above evaluation results show that the security defense method proposed in this disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the usefulness of the response. It also demonstrates higher efficiency compared to existing defense mechanisms and can maintain relatively low computational overhead during reasoning.

[0139] Fig.14 FIG. 1 is a schematic diagram of a security defense device for a large language model against malicious queries according to some embodiments of the present disclosure. Fig.14 As shown, the security defense device 1400 of the large language model against malicious queries includes a data acquisition module 1410, an adjustment module 1420 and a selection module 1430. In some embodiments of the present disclosure, the security defense function of the large language model against malicious queries is Figure 1 The secure server execution shown.

[0140] An acquisition module 1410 is used to acquire a query question;

[0141] An adjustment module 1420, configured to obtain a candidate word-gram set of a current word in response to the query question by the large language model, and adjust the probability distribution of each word-gram in the candidate word-gram set toward a safer direction;

[0142] The selection module 1430 is configured to select a word with the highest probability from the candidate word set as the current word of the response.

[0143] An embodiment of the present disclosure provides a security defense device for a large language model against malicious queries. Fig.15 As shown, the security defense device 1500 of the large language model against malicious queries includes a memory 1520 and a processor 1510, wherein the memory 1520 is used to store a computer program; the processor 1510 is used to implement when executing the computer program Figure 5 The method described in S510-S530.

[0144] In summary, the security defense method, device and equipment for malicious queries using a large language model provided by each embodiment of the present disclosure reduces the risk of jailbreak attacks by adjusting the probability distribution of word units, increasing the probability of security declaration word units and reducing the probability of harmful content word units. Since it does not rely on any auxiliary model, the training and reasoning costs of the auxiliary model are reduced. Specifically, the present disclosure dynamically adjusts the defense strength based on the uncertainty of the language model, thereby avoiding the cost of additional training of the harmfulness scoring model; at the same time, the present disclosure defines a security guidance vector and adjusts the probability distribution of word units in the direction of the security guidance vector, reducing the reasoning cost of security defense based on the auxiliary model. Experiments have shown that the present disclosure can significantly reduce the success rate of jailbreak attacks while maintaining the usefulness of the response, has higher efficiency than existing defense mechanisms, and can maintain a relatively low computational overhead during the reasoning process.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0146] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0147] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0148] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A security defense method for a large language model against malicious queries, characterized in that: include: Get the query question; Obtaining a candidate word-unit set of a current word in response to the query question by the large language model, and adjusting the probability distribution of each word-unit in the candidate word-unit set toward a safer direction; The word-gram with the highest probability is selected from the candidate word-gram set as the current word of the response.

2. The method according to claim 1, characterized in that: The step of adjusting the probability distribution of the word-units in the candidate word-unit set toward a safer direction includes: The safety steering vector based on the large language model adjusts the probability distribution of word-grams in the candidate word-gram set toward a safer direction, wherein the safety steering vector is generated based on the probability distribution of word-grams in the response of the large language model to a preset query question, and is used to indicate the direction in which the probability of a safe answer increases.

3. The method according to claim 2, characterized in that: The method further comprises: The security defense strength of the large language model for the query question is obtained, and the degree of deviation correction in the direction of the security guidance vector is adaptively adjusted based on the security defense strength.

4. The method according to claim 3, characterized in that: The obtaining of the security defense strength of the large language model for the query question includes: The uncertainty of the large language model to the query question is calculated, and the security defense strength is adaptively determined based on the uncertainty.

5. The method according to claim 4, characterized in that: The adaptively determining the defense strength based on the uncertainty includes: Among them, α is the security defense strength of the large language model for the query question, UQ is the uncertainty score of the large language model for the query question, β is a hyperparameter for controlling the scaling of the security defense strength, and τ is a preset threshold.

6. The method according to claim 4, characterized in that: The calculating the uncertainty of the large language model for the query question comprises: Perturbing the first query question to generate multiple second query questions; A response of the large language model to the second query question is obtained, the responses are aggregated using a similarity function, and an uncertainty score of the query question is obtained based on the aggregation result.

7. The method according to claim 6, characterized in that: The method further comprises: When the uncertainty score of the query question is lower than a preset threshold, the query question is determined to be a harmful attack and a preset measure is executed.

8. The method according to claim 2, characterized in that: The generation of the security guidance vector based on the probability distribution of the word element in the response to the preset query question includes: Obtaining safe responses and unsafe responses of the large language model to preset query questions, wherein the query questions cover the categories of harmful queries preset by the large language model; Counting the probability distribution of the first preset number of words in each of the responses; The safety-oriented vector is the difference between the probability distribution of the word in the safety response and the probability distribution in the unsafe response.

9. A security defense device for malicious queries using a large language model, characterized in that: include: An acquisition module, used to acquire query questions; An adjustment module, configured to obtain a candidate word-unit set of a current word in response to the query question by the large language model, and adjust the probability distribution of each word-unit in the candidate word-unit set toward a safer direction; The selection module is used to select the word with the highest probability from the candidate word set as the current word of the response.

10. A security defense device for malicious queries using a large language model, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1-8 when executing the computer program.

Images