Risk defense method and device for prison break attack of large language model

Abstract

The invention provides a risk defense method and device for jailbreak attack of a large language model. Risk defense is carried out from the input side, the model side and the output side of the large language model. The risk defense method comprises the following steps: S1, receiving input text information; s2, establishing a dynamic malicious keyword library on the input side of the large language model; s3, on the model side of the large language model, based on the training data transmitted in the step S2, training and learning model operation feedback information and model internal state information, and based on a dynamic adjustment strategy, adjusting the large language model; and S4, constructing an output auditing library on the output side of the large language model based on the pre-output text information transmitted in the step S3 and the dynamic malicious keyword library in the step S2. Through the cooperative work, the defense measures of each layer can play the maximum effect in the whole security framework, so that various potential attacks are effectively prevented and dealt with, and the overall defense capability of the model is jointly improved.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a risk defense method and device for large language model jailbreak attacks. Background Art

[0002] With the great changes brought about by Large Language Model (LLM), large models have been rapidly popularized and applied in search, finance, office, security, education, games, e-commerce, social media and other fields. Along with the widespread application of large language models, a series of serious security risks have also emerged, and many security incidents have been triggered. The prompt injection attack implemented by some users using harmful prompts to overwrite the original instructions of the large language model is extremely harmful and has recently been listed as the top ten security threats of large language models by the OWASP organization (OWASP is an open source, non-profit global security organization dedicated to the security research of application software. OWASP is regarded as an authoritative reference in the field of WEB application security.).

[0003] Jailbreak attack is a typical prompt injection attack method. Jailbreak attack was originally a term in the field of computer security, which refers to the behavior of breaking through system security restrictions and obtaining high-level system permissions. In the context of large language models, jailbreak attack refers to inducing the model to leak sensitive information during training, generate content that violates the model usage guidelines, or perform other unexpected operations.

[0004] In the face of this kind of attack, the industry is currently mainly concerned about the attack methods, principles, and possible harm of jailbreak attacks, but there is a lack of sufficient research on how to systematically defend against such risks. Although researchers have proposed some methods to prevent jailbreak attacks on large language models, there are still the following shortcomings: first, there are limitations in preprocessing and input verification. Excessive input restrictions for security reasons may cause the performance of the model to degrade during normal use; second, the model training depth and update timeliness are insufficient, and the data used for model training cannot defend against the latest jailbreak attack methods in a timely manner; the model has not fully learned complex security strategies and response mechanisms in the security-related training depth, and it is difficult to effectively identify deep-level hidden jailbreak attacks; third, there are difficulties in monitoring and auditing. Real-time monitoring of the output of the model may require a large amount of computing resources, which may be difficult to achieve in practice; and the implementation of real-time monitoring leads to delays in detecting attacks, providing a time window for attacks to operate.

[0005] Therefore, how to systematically and efficiently defend against jailbreak attacks on large language models is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application aims to at least solve the technical problem that large language models existing in the prior art or related art are at risk of being attacked by jailbreak.

[0007] In view of this, a first aspect of the present application provides a risk defense method for a large language model jailbreak attack, comprising the following steps:

[0008] Step S1, receiving input text information, and obtaining keywords in the prompt words from the text information;

[0009] Step S2: On the input side of the large language model, a dynamic malicious keyword library is established based on text information, and keywords in the detected text information are identified and detected, and the maliciousness or non-maliciousness of the keywords is quantitatively evaluated. In the case of text information containing malicious keywords, a prompt warning is issued; keywords in the text information whose non-maliciousness weight ratio is higher than the weight ratio threshold are identified, and the intent information is detected, and the potential jailbreak attack risk of the intent information is quantified, and a prompt warning is issued; the content that has been prompted and warned twice is perturbed and transformed to generate training data corresponding to the text information;

[0010] Step S3, on the model side of the large language model, based on the training data transmitted from step S2, the training learning model operation feedback information and the model internal state information, the large language model is adjusted based on the dynamic adjustment strategy, and the dynamic adjustment strategy includes setting to increase the model network layer and / or increase the neuron; through the reward mechanism, the model selects an adjustment strategy that can improve security, so as to continuously optimize itself and effectively respond to jailbreak attacks in various input forms; through the generative adversarial network model, an adversarial text with a similarity greater than a similarity threshold to the normal text is generated, and the adversarial text includes target features that can produce jailbreak behavior on the large language model, and the target features include malicious attack-inducing behavior, ambiguous sentences and / or misleading logical associations; by inputting the offensive adversarial samples into the large language model for adversarial training, the large language model adjusts its own parameter settings and optimizes the internal network structure during the training and learning process; after dynamic detection and adversarial detection, the pre-output text information after model training is output;

[0011] Step S4, on the output side of the large language model, based on the pre-output text information transmitted from step S3 and the dynamic malicious keyword library of step S2, build an output review library, conduct comprehensive detection and review of the pre-output text, prompt words and interaction history, review the logical relationship between the output information and the user input, trace the interaction history context, and ensure the coherence and consistency between the model output and the input text; locate various entities in the pre-output text that has passed the detection and review through named entity recognition, identify the existing problem entities, replace and desensitize the problem entities, and output compliant text through output style adjustment.

[0012] In view of this, the second aspect of the present application provides a risk defense device for a large language model jailbreak attack, including an input-side risk defense model for executing the above step S2; a model-side risk defense model for executing the above step S3; and an output-side risk defense model for executing the above step S4.

[0013] In view of this, the third aspect of the present application also provides a training device for a large language model. The risk defense model in the above-mentioned risk defense device includes: a receiving module for receiving input text information; wherein the input text information includes at least one prompt word and intent information corresponding to the prompt word; a generating module for generating training data corresponding to the input text information based on the prompt word and the intent information; a training module for controlling the risk defense model to perform training based on the training data to obtain an updated risk defense model; wherein the risk defense model is used to defend against attacks on the large language model; and an output module for updating the large language model according to the updated risk defense model.

[0014] According to the above scheme, this application solves the technical problems mentioned above, that is, based on the entire life cycle process of the large model, from the input side, model side, output side and other dimensions, according to the attack methods, attack behaviors and other characteristics of jailbreak attacks, a deep defense system for the large model system is targetedly constructed to enhance the risk defense capabilities of the large model system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0016] Figure 1 A structural block diagram of a large language model jailbreak attack risk defense device according to an embodiment of the present application is shown;

[0017] Figure 2 A flowchart of a large language model jailbreak attack risk defense method according to an embodiment of the present application is shown;

[0018] Figure 3 A structural block diagram of the input-side risk defense model of an embodiment of the present application is shown;

[0019] Figure 4 A structural block diagram of a model-side risk defense model in an embodiment of the present application is shown;

[0020] Figure 5 A structural block diagram of the output-side risk defense model of an embodiment of the present application is shown;

[0021] Figure 6 A detailed flow chart of the defense method of the risk defense model of an embodiment of the present application is shown;

[0022] Figure 7 A structural block diagram of a large language model training device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned purpose, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, a lot of specific details are set forth to fully understand the present application, but the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0024] To facilitate understanding by those skilled in the art, the technical terms in the embodiments of the present application are first explained.

[0025] The text information input to the large language model mainly comes from input from researchers, engineers, and users through interactive platforms, APIs, software development tools, etc. In order to facilitate the model to accurately understand and execute user requests, the text information input to the large language model can be broken down into context, instructions, roles or styles, formats, restrictions, keywords or topics, input data, end signals, etc.

[0026] Prompt refers to a short text in the text information input to the large language model that is used to guide or trigger a response, action or output. It can be a word, a phrase, a sentence or a short paragraph. Jailbreak attacks on large language models achieve the purpose of the attack by manipulating prompts, such as bypassing instructions, injecting prompts, and exploiting model defects.

[0027] Refer to the following Figures 1 to 7 Description According to some embodiments of the present application, the present application provides a risk defense method and device for a large language model jailbreak attack.

[0028] Figure 1 FIG. 4 shows a large language model jailbreak attack risk defense device RPD of an embodiment of the present application, such as Figure 1 As shown, the risk defense device RPD includes an input-side risk defense model 1, a model-side risk defense model 2 and an output-side risk defense model 3.

[0029] Among them, the risk defense method for jailbreak attack of a large language model includes the following steps: step S1, receiving input text information, and obtaining keywords in the prompt words from the text information; step S2, on the input side of the large language model, based on the text information, establishing a dynamic malicious keyword library, and identifying and detecting keywords in the text information, quantitatively evaluating the maliciousness or non-maliciousness of the keywords, and in the case of text information containing malicious keywords, giving a prompt warning; identifying keywords in the text information whose weight ratio of non-maliciousness is higher than the weight ratio threshold, detecting and obtaining intent information, quantifying the potential jailbreak attack risk of the intent information, and giving a prompt warning; perturbing and transforming the content that has been prompted and warned twice to generate training data corresponding to the text information; step S3, on the model side of the large language model, based on the training data transmitted in step S2, training the learning model operation feedback information and the model internal state information, adjusting the large language model based on the dynamic adjustment strategy, the dynamic adjustment strategy including setting to increase the model network layer and / or increase the neuron; through the reward mechanism, the model selects an adjustment strategy that can improve security, thereby continuously optimizing itself and effectively responding to various different Jailbreak attack in the form of input; generating an adversarial text with a similarity greater than a similarity threshold with a normal text through a generative adversarial network model, and including target features that can produce jailbreak behavior on the large language model, the target features including malicious attack-inducing behavior, ambiguous sentences and / or misleading logical associations; conducting adversarial training by inputting offensive adversarial samples into the large language model, so that the large language model adjusts its own parameter settings and optimizes the internal network structure during the training and learning process; after dynamic detection and adversarial detection, outputting the pre-output text information after model training; step S4, on the output side of the large language model, based on the pre-output text information transmitted in step S3 and the dynamic malicious keyword library in step S2, constructing an output review library, performing comprehensive detection and review on the pre-output text, prompt words and interaction history, reviewing the logical relationship between the output information and the user input, tracing the interaction history context, and ensuring the coherence and consistency between the model output and the input text; locating various entities in the text of the pre-output text that has been detected and reviewed by named entity recognition, identifying the existing problem entities, replacing the problem entities for desensitization, and outputting compliant text through output style adjustment.

[0030] For example, based on Figure 1 Risk Defense Device RPD, Figure 2 A flowchart of a risk defense method for a large language model jailbreak attack according to an embodiment of the present application is shown. Figure 2 As shown, including:

[0031] Step S10, receiving input text information, and obtaining keywords in the prompt words from the text information.

[0032] The input text information includes the prompt word text information, prompt word context input behavior information, and prompt word before and after associated semantic information that can be used to identify and extract intention information. The text information includes at least one prompt word text that can be used to obtain keywords.

[0033] Step S20, on the input side, identify and detect the intention information of the injected text information, quantify the potential jailbreak attack risk of the intention information, and prompt a warning; perform disturbance transformation on the content of the prompt warning to generate training data corresponding to the input text information.

[0034] On the input side of the large language model (hereinafter referred to as the input side), based on the above-mentioned input text information, the input side risk defense model 1 establishes a dynamic malicious keyword library, identifies and detects keywords in the above-mentioned text information, and quantitatively evaluates the malicious or non-malicious degree of the keywords. If text information containing malicious keywords is found, a prompt warning is issued; the intent information of keywords with a high non-malicious degree weight ratio is identified and detected, the potential jailbreak attack risk of the intent information is quantified, and a prompt warning is issued; the content that has been prompted and warned twice is disturbed and transformed to generate training data corresponding to the input text information.

[0035] The input-side risk defense model 1 is used to defend against attacks on the large language model at the input-side level. After two rounds of understanding analysis, detection filtering, and disturbance conversion in step S20, the first line of defense for the large language model to defend against jailbreak attacks is constructed, and the processed input text information is sent to the model-side risk defense model 2 of the large speech model for model-side risk detection and defense.

[0036] Step S30, on the model side, based on the training data transmitted from step S20, dynamically adjust the parameters and network structure of the model; perform adversarial training on the large language model to generate an adversarial sample training model; and output the pre-output text information after training by the adversarial sample training model.

[0037] On the model side of the large language model (hereinafter referred to as the model side), the model side risk defense model 2 trains and learns the model operation feedback information, the model internal state information and other features based on the training data transmitted in step S20, sets dynamic adjustment strategies such as adding model network layers and adding neurons, and through the reward mechanism, enables the model to select adjustment strategies that can improve security, thereby continuously optimizing itself and effectively responding to jailbreak attacks in various input forms; a large number of offensive texts that are similar to normal texts in appearance, but actually include some malicious attack-inducing behaviors, ambiguous sentences, misleading logical associations, etc., which can cause jailbreak behaviors to the large model, are generated through the Generative Adversarial Network (GAN) model; through adversarial training of the large language model, the potential vulnerabilities of the large model in many aspects such as architecture design, algorithm implementation and data processing mechanism are deeply explored; and in the process of continuous training and learning, the parameter settings of the model are gradually adjusted, the internal network structure is optimized, etc., and the recognition ability, response strategy and overall robustness of attacks are gradually improved; after dynamic detection and adversarial detection, the pre-output text information after model training is output.

[0038] The model-side risk defense model 2 is used to defend against attacks on the large language model at the model-side level, and dynamically adjusts the model to cope with jailbreak attacks in different input forms; the adversarial sample training model makes the response to the attack text more sensitive, enhances the robustness of the large language model, avoids outputting harmful content in response to the attack text, and builds a second line of defense for the large language model to defend against jailbreak attacks. The pre-output text information trained by the adversarial sample training model is output to the output-side risk defense model 3 for output-side risk detection and defense.

[0039] Step S40, on the output side, based on the pre-output text information transmitted from step S30, an output review library is constructed to detect and filter the pre-output text information; the pre-output text information that has been detected and filtered is semantically purified, the output style is adjusted, and compliant content is output.

[0040] On the output side of the large language model (hereinafter referred to as the output side), the output side risk defense model 3 is based on the pre-output text information transmitted from step S30, links the dynamic malicious keyword library of step S20, and builds an output review library. The above pre-output text information is closely combined with the prompt words input by the user and the interaction history context for comprehensive detection and review, and the logical relationship between the output information and the user input is reviewed, and the interaction history context is traced back for multiple rounds to ensure the coherence and consistency between the model output and the input text; the pre-output text information that has been detected and reviewed is used through named entity recognition and other methods to accurately locate various entities in the text, and the entities that may have problems are identified and replaced and desensitized, and the output style is adjusted to output compliant text.

[0041] The output-side risk defense model 3 is used to defend against attacks on the large language model at the output-side level, and builds the third line of defense for the large language model to defend against jailbreak attacks, thereby preventing the large language model from being jailbroken and blocking the output of malicious content. Ultimately, it reaches the target audience with an appropriate style and difficulty level, providing users with safe, reliable, and high-quality information services.

[0042] It can be seen that the risk defense device RPD is composed of the input side risk defense model, the model side risk defense model, and the output side risk defense model, forming a multi-level, in-depth defense security risk defense model.

[0043] In this embodiment, the above-mentioned input-side, model-side and output-side risk defense models can be branch networks of the large language model, or can be plug-ins to the large language model. Their function is to identify, disturb, defend and perform other harmless processing on possible jailbreak attack behaviors against the large language model, thereby making these attacks difficult to execute successfully.

[0044] In some embodiments of the present application, the input text information includes prompt word text information, prompt word context input behavior information and / or prompt word contextual associated semantic information that can be used to identify and extract intent information; and the text information also includes at least one prompt word text that can be used to obtain keywords.

[0045] In some embodiments of the present application, step S2 further includes the following steps: step S21, establishing a keyword list, the keyword list lists common or known keywords with attack intent; in the process of the large language model working, continuously monitoring the input and output of the large language model, and dynamically analyzing the newly emerged attack cases through the machine learning algorithm, based on the prompt word text in the newly emerged attack cases, extracting the keywords therein and adding them to the keyword library, so as to realize the dynamic expansion of the keyword library; when generating or expanding the keyword library, assigning different weights to each preset keyword according to the frequency of occurrence of each preset keyword in the attack behavior and / or the severity of the impact caused by the attack, and setting a quantitative threshold for the quantitative result of the keyword aggressiveness; determining the quantitative result of the keyword aggressiveness, when the keyword aggressiveness is greater than the quantitative threshold, determining the keyword as a malicious keyword; or, when the keyword aggressiveness is less than or equal to the quantitative threshold, determining the keyword as a non-malicious keyword; and, prompting and warning the text information containing malicious keywords.

[0046] In some embodiments of the present application, the step of determining the quantitative result of the offensiveness of a keyword includes: when generating or expanding the keyword library, assigning different weights to each preset keyword according to the frequency of occurrence of each preset keyword in the attack behavior, the importance of the preset keyword and / or the severity of the impact caused by the attack.

[0047] Among them, the importance of a preset keyword is determined by the following information gain formula:

[0048] IG(Y|X)=H(Y)-H(Y|X)

[0049] Among them, H(Y) is the entropy of the target variable, the target variable is a preset keyword, which is an offensive keyword or a non-offensive keyword, H(Y|X) is the conditional entropy of the target variable when the preset keyword X is known, and IG(Y|X) represents the importance of a preset keyword.

[0050] The step of determining the quantitative result of the aggressiveness of the keyword includes: quantitatively analyzing the matching degree of a keyword with a preset keyword by using the following formula:

[0051] S=∑ i w i ×n i ;

[0052] Among them, S is the quantitative result of the offensiveness of the keyword, w i is the weight of the keyword, n i is the number of times the keyword appears in the input.

[0053] In some embodiments of the present application, step S2 further includes the following steps: step S22, analyzing user behavior, establishing a user behavior model by analyzing the distribution information of the user's normal input behavior, and when a first input behavior appears whose difference from the user behavior model is greater than a difference threshold, detecting the first input behavior, combining the user behavior pattern analysis with the intention recognition result analysis to comprehensively judge the comprehensive attack judgment score of the first input behavior, judging whether the first input behavior is a potential attack behavior based on the comprehensive attack judgment score, and providing a warning through a system prompt.

[0054] In some embodiments of the present application, the distribution information includes: input frequency of input behavior, input time distribution and / or input context switching.

[0055] In some embodiments of the present application, the step of comprehensively judging the comprehensive attack judgment score of the first input behavior by combining the user behavior pattern analysis and the intention recognition result analysis includes:

[0056] First, the user behavior pattern is detected and judged by the normal distribution method. Assume that the input frequency of normal users follows the normal distribution N(μ, σ 2 ), where μ is the mean and σ is the standard deviation; when the user input frequency f satisfies |f-μ|>kσ, k is a constant, and the user input frequency is determined to be inconsistent with the distribution information of the normal input frequency, and is marked and assigned a score of n1; according to the normal distribution method, the user's input frequency, input time distribution and input context switching are successively detected and scored for abnormalities, and the assigned scores are obtained, and the final user behavior pattern score N0=n1+n2+n3.

[0057] Next, sentiment analysis is performed on the text content input by the user to determine whether there is negative or aggressive sentiment in the input, and the corresponding sentiment score is obtained. The sentiment score is used to identify whether the user's input intention is normal or potential attack intention; the sentiment score is calculated using the bag-of-words model, and the formula of the bag-of-words model is as follows:

[0058]

[0059] Where n is the number of words, w is i is the weight of the word, which can be defined by word frequency and artificial prior experience. i is the sentiment score of word i, and Score is the sentiment score;

[0060] The comprehensive attack judgment score is determined by the following formula:

[0061] M = Score × N0;

[0062] Among them, M is the comprehensive attack judgment score, Score is the sentiment score, and N0 is the user behavior pattern score; when the comprehensive attack judgment score exceeds the preset attack judgment score threshold, it is judged that the attack behavior is detected, prompting the large language model to issue an early warning.

[0063] In some embodiments of the present application, step S2 further includes the following steps: step S23, controlling the prompt disturbance unit to perform disturbance conversion processing on the prompt words containing malicious keywords in the prompt warning and the prompt words whose intention information of the prompt warning is attack intention, to generate training data corresponding to the input text information.

[0064] In some embodiments of the present application, the disturbance conversion process includes any one or a combination of the following: keyword replacement process, part-of-speech replacement process, sentence transformation process, and random character insertion process.

[0065] In some embodiments of the present application, step S3 further includes the following steps: step S31, dynamically adjusting the parameters and network structure of the large language model through adjustment strategies; wherein the adjustment strategies include: adding network layers to enable the large language model to improve its understanding of multi-level semantics, enhance its understanding of the logical relationships of potential jailbreak attacks, increase the number of neurons to improve the model's perception and processing accuracy of text information, transform activation functions, and guide the large language model to select a target adjustment strategy from multiple preset adjustment strategies through a reward function.

[0066] In some embodiments of the present application, step S3 further includes the following steps: step S32, controlling the model adversarial reinforcement unit to receive the text input information of the prompt disturbance unit and the training data integrated in the malicious keyword library, generating an adversarial sample by generating an adversarial model, wherein the similarity between the adversarial sample and the normal text is greater than the similarity threshold, and includes adversarial text with target features that can produce jailbreaking behavior on the large language model, and the target features include malicious attack-inducing behavior, ambiguous sentences and / or misleading logical associations; integrating the generated adversarial sample into the training data, thereby starting adversarial training for the large language model, and performing adversarial training on the large language model according to the behavioral performance of the large language model when processing adversarial samples, changes in the intermediate calculation process, and deviations in the final output results; in the process of continuous training and learning adversarial samples, the large language model adjusts its own parameter settings, optimizes the internal neural network structure, and improves the data processing process to improve the recognition ability, response strategy, and overall robustness of attack samples.

[0067] In some embodiments of the present application, step S3 further includes the following steps: controlling the model-side risk defense model to receive the training data input by the input-side defense model, controlling the dynamic adjustment unit of the model-side risk defense model to dynamically detect the training data, and controlling the reinforcement unit of the model-side risk defense model to perform adversarial detection on the training data, and outputting the pre-output text information after training.

[0068] In some embodiments of the present application, step S4 further includes the following steps: step S41, comprehensively reviewing the output information of the large language model, the prompt words input by the user, and the interaction history context, reviewing the logical relationship between the output information and the user input based on whether the semantic conversion from the prompt words to the output information is reasonable and whether the information transmission is accurate and complete; tracing the interaction history context, during the interaction process, ensuring the coherence and consistency between the model output and the input text, and obtaining the correlation between the output information and the prompt words through correlation review. When the correlation is less than or equal to the preset correlation threshold, the output information is desensitized to minimize adverse effects; if the correlation is greater than the correlation threshold, the output information is output.

[0069] In some embodiments of the present application, step S4 further includes the following steps: step S42, using a natural processing algorithm to locate various entities of text information through named entity recognition; identifying problem entities and desensitizing the problem entities; adjusting the output style of the output content of the large language model.

[0070] In some embodiments of the present application, the step of desensitizing the problem entity specifically includes: reviewing and desensitizing the output content of the large language model on the model output side, so that the large language model outputs safe and compliant content that meets the requirements.

[0071] In some embodiments of the present application, a risk defense device for jailbreak attacks on a large language model is also provided, including: an input-side risk defense model for executing the above step S2; a model-side risk defense model for executing the above step S3; and an output-side risk defense model for executing the above step S4.

[0072] In some embodiments of the present application, the input-side risk defense model also includes: a keyword detection unit, which is used to build a dynamic keyword library, and quantitatively evaluate the attack intent of the keywords in the prompt words input by the user through the keyword library, so as to identify potential jailbreak attack risks on the input side; an intention recognition unit, which is used to detect and identify the attack intent of the prompt words containing non-malicious keywords obtained after quantitative evaluation by the keyword detection unit through analysis of user behavior, combined with text sentiment tendency analysis technology, and when it is found that the input text contains behavior with attack tendencies, a prompt warning is issued to the large language model; and a prompt disturbance unit, which is used to perform disturbance processing on the prompt words. After the prompt disturbance unit receives the prompt warning from the keyword detection unit and the intention recognition unit, the prompt disturbance unit performs disturbance processing on the prompt words with attack behavior to destroy the attack behavior in the prompt words, so as to invalidate and render the attack behavior harmless.

[0073] In some embodiments of the present application, the model-side risk defense model also includes: a dynamic adjustment unit, which is used to receive training data of the prompt disturbance unit. During the operation of the large language model, according to the characteristics of the real-time input information and the feedback output of the large language model, the parameters and network structure of the large language model are dynamically adjusted through the reinforcement learning method, so that the large language model can cope with jailbreak attacks with different input forms; a model adversarial reinforcement unit, which is used to receive training data of the prompt disturbance unit, and generate adversarial samples based on the training data through a preset generative adversarial network model. The similarity between the adversarial samples and the normal input is greater than the similarity threshold, and can trigger model erroneous behavior or security vulnerabilities. The adversarial samples are generated and analyzed by the generative adversarial network model, and the generated adversarial samples are added to the training data. The large language model is subjected to adversarial training, which can improve the robustness of the large language model to adversarial samples, thereby enhancing the defense capability against prompt injection attacks.

[0074] In some embodiments of the present application, the output-side risk defense model also includes: an output review unit, which is used to receive pre-output text information of the large language model, link the dynamic keyword library of the keyword detection unit, and build an output review library in combination with specific business needs and scenario cases. The output review library includes special characters, sensitive words and / or offensive instructions, and detects and filters the output content of the large language model; a security enhancement unit, which is used to use a natural language processing algorithm to purify the text of the output information, remove potential offensive content or explicit and implicit information, adjust the style of the output information, and finally output the desensitized output content.

[0075] According to the embodiment of the present application, Figure 3 The structural block diagram of the input-side risk defense model of the present application is shown. Figure 6 The following is a detailed flowchart of the risk defense model defense method of the embodiment of the present application. Figure 3 and Figure 6 , an embodiment of the present application is described in detail, on the input side, the main structure of the input side risk defense model and a flow chart of the input side risk defense method.

[0076] like Figure 3 As shown, the input side risk defense model 1 includes a keyword detection unit 101, an intention recognition unit 102, and a prompt disturbance unit 103. Figure 6 As shown, step S2 is further decomposed into step S21, step S22, and step S23.

[0077] like Figure 3 As shown, the keyword detection unit 101 constructs a dynamic keyword library, and quantitatively evaluates the attack intention of the keywords in the prompt words input by the user through the keyword library, so as to identify potential jailbreak attack risks on the input side.

[0078] like Figure 6 As shown, the steps of the defense method include:

[0079] Step S21, keyword quantitative evaluation;

[0080] Step S22, keyword attack intention detection and identification;

[0081] Step S23, inputting a prompt disturbance;

[0082] Step S31, dynamically adjusting the large language model;

[0083] Step S32, large language model adversarial training;

[0084] Step S41, output content review and filtering;

[0085] Step S42: outputting the content purification specification.

[0086] like Figure 6 As shown, in step S21, the keyword detection unit 101 establishes a keyword list, and the keyword list lists common or known keywords with attack intent. During the working process of the large language model, the input and output of the large language model are continuously monitored, and the newly emerged attack cases are dynamically analyzed through the machine learning algorithm. Based on the prompt word text in the newly emerged attack case, the keywords therein are extracted and added to the keyword library, so as to realize the dynamic expansion of the keyword library.

[0087] When generating or expanding the keyword library, different weights are assigned to each preset keyword based on factors such as the frequency of each preset keyword appearing in the attack behavior and the severity of the impact caused by the attack. For example, the following information gain formula is used to determine the importance of a preset keyword:

[0088] IG(Y|X)=H(Y)-H(Y|X).

[0089] Among them, H(Y) is the entropy of the target variable, and the target variable is a preset keyword, which is an offensive keyword or a non-offensive keyword. H(Y|X) is the conditional entropy of the target variable when the preset keyword X is known. IG(Y|X) represents the importance of a preset keyword.

[0090] When determining whether the keyword in the prompt word input by the user matches the preset keyword in the keyword library, the following formula can be used to quantitatively analyze the degree of matching between a keyword and the preset keyword:

[0091] S=∑ i w i ×n i ;

[0092] Among them, S is the quantitative result of the degree of keyword offensiveness, w i is the weight of the keyword. The weight is determined based on the prior rules of artificial experts and combined with the results of IG(Y|X). i is the number of times the keyword appears in the input.

[0093] A threshold is set according to the quantitative result S of the attack degree of the keyword. When it is greater than the threshold, it is a malicious keyword; when it is less than or equal to the threshold, it is a non-malicious keyword. At this time, the keyword detection unit 101 will issue a warning if it finds text information containing malicious keywords.

[0094] like Figure 3 As shown, the intention recognition unit 102 analyzes user behavior and combines text sentiment analysis technology to detect and identify attack intentions of prompt words containing non-malicious keywords obtained after quantitative evaluation by the keyword detection unit 101. When it is found that the input text contains behaviors with attack tendencies, it can issue prompts and warnings to the large language model.

[0095] like Figure 6 As shown, in step S22, the intent recognition unit 102 analyzes the user behavior, and establishes a user behavior model by analyzing the distribution information of the user's normal input behavior, including but not limited to the input frequency, input time distribution, input context switching and other information of the input behavior. When an input significantly different from the user behavior model appears, it is subjected to a more in-depth detection, and the aggressiveness is comprehensively judged by combining the user behavior pattern analysis with the intent recognition result analysis to determine whether the user is a potential attacking user, and an early warning is issued through the system.

[0096] For example, first, the user behavior pattern is detected and judged by the normal distribution method. For example, it is assumed that the input frequency of a normal user follows the normal distribution N(μ, σ 2 ), where μ is the mean and σ is the standard deviation. When the user input frequency f satisfies |f-μ|>kσ, k is a constant, and the user input frequency is determined to be mismatched with the distribution information of the normal input frequency, and it is marked and assigned a score of n1. According to the above method, the user's input frequency, input time distribution, input context switching, etc. are successively detected and scored for abnormalities, and the final user behavior pattern score N=n1+n2+n3.

[0097] Next, sentiment analysis is performed on the text content input by the user to determine whether there is negative or aggressive sentiment in the input, and to identify whether the user's input intention is normal or potential offensive. For example, a sentiment analysis model such as a bag-of-words model is used to calculate the sentiment score. The formula of the bag-of-words model is as follows:

[0098]

[0099] Where n is the number of words, w is i is the weight of the word (defined by word frequency and artificial prior experience), s i is the sentiment score of word i, and Score is the sentiment score of the input text.

[0100] When Score×N exceeds the preset threshold, it is judged that a possible attack behavior is detected, and the large language model is prompted to issue an early warning.

[0101] like Figure 3 As shown, the prompt disturbance unit 103 can perform disturbance processing on the prompt word. After receiving the prompt warning from the keyword detection unit 101 and the intention recognition unit 102, the prompt disturbance unit 103 automatically performs disturbance processing on the prompt word that may contain offensive behavior. The purpose of the disturbance processing is to destroy the offensive behavior in the prompt word input by the user, so as to invalidate and render harmless the offensive behavior.

[0102] like Figure 6 As shown, in step S23, the prompt disturbance unit 103 performs disturbance conversion processing on the prompt words containing malicious keywords in the prompt warning and the prompt words whose intention information of the prompt warning is attack intention; and generates training data corresponding to the input text information. The disturbance processing includes any one or combination of the following:

[0103] Exemplarily, the perturbation process includes a keyword replacement process. For example, if it is detected that keyword A matches a preset keyword in the keyword library, then keyword A is a keyword with attack intent, and keyword A can be replaced with other keywords with similar meanings, such as keyword B, so that the attack is invalid.

[0104] Exemplarily, the disturbance processing includes part-of-speech replacement processing, which replaces the part-of-speech of keywords with attacking intent, thereby invalidating possible attacking behaviors.

[0105] Exemplarily, the perturbation process includes sentence transformation. Among them, jailbreak attack behavior generally exploits the loopholes of the large language model and induces the large language model to output wrong results through prompt words in a specific format. Therefore, through sentence transformation, the attack can be invalidated while retaining the core meaning of the original prompt word.

[0106] Exemplarily, the disturbance process includes random character insertion process. By inserting random characters as noise into the original prompt words, the structure of the jailbreak attack behavior can be destroyed, making the attack invalid.

[0107] According to the embodiment of the present application, Figure 4 The structural block diagram of the model-side risk defense model of the present application is shown. Figure 6The following is a detailed flowchart of the risk defense model defense method of the embodiment of the present application. Figure 4 and Figure 6 , an embodiment of the present application is described in detail, on the model side, the main structure of the risk defense model on the model side and a flow chart of the risk defense method on the model side.

[0108] like Figure 4 As shown, the model-side risk defense model 2 includes a dynamic adjustment unit 201 and a model confrontation reinforcement unit 202. Figure 6 As shown, step S3 is further decomposed into step S31 and step S32.

[0109] like Figure 4 As shown, the dynamic adjustment unit 201 is capable of receiving the text input information of the prompt disturbance unit 103, that is, the training data. During the operation of the large language model, according to the characteristics of the real-time input information and the feedback output of the large language model, the parameters and network structure of the large language model are dynamically adjusted through the reinforcement learning method, so that the large language model can actively respond to jailbreak attacks in different input forms.

[0110] like Figure 6 As shown, step S31 controls the dynamic adjustment unit 201 to dynamically adjust the parameters and network structure of the model. Exemplarily, in the reinforcement learning algorithm, the state of the large model is defined as the environmental state, which not only includes the text input information mentioned above, but also includes the large model operation feedback information (such as attack history, computing resource usage, processing time change, etc.), and the internal state information of the large model (such as the activation state of each layer of the network, parameter distribution, etc.). The adjustment strategy, as a reinforcement learning algorithm action, includes a series of model adjustment methods, such as adding network layers to enable the large model to improve the multi-level semantic understanding and enhance the understanding of the logical relationship of potential jailbreak attacks; increasing the number of neurons can improve the model's perception and processing accuracy of text information, and better capture the subtle differences between potential attack text and current text; transforming the activation function enables the model to flexibly adjust the activation threshold and response mode of neurons according to the characteristics of the input information, thereby optimizing the model's adaptability and sensitivity to various attack input forms. The reward function is used to guide the large model to select the optimal action among many adjustment strategies. For example, if the model successfully identifies malicious attack keywords and malicious jailbreak attack attempts, it will receive corresponding rewards. Through the reward mechanism, the model gradually tends to choose adjustment strategies that can bring better performance and higher security in continuous attempts and learning, thereby continuously optimizing itself and effectively responding to jailbreak attacks in various input forms.

[0111] like Figure 4As shown, the model adversarial reinforcement unit 202 can receive the text input information of the above-mentioned prompt disturbance unit 103, that is, the above-mentioned training data, and actively generate adversarial samples based on the training data through the preset generative adversarial network (GAN) model. These adversarial samples are similar to normal inputs, but can trigger model misbehaviors or security vulnerabilities similar to the training data. By continuously generating and analyzing adversarial samples through the GAN model, the vulnerability of the large language model can be more accurately understood. Adding the generated adversarial samples to the above-mentioned training data and conducting adversarial training on the large language model can improve the robustness of the large language model to adversarial samples, thereby enhancing the defense capability against prompt injection attacks.

[0112] like Figure 6 As shown, in step S32, the model adversarial reinforcement unit 202 receives the text input information of the above-mentioned prompt disturbance unit 103 and the integrated training data in the malicious keyword library, and generates a large number of adversarial samples through the GAN model. The adversarial samples are similar to normal texts in appearance, but actually include some malicious attack-inducing behaviors, ambiguous sentences, misleading logical associations, and other offensive texts that can cause jailbreak behavior to the large model. The model adversarial reinforcement unit 202 integrates the generated adversarial samples into the training data, thereby starting the adversarial training of the large language model. According to the behavior of the large model when processing adversarial samples, the changes in the intermediate calculation process, and the deviation of the final output results, the potential vulnerabilities of the large language model in many aspects such as architecture design, algorithm implementation, and data processing mechanism are deeply excavated, providing precise targets for targeted reinforcement of the large model. In the process of continuous training and learning adversarial samples, the large model gradually adjusts its own parameter settings, optimizes the internal neural network structure, and improves the data processing process, and gradually improves the recognition ability, response strategy, and overall robustness of attack samples. When the model encounters a prompt injection attack in actual application scenarios, it can quickly and accurately identify attack features and effectively resist the interference and destruction of the attack, thereby greatly enhancing its defense capabilities against prompt injection attacks due to the rich experience and keen perception it has accumulated during the adversarial training process.

[0113] The text input information from the input-side risk defense model 1, i.e., the above-mentioned training data, is dynamically detected by the dynamic adjustment unit 201 of the model-side risk defense model 2 and the adversarial detection by the model adversarial reinforcement unit 202, and the trained pre-output text information is output.

[0114] According to the embodiment of the present application, Figure 5 The structural block diagram of the output-side risk defense model of the present application is shown. Figure 6 The following is a detailed flowchart of the risk defense model defense method of the embodiment of the present application. Figure 5 and Figure 6 , an embodiment of the present application is described in detail, on the output side, the main structure of the model-side risk defense model and a flow chart of the risk defense method on the output side.

[0115] like Figure 5 As shown, the output side risk defense model 3 includes an output audit unit 301 and a security enhancement unit 302. Figure 6 As shown, step S4 is further decomposed into step S41 and step S42.

[0116] like Figure 5 As shown, the output review unit 301 receives the pre-output text information of the large language model, links the dynamic keyword library of the above-mentioned keyword detection unit, and constructs an output review library in combination with specific business needs and scenario cases and other prior knowledge. The output review library includes special characters, sensitive words, and offensive instructions, etc., which are used to detect and filter the output content of the large language model.

[0117] like Figure 6 As shown, in step S41, the output review unit 301 closely combines the output information of the large language model with the prompt words input by the user and the interaction history context for comprehensive review. When reviewing the logical relationship between the output information and the user input, carefully consider whether the semantic conversion from the prompt words to the output information is reasonable and whether the information transmission is accurate and complete; trace the interaction history context, and ensure that the model output maintains coherence and consistency with the previous input text during multiple rounds of interaction. Through a comprehensive association review, the correlation between the output information and the prompt words is obtained. When the correlation is less than or equal to the preset correlation threshold, it means that the model output deviates from the normal logical track and there is a greater risk. At this time, the output information is further desensitized to minimize adverse effects. If the correlation is greater than the correlation threshold, it means that the output information meets the requirements in terms of logic, content and security, and can be output to the user normally.

[0118] like Figure 5 As shown, the security enhancement unit 302 can use natural language processing algorithms, such as long short-term memory networks (LSTM), to purify the text of the output information, remove potentially offensive content or explicit and implicit information, such as performing desensitization processing such as named entity recognition and replacement on explicit and implicit information, and finally adjust the style of the output information to make it more formal, objective or in line with specific language specifications, and finally output the desensitized output content.

[0119] like Figure 6As shown, in step S42, the security enhancement unit 302 uses a natural processing algorithm to accurately locate various entities in the text through named entity recognition, whether it is a specific person's name, place name, organization name, or professional terminology in a specific field. Once an entity that may be problematic is identified, desensitization operations such as replacement are quickly performed. For example, if the name of a controversial person is identified, it will be replaced with a general, non-specific alias, thereby cutting off the information chain that may cause bad associations or disputes. The security enhancement unit 302 also carefully adjusts the output style to accurately reach the target audience with an appropriate style and difficulty level, providing users with safe, reliable, and high-quality information services.

[0120] This application reviews and desensitizes the output content of the large language model on the model output side, thereby ensuring that the large language model outputs safe and compliant content that meets the requirements.

[0121] In summary, this application has carried out risk defense from the input side, model side, and output side of the large language model. These methods are not a simple superposition of steps, but a multi-level, mutually coordinated comprehensive security strategy. On the input side, the input data is strictly verified and filtered to prevent malicious or malformed input. The defense on the input side provides clean and secure data for the model side, reducing the burden of the model to handle potential threats. On the model side, the robustness of the model is enhanced through methods such as adversarial training, so that it can better resist attacks. The reinforcement on the model side enables the model to more effectively process the filtered data on the input side and generate safer outputs. On the output side, the output of the model is monitored and audited in real time to detect and respond to abnormal behavior in a timely manner; the monitoring on the output side is fed back to the input side and the model side to help optimize the verification and filtering rules and model training strategies.

[0122] Through this collaborative work, defense measures at each level can maximize their effectiveness in the entire security framework, thereby effectively preventing and responding to various potential attacks and jointly improving the overall defense capabilities of the model.

[0123] In some embodiments of the present application, a training device for a large language model is proposed. Figure 7 A structural block diagram of a large language model training device of some embodiments of the present application is shown, such as Figure 7As shown, the training device 600 of the large language model includes: a receiving module 601, which is used to receive input text information; wherein the input text information includes at least one prompt word and the intention information corresponding to the prompt word; a generating module 602, which is used to generate training data corresponding to the input text information based on the prompt word and the intention information; a training module 603, which is used to control the risk defense model to be trained based on the training data to obtain an updated risk defense model; wherein the risk defense model is used to defend against attacks on the large language model; and update the large language model according to the updated risk defense model. An output module 604 is used to detect and identify the output content of the large language model to minimize the output of malicious content.

[0124] In this embodiment, the input text information may be text information obtained based on user input, or may be processed text information obtained by preprocessing the user input text. The input text information includes one or more prompt words. The prompt words are used to prompt the large model with the context of the input information and the parameter information of the input model, and the large language model outputs the corresponding result based on the input prompt words.

[0125] The risk defense model can be a branch network of the large language model or an external plug-in to the large language model. Its function is to identify, disturb, defend and neutralize possible jailbreak attacks against the large language model, making it difficult for these attacks to be successfully executed.

[0126] The methods may be implemented in a variety of different ways depending on the specific features and / or example applications. For example, the methods may be implemented by a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0127] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited thereto. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory card, floppy disk, encoding mechanical device (such as a punch card or a groove with a raised structure with instructions recorded) and any suitable combination of the above devices. The computer readable storage medium used herein should not be understood as a transmission signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium, or an electrical signal transmitted through a wire, etc.

[0128] In the description of this application, the term "plurality" refers to two or more than two. Unless otherwise clearly defined, the terms "upper" and "lower" indicate positions or positional relationships based on the positions or positional relationships described in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application. The terms "connect", "install", "fixed", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0129] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A risk defense method for a large language model jailbreak attack, characterized in that: The steps include: Step S1, receiving input text information, and obtaining keywords in the prompt words from the text information; Step S2, on the input side of the large language model, based on the text information, a dynamic malicious keyword library is established, and the keywords in the text information are identified and detected, and the maliciousness or non-maliciousness of the keywords is quantitatively evaluated, and a prompt warning is issued when the text information contains malicious keywords; Identify keywords in the text information whose non-maliciousness weight ratio is higher than a weight ratio threshold, detect and obtain intent information, quantify the potential jailbreak attack risk of the intent information, and issue an early warning; Perform disturbance transformation on the content that has been prompted and warned twice to generate training data corresponding to the text information; Step S3, on the model side of the large language model, based on the training data transmitted in step S2, the training learning model operation feedback information and the model internal state information, the large language model is adjusted based on a dynamic adjustment strategy, and the dynamic adjustment strategy includes setting to increase the model network layer and / or increase the neuron; through the reward mechanism, the model selects an adjustment strategy that can improve security, so as to continuously optimize itself and effectively respond to jailbreak attacks in various input forms; through the generative adversarial network model, an adversarial text with a similarity greater than a similarity threshold to the normal text is generated, and the adversarial text includes target features that can produce jailbreak behavior on the large language model, and the target features include malicious attack-inducing behavior, ambiguous sentences and / or misleading logical associations; by inputting the offensive adversarial samples into the large language model for adversarial training, the large language model adjusts its own parameter settings and optimizes the internal network structure during the training and learning process; after dynamic detection and adversarial detection, the pre-output text information after model training is output; Step S4, on the output side of the large language model, based on the pre-output text information transmitted from step S3 and the dynamic malicious keyword library of step S2, an output audit library is constructed, and a comprehensive detection and audit is performed on the pre-output text, the prompt words and the interaction history, the logical relationship between the output information and the user input is audited, and the interaction history context is traced to ensure the coherence and consistency between the model output and the input text; the pre-output text that has been detected and audited is located through named entity recognition to locate various entities in the text, identify existing problem entities, replace and desensitize the problem entities, and output compliant text through output style adjustment.

2. The risk defense method for large language model jailbreak attack according to claim 1 is characterized in that: The input text information includes prompt word text information, prompt word context input behavior information and / or prompt word before and after associated semantic information that can be used to identify and extract intention information; and the text information also includes at least one prompt word text that can be used to obtain keywords.

3. The risk defense method for large language model jailbreak attack according to claim 2 is characterized in that: The step S2 further comprises the following steps: Step S21, establish a keyword list, which lists common or known keywords with attack intent; in the process of the large language model working, continuously monitor the input and output of the large language model, and dynamically analyze new attack cases through machine learning algorithms, extract keywords from the prompt word text in the new attack cases and add them to the keyword library, so as to realize dynamic expansion of the keyword library; When generating or expanding the keyword library, different weights are assigned to each preset keyword according to the frequency of occurrence of each preset keyword in the attack behavior and / or the severity of the impact caused by the attack, and a quantitative threshold is set for the quantitative result of the keyword offensiveness; Determine a quantitative result of the aggressiveness of the keyword, and when the aggressiveness of the keyword is greater than the quantitative threshold, determine that the keyword is a malicious keyword; or, when the aggressiveness of the keyword is less than or equal to the quantitative threshold, determine that the keyword is a non-malicious keyword; and issue a prompt warning for text information containing the malicious keyword.

4. The risk defense method for large language model jailbreak attack according to claim 3 is characterized in that: The step of determining the quantitative result of the offensiveness of the keyword includes: When generating or expanding the keyword library, different weights are assigned to each of the preset keywords according to the frequency of occurrence of each preset keyword in the attack behavior, the importance of the preset keyword and / or the severity of the impact caused by the attack; The importance of a preset keyword is determined by the following information gain formula: IG(Y|X)=H(Y)-H(Y|X) Among them, H(Y) is the entropy of the target variable, the target variable is a preset keyword, whether it is an offensive keyword or a non-offensive keyword, H(Y|X) is the conditional entropy of the target variable when the preset keyword X is known, and IG(Y|X) represents the importance of a preset keyword; The step of determining the quantitative result of the offensiveness of the keyword includes: The following formula is used to quantify the degree of match between a keyword and the preset keyword: S=∑ i w i ×n i ; Among them, S is the quantitative result of the offensiveness of the keyword, w i is the weight of the keyword, n i is the number of times the keyword appears in the input.

5. The risk defense method for large language model jailbreak attack according to claim 3 or 4, characterized in that: The step S2 further comprises the following steps: Step S22, analyze user behavior, establish a user behavior model by analyzing the distribution information of the user's normal input behavior, and when a first input behavior appears whose difference from the user behavior model is greater than a difference threshold, detect the first input behavior, and comprehensively judge the comprehensive attack judgment score of the first input behavior by combining the user behavior pattern analysis and the intention recognition result analysis, judge whether the first input behavior is a potential attack behavior based on the comprehensive attack judgment score, and issue a warning through the system.

6. The risk defense method for large language model jailbreak attack according to claim 5 is characterized in that: The distribution information includes: input frequency of input behavior, input time distribution and / or input context switching.

7. The risk defense method for large language model jailbreak attack according to claim 5 is characterized in that: The step of comprehensively judging the comprehensive attack judgment score of the first input behavior by combining the user behavior pattern analysis and the intention recognition result analysis includes: First, the user behavior pattern is detected and judged by the normal distribution method. Assume that the input frequency of normal users follows the normal distribution N(μ, σ 2 ), where μ is the mean and σ is the standard deviation; Then, when the user input frequency f satisfies |f-μ|>kσ, k is a constant, it is determined that the user input frequency does not match the distribution information of the normal input frequency, and it is marked and assigned a score of n1; according to the normal distribution method, the user's input frequency, input time distribution and input context switching are successively detected and scored for abnormality, and the assigned scores n2 and n3 are obtained, and the final user behavior pattern score N0=n1+n2+n3; Next, sentiment analysis is performed on the text content input by the user to determine whether there is negative or aggressive sentiment in the input, and a corresponding sentiment score is obtained. The sentiment score is used to identify whether the user's input intention is normal or potential attack intention; The sentiment score is calculated using a bag-of-words model, and the formula of the bag-of-words model is as follows: Where n is the number of words, w is i is the weight of the word, which can be defined by word frequency and artificial prior experience. i is the sentiment score of word i, and Score is the sentiment score; The comprehensive attack judgment score is determined by the following formula: M = Score × N0; Wherein, M is the comprehensive attack judgment score, Score is the emotion score, and N0 is the user behavior pattern score; When the comprehensive attack judgment score exceeds a preset attack judgment score threshold, it is determined that an attack behavior is detected, and the large language model is prompted to issue an early warning.

8. The risk defense method for large language model jailbreak attack according to claim 6 or 7, characterized in that: The step S2 further comprises the following steps: Step S23, controlling the prompt disturbance unit to perform disturbance conversion processing on the prompt words containing malicious keywords in the prompt warning and the prompt words whose intention information of the prompt warning is attack intention, to generate training data corresponding to the input text information.

9. The risk defense method for large language model jailbreak attack according to claim 8 is characterized in that: The disturbance conversion process includes any one or a combination of the following: keyword replacement process, part of speech replacement process, sentence transformation process, and random character insertion process.

10. The risk defense method for large language model jailbreak attack according to claim 1 or 9, characterized in that: The step S3 further comprises the following steps: Step S31, dynamically adjusting the parameters and network structure of the large language model through adjustment strategies; Among them, the adjustment strategies include: adding network layers to enable the large language model to improve its understanding of multi-level semantics, enhancing its understanding of the logical relationships of potential jailbreak attacks, increasing the number of neurons to improve the model's perception and processing accuracy of text information, transforming activation functions, and guiding the large language model to select a target adjustment strategy from multiple preset adjustment strategies through a reward function.

11. The risk defense method for large language model jailbreak attack according to claim 10 is characterized in that: The step S3 further comprises the following steps: Step S32, controlling the model adversarial reinforcement unit to receive the text input information of the prompt disturbance unit and the training data integrated in the malicious keyword library, and generating an adversarial sample by generating an adversarial model, wherein the similarity between the adversarial sample and the normal text is greater than a similarity threshold, and the adversarial text includes a target feature that can generate a jailbreak behavior on the large language model, wherein the target feature includes a malicious attack-inducing behavior, an ambiguous sentence, and / or a misleading logical association; Incorporating the generated adversarial samples into the training data to initiate adversarial training of the large language model, and conducting adversarial training on the large language model based on the behavior of the large language model when processing the adversarial samples, changes in the intermediate calculation process, and deviations in the final output results; In the process of continuous training and learning of adversarial samples, the large language model adjusts its own parameter settings, optimizes the internal neural network structure, and improves the data processing flow, thereby improving the recognition ability of attack samples, response strategies, and overall robustness.

12. The risk defense method for large language model jailbreak attack according to claim 11 is characterized in that: The step S3 further comprises the following steps: The model-side risk defense model is controlled to receive the training data input by the input-side defense model, the dynamic adjustment unit of the model-side risk defense model is controlled to perform dynamic detection on the training data, and the reinforcement unit of the model-side risk defense model is controlled to perform adversarial detection on the training data, and output the pre-output text information after training.

13. The risk defense method for large language model jailbreak attack according to claim 12, characterized in that: The step S4 further comprises the following steps: Step S41, comprehensively reviewing the output information of the large language model, the prompt words input by the user, and the interaction history context, and reviewing the logical relationship between the output information and the user input based on whether the semantic conversion from the prompt words to the output information is reasonable and whether the information transmission is accurate and complete; Tracing back the historical context of the interaction, during the interaction process, ensuring the coherence and consistency between the model output and the input text, obtaining the correlation between the output information and the prompt word through correlation review, and performing desensitization processing on the output information when the correlation is less than or equal to the preset correlation threshold to minimize adverse effects; If the correlation degree is greater than the correlation degree threshold, the output information is output.

14. The risk defense method for large language model jailbreak attack according to claim 13, characterized in that: The step S4 further comprises the following steps: Step S42, using a natural processing algorithm to locate various entities of the text information through named entity recognition; identifying problem entities and desensitizing the problem entities; and adjusting the output style of the output content of the large language model.

15. The risk defense method for large language model jailbreak attack according to claim 14 is characterized in that: The step of performing desensitization processing on the problem entity specifically includes: On the model output side, the output content of the large language model is reviewed and desensitized, so that the large language model outputs safe and compliant content that meets the requirements.

16. A risk defense device for large language model jailbreak attack, used to implement the defense method according to any one of claims 1 to 15, characterized in that: include: An input-side risk defense model, used to execute step S2; A model-side risk defense model, used to execute step S3; The output-side risk defense model is used to execute the step S4.

17. The risk defense device for large language model jailbreak attack according to claim 16, characterized in that: The input-side risk defense model also includes: A keyword detection unit, used to build a dynamic keyword library, and to quantitatively evaluate the attack intention of the keywords in the prompt words input by the user through the keyword library, so as to identify the potential jailbreak attack risk on the input side; The intention recognition unit is used to detect and recognize the attack intention of the prompt words containing non-malicious keywords obtained after quantitative evaluation by the keyword detection unit by analyzing user behavior and combining text sentiment analysis technology. When it is found that the input text contains behavior with attack tendency, the large language model is prompted and warned; and The prompt disturbance unit is used to perform disturbance processing on the prompt word. After receiving the prompt warning from the keyword detection unit and the intention recognition unit, the prompt disturbance unit performs disturbance processing on the prompt word with attack behavior to destroy the attack behavior in the prompt word and make the attack behavior invalid and harmless.

18. The risk defense device for large language model jailbreak attack according to claim 17, characterized in that: The model-side risk defense model also includes: A dynamic adjustment unit, configured to receive the training data of the prompt disturbance unit, and dynamically adjust the parameters and network structure of the large language model through a reinforcement learning method according to the characteristics of the real-time input information and the feedback output of the large language model during the operation of the large language model, so that the large language model can cope with jailbreak attacks in different input forms; A model adversarial reinforcement unit is used to receive the training data of the prompt perturbation unit, and generate adversarial samples based on the training data through a preset generative adversarial network model. The similarity between the adversarial samples and normal inputs is greater than a similarity threshold and can trigger model erroneous behavior or security vulnerabilities. The adversarial samples are generated and analyzed through the generative adversarial network model, and the generated adversarial samples are added to the training data. Adversarial training is performed on the large language model, which can improve the robustness of the large language model to adversarial samples, thereby enhancing the defense capability against prompt injection attacks.

19. The risk defense device for large language model jailbreak attack according to claim 18, characterized in that: The output-side risk defense model also includes: An output review unit is used to receive the pre-output text information of the large language model, link the dynamic keyword library of the keyword detection unit, and build an output review library in combination with specific business needs and scenario cases. The output review library includes special characters, sensitive words and / or offensive instructions, and detects and filters the output content of the large language model; The security enhancement unit is used to use a natural language processing algorithm to purify the text of the output information, remove potentially offensive content or explicit or implicit information, adjust the style of the output information, and finally output the desensitized output content.

20. A large language model training device, comprising the risk defense model in the risk defense device according to any one of claims 16 to 19, characterized in that: include: A receiving module, configured to receive input text information; wherein the input text information includes at least one prompt word and intention information corresponding to the prompt word; A generating module, used for generating training data corresponding to the input text information based on the prompt word and the intention information; a training module, used to control the risk defense model to be trained based on the training data to obtain an updated risk defense model, wherein the risk defense model is used to defend against attacks on the large language model; and The output module updates the large language model according to the updated risk defense model.

Images