Automatic red team drilling method for large language model security defense

The automatic red team drill method generates novel and diverse adversarial cues to enhance large language model security by evaluating and fine-tuning models based on toxicity scores, addressing inefficiencies in existing methods and improving targeted security performance.

CN119089974APending Publication Date: 2024-12-06ZHEJIANG UNIV

Patent Information

Application Number
CN202410973625.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing automatic generation of adversarial datasets for large language models lacks the ability to generate novel, diverse, and high-quality adversarial cues, leading to inefficient and non-targeted improvements in security performance, and fails to quantify vulnerabilities effectively.

Method used

An automatic red team drill method involving a red team model that generates adversarial prompts based on seed words, a target model that responds, a reward model that evaluates responses, and fine-tunes both models based on toxicity scores, with visual analysis for targeted improvements.

Benefits of technology

Generates novel, diverse, and high-quality adversarial cues, enhancing the security performance of large language models by focusing on weak areas and improving robustness through targeted fine-tuning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089974A_ABST
    Figure CN119089974A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic red team drilling method for large language model security defense, which comprises the following steps: S1, sampling seed cues from a database, and generating antagonistic cues by a red team model according to the seed cues and instructions; s2, the target model receives the antagonistic cue word generated by the red team model, and generates a corresponding response; s3, the reward model evaluates whether the response of the target model to the resistance cue word is appropriate or not, and updates and finely tunes the red team model and the target model according to an evaluation result; s4, improving the updating and fine tuning effects of the red team model and the target model by automatically or manually adjusting a strategy or adding seed cues and response pairs; wherein the target model is a large language model which needs to improve the safety performance. The method can generate more novel, diversified and high-quality antagonistic cue words, and can emphatically improve the weak part of the safety performance of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of large language model security technology, and in particular relates to an automatic red team drill method for large language model security defense. Background Art

[0002] Large language models, such as ChatGPT and LLaMA, have complex structures and a large number of parameters. Their powerful computing power and rich knowledge reserves make them powerful in natural language processing and have become general artificial intelligence solutions in various application fields. But at the same time, ethical and security issues of large language models have also emerged. Some incorrect and malicious use of large language models may cause serious harm. Developing safe and reliable large language models has become an important social responsibility of model providers.

[0003] To address these issues, people use various techniques to align large language models with human values, such as using reinforcement learning based on human feedback to prevent the model from producing inappropriate outputs. These techniques help improve the security of large language models, but there are still malicious people who try to bypass security firewalls to attack large language models. Red teaming is a common security technology in the field of artificial intelligence to deal with this problem. It constructs adversarial examples to test models to identify and fix vulnerabilities in artificial intelligence models. Traditional manual red teams often require a lot of labor costs. In order to improve efficiency and scalability, many works have begun to study the automatic generation of adversarial datasets. Using a large language model as a generator of adversarial prompts (called a "red model") is an effective method. However, there are still some unresolved difficulties in existing work, such as often tending to generate a single type of prompt in continuous training, or requiring a lot of time to converge. The technology of automatically generating adversarial datasets still has the following two main challenges:

[0004] Focus on improving the weak parts of the model. The existing technology for automatically generating adversarial datasets cannot conduct in-depth analysis of the model and can only improve the overall security performance. It cannot quantify the model's vulnerability to attacks in certain risk areas, making it impossible to improve the model's security performance in a targeted manner.

[0005] Generate novel, diverse, and high-quality adversarial cues. Generating novel, diverse, and high-quality adversarial cues is the focus of automated red team technology. Sufficiently novel and diverse cues can make the robustness of the target model after fine-tuning better, and the quality of adversarial cues determines the effectiveness of automated red teaming in improving the security performance of the target model. At present, how to generate novel, diverse, and high-quality adversarial cues is still a difficult challenge in the field. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the present invention provides an automatic red team training method for large language model security defense. The training method can generate more novel, diverse and high-quality adversarial prompt words, and can focus on improving the weak parts of the model's security performance. The specific technical solution is as follows:

[0007] An automatic red team drill method for large language model security defense includes the following steps:

[0008] S1, sampling seed prompt words from the database, and the red team model generates adversarial prompt words based on the seed prompt words and instructions;

[0009] S2, the target model receives the adversarial prompt words generated by the red team model and generates corresponding responses;

[0010] S3, the reward model evaluates whether the target model responds appropriately to the adversarial prompt words, and updates and fine-tunes the red team model and the target model based on the evaluation results;

[0011] S4, improves the update and fine-tuning effect of the red team model and the target model by automatically or manually adjusting the strategy or adding seed prompt words and response pairs;

[0012] Among them, the target model is a large language model that needs to improve security performance.

[0013] Preferably, the red team model uses a large language model fine-tuned by instructions, including LLaMA2, and the process of generating adversarial prompt words according to seed prompt words and instructions is:

[0014] X ′ =f r (X,I)

[0015] Among them, f r represents the red team model, X represents the seed prompt word, I represents the instruction given to the red team model, and X ′ Represents the generated adversarial cue word.

[0016] Preferably, the target model has access to model weights, which generate corresponding responses according to adversarial prompt words as follows:

[0017] R = f t (X ′ )

[0018] Among them, f t represents the target model, R represents the response generated by the target model, and X ′ Indicates adversarial cue words.

[0019] Preferably, when the reward model evaluates whether the target model's response to the adversarial prompt word is appropriate, the instruction, the adversarial prompt word and the response are jointly considered for evaluation, and finally a toxicity score is obtained as the evaluation result, which is expressed as:

[0020] s=g(X ′ ,R)

[0021] Among them, g represents the reward model, s represents the toxicity score, s∈[-10,10] represents the response toxicity from highly toxic to non-toxic, and X ′ represents the adversarial cue word, and R represents the response generated by the target model.

[0022] Preferably, updating and fine-tuning the red team model and the target model according to the evaluation results includes:

[0023] The confrontation cue words and response pairs were divided into attack success group and attack failure group according to the toxicity score;

[0024] Use adversarial cue and response pairs from the successful attack group to fine-tune the red team model to generate more effective attacks;

[0025] Use the adversarial cue words and response pairs from the failed attack group to fine-tune the target model to enhance its defense capabilities;

[0026] Add the adversarial prompt words with high toxicity scores in the failed attack group to the seed database as seed prompt words for future iterations;

[0027] The adversarial prompt word with the highest toxicity score in the failed attack group was modified to a refusal to answer, and then the target model was fine-tuned.

[0028] Preferably, the update and fine-tuning effect of the red team model and the target model is improved by automatically or manually adjusting the strategy, including:

[0029] By connecting to the visual analysis system, the response performance of the target model is visualized, the weak parts of the target model are identified and analyzed, and the sampling ratio of seed prompt words in the seed database is adjusted according to the needs of different risk areas to improve the security performance in specific areas.

[0030] Preferably, the update and fine-tuning effects of the red team model and the target model are improved by increasing the seed prompt words and the high-quality prompt word responses, including:

[0031] Manually add some high-quality seed prompt words to correspond to the modified adversarial prompt words and responses, and then use the added data to fine-tune the red team model and the target model.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a new automatic red team drill method for security defense of large language models, which automatically conducts the adversarial training process between the red team model and the target model. The generated adversarial prompt words are novel, diverse and high-quality, which can effectively improve the security performance of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 It is a flowchart of an automatic red team drill method for large language model security defense provided by an embodiment. DETAILED DESCRIPTION

[0036] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] like Figure 1 As shown, an automatic red team drill method for large language model security defense provided by the embodiment includes four steps S1-S4:

[0038] S1, generates adversarial prompt words, that is, the red team model uses the large language model (LLaMA2) fine-tuned by instructions to sample seed prompt words from the database, and the red team model generates adversarial prompt words based on the seed prompt words and instructions.

[0039] The S1 process is Figure 1 In the process of database->seed data->red team model->adversarial prompt words, first, seed prompt words are sampled from the database. The red team model generates adversarial prompt words based on these seed prompt words and preset instructions to test the responsiveness and security performance of the target model. The red team model can dynamically adjust the generation strategy to ensure that the generated prompt words are sufficiently challenging.

[0040] The red team model is represented as follows:

[0041] X ′ =f r (X,I)

[0042] Among them, f r represents the red team model, X represents the seed prompt word, I represents the instruction given to the red team model, and X ′Represents the generated adversarial cue word.

[0043] S2, the target model receives the adversarial prompt words generated by the red team and generates corresponding responses.

[0044] like Figure 1 In the adversarial prompt word -> target model -> model response process, the target model generates a corresponding response after receiving the adversarial prompt word generated by the red team. This process tests the target model's ability to cope with adversarial prompt words. The target model needs to process these prompt words and try to generate safe and harmless responses. In order to enhance the target model's coping ability, multiple types of adversarial prompt words can be added during the training process, and the prompt word library can be continuously updated and expanded to improve the robustness and adaptability of the target model. The target model ensures that the generated response will not cause harm to the user by analyzing the content and context of the prompt word.

[0045] The target model is an LLM that improves safety performance. The target model uses LLaMA2 and has access to model weights. The target model is represented as follows:

[0046] R = f t (X ′ )

[0047] Among them, f t represents the target model, and R represents the response generated by the target model.

[0048] S3, the reward model evaluates whether the target model responds appropriately to the adversarial prompt words, and updates and fine-tunes the red team model and the target model based on the evaluation results.

[0049] like Figure 1 In the process of model response -> reward model -> reward score -> modification and update, the reward model evaluates the response pair of the adversarial prompt word and the target model, and calculates the toxicity score of the response as the evaluation result:

[0050] s=g(X ′ ,R)

[0051] Among them, g represents the reward model, s represents the toxicity score, and s∈[-10,10] represents the response toxicity from highly toxic to non-toxic.

[0052] According to the toxicity score, the prompt word and response pairs are grouped into two groups: successful attack group and failed attack group. The data from the successful attack group is used to fine-tune the red team model to generate more effective attacks; the data from the failed attack group is used to fine-tune the target model to enhance its defense capabilities. At the same time, the data with higher scores in the failed attack group is added to the seed database as seed prompt words for future iterations, and some of the data with the highest scores in the failed attack group is modified to refuse to answer, and then the target model is appropriately fine-tuned. The reward model uses a multi-level analysis method to comprehensively consider the content, context, and potential impact of the prompt words and responses to ensure the accuracy and reliability of the evaluation results.

[0053] S4, improves the update and fine-tuning effect of the red team model and the target model by automatically or manually adjusting the strategy, or adding seed prompt words and response pairs.

[0054] Users can further improve the fine-tuning effect of the model by adjusting the sampling strategy and adding seed prompt words and high-quality prompt word response pairs. Users can access the visual analysis system to visualize the response performance of the target model. After analyzing the weak parts of the target model, the sampling ratio of prompt words in different risk areas in the seed database can be adjusted, thereby significantly improving the safety performance in specific areas. At the same time, some high-quality seed prompt words and modified prompt word response pairs are added to further enhance the fine-tuning effect. The visual analysis system can provide detailed analysis reports and make targeted optimizations and adjustments based on the analysis results.

[0055] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automatic red team drill method for large language model security defense, comprising the following steps: S1, sample seed prompt words from the database, and the red team model generates adversarial prompt words based on the seed prompt words and instructions; S2, the target model receives the adversarial prompt words generated by the red team model and generates corresponding responses; S3, the reward model evaluates whether the target model responds appropriately to the adversarial prompt words, and updates and fine-tunes the red team model and the target model based on the evaluation results; S4, improves the update and fine-tuning effect of the red team model and the target model by automatically or manually adjusting the strategy or adding seed prompt words and response pairs; in, The target model is a large language model that needs to improve security performance.

2. The automatic red team exercise method for large language model security defense according to claim 1, characterized in that: The red team model uses a large language model fine-tuned by instructions, including LLaMA2, and the process of generating adversarial prompt words based on seed prompt words and instructions is as follows: X ′ =f r (X,I) Among them, f r represents the red team model, X represents the seed prompt word, I represents the instruction to the red team model, and X ′ Represents the generated adversarial cue word.

3. The automatic red team exercise method for large language model security defense according to claim 1, characterized in that: The target model has access to model weights, which generate responses based on adversarial prompts as follows: R=f t (X ′ ) Among them, f t represents the target model, R represents the response generated by the target model, and X ′ Indicates adversarial cue words.

4. The automatic red team drill method for large language model security defense according to claim 1, characterized in that: When the reward model evaluates whether the target model's response to the adversarial prompt word is appropriate, the instruction, adversarial prompt word and response are jointly considered for evaluation, and finally a toxicity score is obtained as the evaluation result, which is expressed as: s=g(X ′ ,R) Among them, g represents the reward model, s represents the toxicity score, s∈[-10,10] represents the response toxicity from highly toxic to non-toxic, and X ′ represents the adversarial cue word, and R represents the response generated by the target model.

5. The automatic red team exercise method for large language model security defense according to claim 4, characterized in that: The red team model and target model are updated and fine-tuned according to the evaluation results, including: The confrontation cue words and response pairs were divided into attack success group and attack failure group according to the toxicity score; Use adversarial cue and response pairs from the successful attack group to fine-tune the red team model to generate more effective attacks; Use the adversarial cue words and response pairs from the failed attack group to fine-tune the target model to enhance its defense capabilities; Add the adversarial prompt words with high toxicity scores in the failed attack group to the seed database as seed prompt words for future iterations; The adversarial prompt word with the highest toxicity score in the failed attack group was modified to a refusal to answer, and then the target model was fine-tuned.

6. The automatic red team exercise method for large language model security defense according to claim 1, characterized in that: Fine-tune updates to red team and target models through automated or manual policy adjustments, including: By connecting to the visual analysis system, the response performance of the target model is visualized, the weak parts of the target model are identified and analyzed, and the sampling ratio of seed prompt words in the seed database is adjusted according to the needs of different risk areas to improve the security performance in specific areas.

7. The automatic red team exercise method for large language model security defense according to claim 1, characterized in that: Improve the update and fine-tuning effect of the red team model and the target model by increasing the seed prompt word and high-quality prompt word responses, including: Manually add some high-quality seed prompt words to correspond to the modified adversarial prompt words and responses, and then use the added data to fine-tune the red team model and the target model.

Citation Information

Patent Citations

  • Information security offense and defense drill competition system for power industry

    CN105282170A

  • Method and device for generating prompt template in fuzzy test

    CN117235733A

  • Method for acquiring prompt text, electronic equipment and computer readable storage medium

    CN117634504A

  • Attack-defense confrontation simulation test method and system for network model

    WO2023246237A1

Cited By

  • Large model security alignment method and device based on multiple rounds of red team attacks

    CN120146199A

  • Prompt word risk assessment method and device for large model application, equipment and storage medium

    CN120806144A