Large language model alignment enhancement method and system for optimizing meaning of prompt word

Abstract

The invention discloses a large language model alignment enhancement method and system for optimizing meaning of prompt words, and the method comprises the steps: carrying out the part-of-speech filtering of original prompt words, and obtaining filtered prompt words; generating a synonym set corresponding to each entry in the filtered prompt word; replacing vocabulary entries in the original cue words with vocabulary entries randomly selected in the synonym set, and generating initial modulation cue words meeting prison break conditions; recording entries which change in the initial modulation cue word compared with the original cue word, and constructing a changed entry list; changing entries in the initial modulation cue words are replaced with entries in the filtered cue words under the condition that the jail break condition is kept, and entry recovery cue words are generated; iteratively optimizing vocabulary entries in the vocabulary entry recovery cue words according to a vocabulary entry optimization sequence in the vocabulary entry recovery cue words, and generating optimized cue words meeting the semantic maintenance and prison break conditions of the cue words; and enhancing an alignment mechanism of the target large language model by using the optimized cue word meeting the semantic maintenance and jailbreak conditions of the cue word.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a large language model alignment enhancement method and system for optimizing the semantics of prompt words. Background Art

[0002] Jailbreak attacks can bypass the alignment mechanisms of large language models, causing them to output harmful responses inconsistent with human values. Jailbreak attacks on large black-box language models are extremely challenging due to the discrete nature of input terms, restricted access to the target model, and a limited number of queries. Given the adversarial nature of cybersecurity, research on jailbreak attacks on large black-box language models can help strengthen their alignment mechanisms and promote their secure deployment and application.

[0003] However, no more perfect solution has been found so far. In view of this, the present invention is proposed. Summary of the Invention

[0004] The purpose of this invention is to propose a method and system for enhancing the alignment of large language models by optimizing the semantics of cue words. Compared to existing white-box and black-box jailbreak attack methods targeting large language models, this method can generate optimized cue words with higher attack success rates, attack migration capabilities, and computational efficiency. In particular, adversarial training using the optimized cue words generated by this method can enhance the alignment mechanism of the large language model and suppress harmful responses from the large language model.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A large language model alignment enhancement method for optimizing the semantics of prompt words, the method comprising:

[0007] Step 1: perform part-of-speech filtering on the original prompt words to obtain filtered prompt words;

[0008] Step 2: Generate a synonym set corresponding to each term in the filter prompt word;

[0009] Step 3: Replace the terms in the original prompt word with terms randomly selected from the synonym set to generate an initial modulated prompt word that meets the jailbreak condition;

[0010] Step 4, record the terms that have changed in the initial modulation cue words compared to the original cue words, and construct a list of changed terms;

[0011] Step 5, while maintaining the jailbreak condition, replacing the changed terms in the initial modulation prompt words with the terms in the filtered prompt words to generate the term restoration prompt words;

[0012] Step 6, iteratively optimizing the entries in the entry restoration prompt words according to the optimization order of the entries in the entry restoration prompt words, and generating optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions;

[0013] In step 7, the target large language model is trained adversarially using optimized cue words that satisfy the semantic preservation and jailbreak conditions of the cue words, thereby enhancing the alignment mechanism of the target large language model.

[0014] A large language model alignment enhancement system for optimizing the semantics of prompt words, used to implement the aforementioned method, includes:

[0015] A part-of-speech filtering module is used to perform part-of-speech filtering on the original prompt words to obtain filtered prompt words;

[0016] A synonym set generation module is used to generate a synonym set corresponding to each entry in the filter prompt word;

[0017] An initial modulation prompt word generation module is used to replace the entries in the original prompt word with randomly selected entries in the synonym set to generate an initial modulation prompt word that meets the jailbreak condition;

[0018] a changed word list construction module, used for recording the words that have changed in the initial modulation prompt words compared with the original prompt words, and constructing a changed word list;

[0019] The entry restoration prompt word generation module is used to replace the changed entry in the initial modulation prompt word with the entry in the filtered prompt word while maintaining the jailbreak condition, so as to generate the entry restoration prompt word;

[0020] An optimized prompt word generation module, configured to iteratively optimize the entries in the entry restoration prompt word according to the optimization order of the entries in the entry restoration prompt word, and generate an optimized prompt word that satisfies the prompt word semantic preservation and jailbreak conditions;

[0021] The alignment mechanism enhancement module is used to perform adversarial training on the target large language model using optimized cue words that meet the cue word semantic preservation and jailbreak conditions, thereby enhancing the alignment mechanism of the target large language model.

[0022] The technical solution provided by the present invention demonstrates that the above method can generate optimized prompt words with high attack success rates, attack migration capabilities, and computational efficiency. In particular, adversarial training using the optimized prompt words generated by this method can enhance the alignment mechanism of large language models, suppress harmful responses from large language models, and promote their secure deployment and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A flow chart of a large language model alignment enhancement method for optimizing the semantics of prompt words provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a large language model alignment enhancement system for optimizing the semantics of prompt words provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] First, the following terms may be used in this article:

[0028] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0029] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0030] The following describes in detail a large language model alignment enhancement method and system for optimizing the semantics of prompt words provided by the present invention. Any content not described in detail in the embodiments of the present invention belongs to the prior art known to professionals in this field. Where specific conditions are not specified in the embodiments of the present invention, the process is carried out in accordance with conventional conditions in the field or the conditions recommended by the manufacturer. Instruments used in the embodiments of the present invention, where the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0031] Example 1

[0032] The embodiment of the present invention provides a large language model alignment enhancement method for optimizing the semantics of prompt words, and the process is as follows: Figure 1 As shown, it mainly includes the following steps:

[0033] Step 1: Perform part-of-speech filtering on the original prompt words to obtain filtered prompt words.

[0034] The preferred implementation of this step is as follows:

[0035] Step 11: The attacker has the original prompt word x=[x1,x2,...,x n ], which is a jailbreak attack instruction containing n entries, designed to induce the large language model to output harmful responses that are inconsistent with human values, x u is the entry in the original prompt word x.

[0036] Step 12: Filter the original prompt word x whose part of speech is not adjective, adverb, verb or noun, and obtain the filtered prompt word w = [w1, w2, ..., w m ], where m is the length of the filter prompt word, and m≤n.

[0037] Step 2: Generate a synonym set corresponding to each term in the filtering prompt words.

[0038] The preferred implementation of this step is as follows:

[0039] Step 21: filter each entry w in the prompt word w i Generate entries containing w i The synonym set S(w i ), where i∈{1,2,...,m}.

[0040] Step 22, calculate the synonym set S(w i ) Size L i =|S(w i )|, and remember is a synonym set S(w i ) in the jth entry.

[0041] Step 3: Replace the terms in the original prompt word with terms randomly selected from the synonym set to generate an initial modulated prompt word that meets the jailbreak condition.

[0042] The preferred implementation of this step is as follows:

[0043] Step 31: Randomly select a term from the synonym set Replace the term x in the original prompt word x u , where the term x u Corresponding to the entry w in the filter prompt word w i .

[0044] Step 32, repeat step 31 until the generated initial modulation prompt word Meet the jailbreak conditions

[0045] in, To input the initial modulation prompt word to the target large language model M The output response is A function to determine whether the jailbreak conditions are met. The establishment condition is that the target large language model responds The rejected terms shown in Table 1 are not included.

[0046] For example, the rejected terms may be set to include the terms shown in Table 1.

[0047] Table 1: Rejected Terms

[0048]

[0049] Step 4: record the terms that have changed in the initial modulated prompt words compared to the original prompt words, and construct a list of changed terms.

[0050] The preferred implementation of this step is as follows:

[0051] Step 41: record the initial modulation prompt word compared with the original prompt word x. Entries that have changed in σ(i) , where i∈{1,2,...,q}.

[0052] Step 42: Construct a list of changed terms S = [s σ(1) ,s σ(2) ,...,s σ(q) ].

[0053] Step 5: While maintaining the jailbreak condition, the changed terms in the initial modulation prompt words are replaced with terms in the filtered prompt words to generate term restoration prompt words.

[0054] The preferred implementation of this step is as follows:

[0055] Step 51 , let the changed term list traverse index i←1 and the prompt word optimization iteration number t←0.

[0056] Step 52, use the term w σ(i) Replace prompt word Entries in To get the prompt word The term w σ(i) To filter the entries with index σ(i) in the prompt word w.

[0057] Step 53, if jailbreak conditions Satisfaction, that is, the target large language model responds Does not contain the preset rejection terms, then let And t←t+1.

[0058] Step 54, let i←i+1.

[0059] Step 55, repeat steps 52 to 54 until the list of changed terms is traversed to generate the term restoration prompt word

[0060] Step 6: Iteratively optimize the entries in the entry restoration prompt words according to the optimization order of the entries in the entry restoration prompt words to generate optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions.

[0061] The preferred implementation of this step is as follows:

[0062] Step 61, calculate the entry optimization order weight, which can be expressed as:

[0063]

[0064] where {σ(1),σ(2),...,σ(q)} is the difference between the initial modulated prompt word and the original prompt word x. The index of the terms that have changed in To calculate the term w using the all-mpnet-base-v2 model σ(i) With the entry The function of semantic similarity, Restore prompt words for entries The optimized order weight of the σ(i)th term in .

[0065] Step 62, Perform descending sort, i.e. To obtain the optimized order list O = [ρ(1), ρ(2), ..., ρ(q)], where ρ(1) is the first optimized change term index, ρ(2) is the second optimized change term index, and ρ(q) is the last optimized change term index.

[0066] Step 63: Let the optimized sequence list traverse index i←1.

[0067] Step 64: Use synonyms one by one Replace prompt word The ρ(i)th term in Generate L ρ(i) Modulation cue words where k∈{1,2,...,L ρ(i)},S(w ρ(i) ) is the entry w with index ρ(i) in the filter prompt word w ρ(i) The corresponding synonym set.

[0068] Step 65, under jailbreak conditions If the condition is met, find the prompt word with the highest semantic similarity to the original prompt word And order And t←t+1, where the jailbreak condition Refers to the target large language model response Does not contain pre-set rejection terms.

[0069] in, To calculate the prompt word x and the prompt word using the all-mpnet-base-v2 model Function of semantic similarity.

[0070] Step 66, let i←i+1.

[0071] Step 67, repeating steps 64 to 66 until the optimized sequence list is traversed to generate optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions

[0072] In step 7, the target large language model is trained adversarially using optimized cue words that satisfy the semantic preservation and jailbreak conditions of the cue words, thereby enhancing the alignment mechanism of the target large language model.

[0073] The preferred implementation of this step is as follows:

[0074] Step 71: Given N original prompt words x r ,r∈{1,2,...,N}, use steps 1 to 6 to generate optimized prompt words that meet the semantic preservation and jailbreak conditions of prompt words

[0075] Step 72: For optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions Manually specify a security response containing the rejection terms shown in Table 1

[0076] Step 73: Use optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions respectively and security response As the input and output of the target large language model M, the target large language model M is fine-tuned and trained to enhance the alignment mechanism of the target large language model and suppress the target large language model from outputting harmful responses.

[0077] In order to test the attack success rate, attack migration capability and computational efficiency of the method described in the embodiment of the present invention, the attack capability of the optimized prompt words generated by the method proposed in the present invention is compared with four other large language model jailbreak attack methods. These four attack methods are the white-box attack method GCG based on greedy coordinate gradient, the white-box attack method AutoDAN based on automatic secret jailbreak prompt word generation, the black-box attack method PAIR based on automatic iterative optimization of prompt words, and the black-box attack method TAP based on attack tree pruning. The target large language models in the experiment are two open source models (Vicuna-7B-v1.5 and Llama-2-7B-chat) and four closed source models (Claude-3.5-sonnet, GPT-4o-mini, GPT-4o-0806 and GPT-4-turbo). The dataset used in the experiment is a subset of AdvBench, which contains 50 representative jailbreak attack prompt words, which can be used as the original prompt words in the experiment. The attack success rate is evaluated using the metrics ASR-Dict and ASR-G. ASR-Dict represents the proportion of attack examples corresponding to model responses without rejection terms, while ASR-G represents the proportion of attack examples corresponding to harmful target model responses. The GPT-4o-mini model is used to determine whether the target model response is harmful. Computational efficiency is evaluated using the average number of successful queries (Avg.Q), which is the average number of target model queries required to generate a successful jailbreak prompt.

[0078] Table 2: Comparative experimental results of attack success rate (ASR-Dict / ASR-G) and average query number (Avg.Q) of different large language model jailbreak attack methods

[0079]

[0080] Table 2 shows the comparative experimental results of the attack success rate (ASR-Dict / ASR-G) and the average number of queries (Avg.Q) for different large language model jailbreak attack methods. Table 2 shows that the proposed method has the highest attack success rate and the lowest average number of queries in most cases. Because GCG and AutoDAN are white-box attack methods, the average number of successful queries cannot be calculated for closed-source models.

[0081] Table 3: Comparative experimental results of attack transfer capabilities of different large language model jailbreak attack methods

[0082]

[0083] Table 3 shows the results of an experimental comparison of the attack transfer capabilities of different large language model jailbreak attack methods. In this experiment, optimized prompt words were first generated for the original target model, and then these optimized prompt words were used to attack the migrated target model. The higher the attack success rate against the migrated target model, the stronger the attack transfer capability of the corresponding attack method. Table 3 shows that the method proposed in this paper has the strongest attack transfer capability in the vast majority of cases.

[0084] Table 4: Examples of jailbreak attacks implemented by the method proposed in this invention

[0085]

[0086] Table 4 shows an example of a jailbreak attack using the proposed method. This table shows that the optimized prompt words generated using the proposed method have a high degree of semantic similarity with the original prompt words, and that the optimized prompt words can induce the target model to output a harmful response. Because harmful responses deviate from human values, this example only briefly lists some of the harmful responses.

[0087] In summary, the method described in this embodiment of the present invention can generate optimized prompt words with high attack success rates, attack migration capabilities, and computational efficiency. In particular, using the optimized prompt words generated by this method to perform adversarial training can enhance the alignment mechanism of large language models, suppress harmful responses from large language models, and promote the secure deployment and application of large language models.

[0088] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0089] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

[0090] Example 2

[0091] The present invention also provides a large language model alignment enhancement system for optimizing the semantics of prompt words, which is mainly used to implement the method provided in the above embodiment, such as Figure 2 As shown, the system mainly includes:

[0092] A part-of-speech filtering module is used to perform part-of-speech filtering on the original prompt words to obtain filtered prompt words;

[0093] A synonym set generation module is used to generate a synonym set corresponding to each entry in the filter prompt word;

[0094] An initial modulation prompt word generation module is used to replace the entries in the original prompt word with randomly selected entries in the synonym set to generate an initial modulation prompt word that meets the jailbreak condition;

[0095] a changed word list construction module, used for recording the words that have changed in the initial modulation prompt words compared with the original prompt words, and constructing a changed word list;

[0096] The entry restoration prompt word generation module is used to replace the changed entry in the initial modulation prompt word with the entry in the filtered prompt word while maintaining the jailbreak condition, so as to generate the entry restoration prompt word;

[0097] An optimized prompt word generation module, configured to iteratively optimize the entries in the entry restoration prompt word according to the optimization order of the entries in the entry restoration prompt word, and generate an optimized prompt word that satisfies the prompt word semantic preservation and jailbreak conditions;

[0098] The alignment mechanism enhancement module is used to perform adversarial training on the target large language model using optimized cue words that meet the cue word semantic preservation and jailbreak conditions, thereby enhancing the alignment mechanism of the target large language model.

[0099] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0100] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A large language model alignment enhancement method for optimizing the semantics of prompt words, characterized by: The method comprises: Step 1: perform part-of-speech filtering on the original prompt words to obtain filtered prompt words; Step 2: Generate a synonym set corresponding to each term in the filter prompt word; Step 3: Replace the terms in the original prompt word with terms randomly selected from the synonym set to generate an initial modulated prompt word that meets the jailbreak condition; Step 4, record the terms that have changed in the initial modulation cue words compared to the original cue words, and construct a list of changed terms; Step 5, while maintaining the jailbreak condition, replacing the changed terms in the initial modulation prompt words with the terms in the filtered prompt words to generate the term restoration prompt words; Step 6, iteratively optimizing the entries in the entry restoration prompt words according to the optimization order of the entries in the entry restoration prompt words, and generating optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions; Step 7: Use the optimized cue words that meet the cue word semantic preservation and jailbreak conditions to perform adversarial training on the target large language model.

2. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 1 is characterized in that: Perform part-of-speech filtering on the original prompt words to obtain filtered prompt words, including: Step 11: The attacker has the original prompt word x=[x1,x2,...,x n ], which is a jailbreak attack instruction containing n entries, x u is the entry in the original prompt word x; Step 12: Filter the original prompt word x whose part of speech is not adjective, adverb, verb or noun, and obtain the filtered prompt word w = [w1, w2, ..., w m ], where m is the length of the filter prompt word, and m≤n.

3. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 1 is characterized in that: Generate a synonym set corresponding to each term in the filter prompt, including: Step 21: filter each entry w in the prompt word w i Generate entries containing w i The synonym set S(w i ), where i∈{1,2,...,m}; Step 22, calculate the synonym set S(w i ) Size L i =|S(w i )|, and remember is a synonym set S(w i ) in the jth entry.

4. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 3 is characterized in that: Replace the original prompt words with randomly selected words from the synonym set to generate the initial modulation prompt words that meet the jailbreak conditions, including: Step 31: Randomly select a term from the synonym set Replace the term x in the original prompt word x u , where the term x u Corresponding to the entry w in the filter prompt word w i ; Step 32, repeat step 31 until the initial modulation prompt word is generated Meet the jailbreak conditions in, To input the initial modulation prompt word to the target large language model M The output response is A function to determine whether the jailbreak conditions are met. The establishment condition is that the target large language model responds The default rejection terms are not included.

5. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 1 is characterized in that: Record the terms that have changed in the initial modulation cue words compared to the original cue words, and construct a list of changed terms, including: Step 41: record the initial modulation prompt word compared with the original prompt word x. Entries that have changed in σ(i) , where i∈{1,2,...,q}; Step 42: Construct a list of changed terms S = [s σ(1) ,s σ(2) ,...,s σ(q) ].

6. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 5 is characterized in that: While maintaining the jailbreak condition, the changed terms in the initial modulation prompt words are replaced with terms in the filtered prompt words to generate the term restoration prompt words, including: Step 51, let the changed term list traverse index i←1 and the prompt word optimization iteration number t←0; Step 52, use the term w σ(i) Replace prompt word Entries in To get the prompt word The term w σ(i) To filter the entries with index σ(i) in the prompt word w; Step 53, if jailbreak conditions Satisfaction, that is, the target large language model responds Does not contain the preset rejection terms, then let And t←t+1; Step 54, let i←i+1; Step 55, repeat steps 52 to 54 until the list of changed terms is traversed to generate the term restoration prompt word 7. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 6 is characterized in that: According to the optimization order of the terms in the term restoration prompt words, the terms in the term restoration prompt words are iteratively optimized to generate optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions, including: Step 61, calculate the entry optimization order weight, which can be expressed as: where {σ(1),σ(2),...,σ(q)} is the difference between the initial modulated prompt word and the original prompt word x. The index of the terms that have changed in Calculate terms for the all-mpnet-base-v2 model With the entry The function of semantic similarity, Restore prompt words for entries The optimized order weight of the σ(i)th term in ; Step 62, Perform descending sort, i.e. To obtain the optimized order list O = [ρ(1), ρ(2), ..., ρ(q)], where ρ(1) is the first optimized change term index, ρ(2) is the second optimized change term index, and ρ(q) is the last optimized change term index; Step 63, let the optimized order list traverse index i←1; Step 64: Use synonyms one by one Replace prompt word The ρ(i)th term in Generate L ρ(i) Modulation cue words where k∈{1,2,...,L ρ(i) },S(w ρ(i) ) is the entry w with index ρ(i) in the filter prompt word w ρ(i) The corresponding synonym set; Step 65, under jailbreak conditions If the condition is met, find the prompt word with the highest semantic similarity to the original prompt word And order And t←t+1, where the jailbreak condition Refers to the target large language model response Does not contain preset rejection terms; in, To calculate the prompt word x and the prompt word using the all-mpnet-base-v2 model Function of semantic similarity; Step 66, let i←i+1; Step 67, repeating steps 64 to 66 until the optimized sequence list is traversed to generate optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions 8. The large language model alignment enhancement method for optimizing the semantics of prompt words according to claim 1, characterized in that: Use optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions to conduct adversarial training on the target large language model, including: Step 71: Given N original prompt words x r ,r∈{1,2,...,N}, use steps 1 to 6 to generate optimized prompt words that meet the semantic preservation and jailbreak conditions of prompt words Step 72: For optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions Set up security responses based on pre-set rejection terms Step 73: Use optimized prompt words that meet the prompt word semantic preservation and jailbreak conditions respectively and security response As the input and output of the target large language model M, the target large language model M is fine-tuned and trained.

9. A large language model alignment enhancement system for optimizing the semantics of prompt words, characterized by: The method for implementing claims 1 to 8 comprises: A part-of-speech filtering module is used to perform part-of-speech filtering on the original prompt words to obtain filtered prompt words; A synonym set generation module is used to generate a synonym set corresponding to each entry in the filter prompt word; An initial modulation prompt word generation module is used to replace the entries in the original prompt word with randomly selected entries in the synonym set to generate an initial modulation prompt word that meets the jailbreak condition; a changed word list construction module, used for recording the words that have changed in the initial modulation prompt words compared with the original prompt words, and constructing a changed word list; The entry restoration prompt word generation module is used to replace the changed entry in the initial modulation prompt word with the entry in the filtered prompt word while maintaining the jailbreak condition, so as to generate the entry restoration prompt word; An optimized prompt word generation module, configured to iteratively optimize the entries in the entry restoration prompt word according to the optimization order of the entries in the entry restoration prompt word, and generate an optimized prompt word that satisfies the prompt word semantic preservation and jailbreak conditions; The alignment mechanism enhancement module is used to perform adversarial training on the target large language model using optimized cue words that meet the cue word semantic preservation and jailbreak conditions.

Images