Generative model safety capability improvement method and device based on thinking chain reinforcement

Abstract

The invention discloses a method and a device for improving safety capability of a generative model based on thinking chain reinforcement, and formulates a two-stage training framework in order to reinforce a generation path of a thinking chain. In the first stage, thinking data generated by the model is utilized to carry out supervised fine tuning on a target model to limit an action space of the model in a formatted reasoning space; in the second stage, in order to enhance the reasonability of the reasoning process in the first stage, the reasoning action of the model is doubly scored at the statement level and the token level so as to screen out offline data to further reinforce learning of the target model. The method has the following advantages that the generalization ability of the model in different attack environments is greatly improved by enhancing the thinking ability of the model; compared with a traditional reward model, a double-level scoring mechanism adopted by the method can efficiently evaluate the model thinking process while keeping the fine granularity of reward signals.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, particularly to the generation of thought chains in generative models in natural language processing and the testing of model security capabilities. In particular, it relates to a method and device for enhancing the security capabilities of generative models based on thought chain reinforcement. Background Art

[0002] Generative models are now widely used in various industries, and how to ensure that the generated content of the model is helpful and harmless is an important issue. Many studies have shown that even after alignment with human values, generative models can still be induced by carefully constructed "jailbreak attacks" to produce content that deviates from the original purpose of use or even is harmful, which also raises concerns about model abuse.

[0003] Currently, common security strategies can generally be divided into two types: detection-based methods and mitigation-based methods. Detection-based methods rely on detectors external to the model, that is, to judge the maliciousness of the input or output to prevent the model from responding to requests. However, such methods have high requirements for the detector. Training an efficient detector often requires high resources, and due to the "static" nature of this training, it is difficult for the detector to adapt to complex and changing external inputs, so it has high limitations. Mitigation-based methods often perform post-training on the input-output process or on the model itself to enhance security. If its security mechanism is too simple, it may be difficult to achieve a good enhancement effect, and if the mechanism is too strict, it may directly damage the usability of the model, resulting in more losses than gains. Therefore, effective mitigation-based methods often need to have a reasonable security mechanism while minimizing damage to usability. Summary of the Invention

[0004] Aiming at the technical problems existing in the prior art, the present invention discloses a method and device for enhancing the security capabilities of generative models based on thought chain reinforcement. The method of the present invention greatly enhances the generalization ability of the model in different attack environments by enhancing the model's thinking ability; compared with traditional reward models, the two-level scoring mechanism adopted by the present invention can efficiently evaluate the model's thinking process while maintaining the fine-grainedness of the reward signal.

[0005] To achieve the above object, the technical solutions adopted by the present invention include:

[0006] In a first aspect, a method for enhancing the security capabilities of a generative model based on thought chain reinforcement is provided, including:

[0007] Using prompt engineering to synthesize instruction data pairs containing predefined inference paths to fine-tune the generative model, so that the generative model obtains initial security inference capabilities;

[0008] Generate attack data based on benign and malicious request data, using attack templates in the field and the introduced attacker model;

[0009] Generate N replies for each piece of attack data, evaluate each reply, construct positive and negative sample pairs based on the evaluation results, and input the constructed positive and negative sample pairs into the generative model to strengthen the model's thinking ability.

[0010] In one implementation, use prompt engineering to synthesize instruction data containing predefined reasoning paths to fine-tune the generative model, including:

[0011] Collect benign and malicious request data in the field, input each request data into the generative model, and use prompt engineering to guide the model to generate predefined reasoning paths during the analysis process;

[0012] Input each request data into the generative model again and directly generate the corresponding reply;

[0013] According to the predefined reasoning paths and the generated replies, obtain the instruction data synthesized by prompt engineering that contains predefined reasoning paths;

[0014] Use the instruction data to perform supervised fine-tuning on the initial generative model.

[0015] In one implementation, according to the predefined reasoning paths and the generated replies, obtain the instruction data synthesized by prompt engineering that contains predefined reasoning paths, including:

[0016] Use a string matching-based method to match the reasoning paths corresponding to each request data and the directly generated replies, eliminate the data with semantic conflicts in the reasoning paths and the directly generated replies, and splice the remaining reasoning paths and directly generated replies to obtain the instruction data synthesized by prompt engineering that contains predefined reasoning paths, as the formatted thought chain instruction fine-tuning data.

[0017] In one implementation, the fine-tuned model can generate three parts of content for each input request, namely, a thinking part for security assessment of the input, a decision-making part for making a final decision on whether to respond to the request, and a reply part for presenting the final output.

[0018] In one implementation, based on benign and malicious request data, generate attack data using attack templates in the field and the introduced attacker model, including:

[0019] Collect attack templates in the field and introduce an attacker model;

[0020] For each piece of malicious request data, combine it with a randomly selected attack template to generate initial attack data, and use the attacker model to paraphrase randomly selected parts of each initial attack data and randomly insert noise symbols to generate attack data corresponding to the malicious request data;

[0021] For each piece of benign request data, use the attacker model to construct a complex scenario to hide the true intention and randomly add noise symbols to generate attack data corresponding to the benign request data.

[0022] In one implementation, generate N reply data for each piece of attack data, evaluate each piece of reply data, and construct positive and negative sample pairs based on the evaluation results, including:

[0023] Generate N different reply data for each piece of attack data with a high-diversity generation setting;

[0024] Use the designed evaluation function to score the N reply data respectively. Among them, the reply with the highest score is the positive sample, and the reply with the lowest score is the negative sample. Set a critical value during evaluation. When the score difference between the positive and negative samples is greater than the critical value, retain the attack data, positive sample, and negative sample as one piece of data. Otherwise, re-enter the original request data into the model to generate attack data and positive and negative sample data.

[0025] In one implementation, use the designed evaluation function to score the N reply data respectively, including:

[0026] Take each reply generated by the generative model for the attack data and the original request data included in the attack data as the input of the evaluation function, where the reply contains text information and the probability distribution corresponding to the generation of each token;

[0027] At the sentence level, calculate the similarity between the text of the thinking part in the reply and the original request data in the attack data to obtain the sentence-level score;

[0028] At the token level, for tokens containing format information, score the tokens according to whether the model outputs the corresponding format tokens at the corresponding positions. For tokens containing decision information, score the tokens according to whether the decision of the model is consistent with the input request data;

[0029] Sum the evaluation scores at the sentence level and the token level to obtain the evaluation score of the evaluation function for the entire reply.

[0030] Based on the same inventive concept, the second aspect of the present invention provides a device for enhancing the security capabilities of a generative model based on the reinforcement of the chain of thought, including:

[0031] An initial security reasoning ability acquisition module, which is used to fine-tune a generative model by using prompt engineering to synthesize instruction data containing predefined reasoning paths, so that the generative model obtains initial security reasoning ability;

[0032] An attack data generation module, which is used to generate attack data based on benign and malicious request data by using attack templates in the field and an introduced attacker model;

[0033] An evaluation and reinforcement module, which is used to generate N replies for each piece of attack data, evaluate each reply, construct positive and negative sample pairs based on the evaluation results, input the constructed positive and negative sample pairs into the generative model, and strengthen the thinking ability of the model.

[0034] Based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method for enhancing the security ability of a generative model based on thought chain reinforcement described in the first aspect.

[0035] Based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the method for enhancing the security ability of a generative model based on thought chain reinforcement described in the first aspect.

[0036] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0037] The present invention provides a method for enhancing the security ability of a generative model based on thought chain reinforcement. First, use prompt engineering to synthesize instruction data containing predefined reasoning paths to fine-tune the generative model, so that the generative model obtains initial security reasoning ability; then generate attack data based on benign and malicious request data by using attack templates in the field and an introduced attacker model; generate N replies for each piece of attack data, evaluate each reply, construct positive and negative sample pairs based on the evaluation results, input the constructed positive and negative sample pairs into the generative model, and strengthen the thinking ability of the model. The present invention uses the thinking data generated by the model itself to perform supervised fine-tuning on the target model, restricting the action space of the model in a formatted reasoning space. In order to enhance the rationality of the reasoning process in the first stage, in the second stage, the present invention performs double scoring on the reasoning actions of the model at the sentence level and the token level to screen out offline data and perform further reinforcement learning on the target model. The present invention has the following advantages: the enhancement of the model's thinking ability greatly improves its generalization ability in different attack environments; compared with the traditional reward model, the double-level scoring mechanism adopted by the present invention can efficiently evaluate the model's thinking process while maintaining the fine-grainedness of the reward signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is the overall flowchart of the method for enhancing the security capabilities of the generative model based on the reinforcement of the chain of thought in the embodiments of the present invention;

[0040] Figure 2 It is the flowchart of generating positive and negative sample data in the embodiments of the present invention;

[0041] Figure 3 It is the flowchart of scoring the model responses using the evaluation function in the embodiments of the present invention;

[0042] Figure 4 It is the module diagram of the device for enhancing the security capabilities of the generative model based on the reinforcement of the chain of thought in the embodiments of the present invention. Detailed implementation manners

[0043] The present invention proposes a method for enhancing the security capabilities of the generative model based on the reinforcement of the chain of thought. For a given malicious input or malicious attack against the generative model, the generative model's thinking and reasoning can be used to identify the core intention of the request and infer the relevant consequences to analyze whether the input content is harmful. The advantages of the present invention include: the enhancement of the model's thinking ability greatly improves its generalization ability in different attack environments; compared with the traditional reward model, the two-level scoring mechanism adopted by the present invention can efficiently evaluate the model's thinking process while maintaining the fine-grainedness of the reward signal.

[0044] To achieve the above technical effects, the main concept of the present invention includes: to strengthen the generation path of the chain of thought, the present invention formulates a two-stage training framework. In the first stage, the target model is supervised and fine-tuned using the thinking data generated by the model itself to limit the action space of the model in the formatted reasoning space. To enhance the rationality of the reasoning process in the first stage, in the second stage, the present invention double-scores the model's reasoning actions at the sentence level and the token level to screen out the offline data for further reinforcement learning of the target model.

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Embodiment 1

[0047] The present invention proposes a method for enhancing the security capabilities of a generative model based on the reinforcement of the chain of thought, including:

[0048] S1: Use prompt engineering to synthesize instruction data containing predefined reasoning paths to fine-tune the generative model, so that the generative model obtains initial security reasoning capabilities.

[0049] Specifically, for the acquisition of the initial security reasoning capabilities of the S1 model, the generative language model obtained through the general training steps does not have direct thinking capabilities. To improve its security analysis capabilities for requests, the present invention uses the generated instruction data to fine-tune the model, so that the model obtains initial security reasoning capabilities (thinking capabilities).

[0050] The synthesis of instruction data containing predefined reasoning paths using prompt engineering is obtained by using the reasoning paths generated by prompt engineering and the responses directly obtained after the request data is input into the model.

[0051] S2: Based on benign and malicious request data, use attack templates within the domain and the introduced attacker model to generate attack data.

[0052] Specifically, for the generation of attack data in S2, after the operation of S101, the model can now perform security analysis on the input requests, and general input models can correctly identify them. Therefore, "hard samples" are needed to promote the reinforcement of the thinking process.

[0053] S3: Generate N responses for each piece of attack data, evaluate each response, construct positive and negative sample pairs based on the evaluation results, and input the constructed positive and negative sample pairs into the generative model to strengthen the thinking capabilities of the model.

[0054] Specifically, for the evaluation and reinforcement of the model's reasoning actions in S3, in order to obtain offline data for further strengthening the chain of thought, the present invention generates N response paths for each piece of attack data and selects the optimal and worst paths, and then uses an optimization algorithm to make the probability generated by the model tend to the optimal path as much as possible and away from the worst path, thereby strengthening the thinking capabilities of the model.

[0055] S1 can be implemented in the following manner:

[0056] S1.1: Collect the benign and malicious request data in the domain, input each request data into the generative model, and guide the model to generate a predefined inference path during the analysis through prompt engineering.

[0057] In the specific implementation process, collect the benign and malicious request data Q = {q1, q2, …, q n} in the domain, input each request data q i into the model, and guide the model to generate a predefined security inference path c i = {s1, s2, s3} during the analysis through prompt engineering, where s1: make an initial judgment on the request and analyze the rationality of the request; s2: analyze whether the response to the request contains possible harmful content; s3: make a judgment on whether to respond to the request based on s1, s2 and the overall tone of the request. The formal expression of the security inference path c i is:

[0058] c i = Model.generate(prompt(q i ))

[0059] where Model is the target model, the generate method is for the model to generate the corresponding output after receiving the input, and prompt(q i ) is the prompt input constructed according to the content of {s1, s2, s3} above for q i . Finally, the set of thinking data obtained is C = {c1, c2, …, c n}.

[0060] S1.2: Input each request data into the generative model again and directly generate the corresponding response.

[0061] In the specific implementation process, input each request data q i into the model again and directly generate the corresponding response r i . That is, here

[0062] r i = Model.generate(q i )

[0063] Finally, the set of responses obtained is R = {r1, r2, …, r n}.

[0064] S1.3: According to the predefined inference path and the generated response, obtain the instruction data synthesized using prompt engineering that contains the predefined inference path.

[0065] Further, a method based on string matching is used to match the inference path corresponding to each request data with the directly generated reply, and the data with semantic conflicts in the inference path and the directly generated reply is eliminated. The remaining inference path and the directly generated reply are concatenated to obtain instruction data synthesized using prompt engineering and containing a predefined inference path, which is used as formatted thought chain instruction fine-tuning data.

[0066] In the specific implementation process, a method based on keyword matching is used to perform screening on each corresponding c i ,r i to eliminate the c i and r i data with semantic conflicts in the original data to obtain C′ (the set of screened thought data) and R′ (the set of screened replies), and then C′ and R′ are concatenated to obtain formatted thought chain instruction fine-tuning data I = {Q, C′ + R′}, where the + sign represents concatenation.

[0067] S1.4: Use the instruction data to perform supervised fine-tuning on the initial generative model.

[0068] Specifically, after fine-tuning, the model can generate three parts of content for each input request: the "thinking part" performs a security assessment on the input; the "decision part" makes a final decision on whether to respond to the request; the "reply part" displays the final output.

[0069] S2 can be implemented in the following way:

[0070] S2.1: Collect attack templates in the field and introduce an attacker model.

[0071] In the specific implementation process, a small number of "attack templates" (such as 10) in the field are collected. The attack template can fill in the request data to hide the true request. Benign and malicious request data Q′ = {q′1, q′2, …, q′ n}(here Q′ can directly use the Q dataset in the previous text or additional request data), and an attacker model Attacker (which can be another generative model) is introduced.

[0072] S2.2: For each malicious request data, combine it with a randomly selected attack template to generate initial attack data, and randomly paraphrase some statements of each initial attack data through the attacker model and randomly insert noise symbols to generate attack data corresponding to the malicious request data.

[0073] Specifically, this step is used to construct a complex scenario to hide the true purpose of the request. For each request q′ i , if the request is a malicious request, q′ iGenerate the initial attack data m_attack in combination with an arbitrarily selected attack template i , and let the attacker model randomly paraphrase some sentences of each attack data and randomly insert noise symbols to increase data diversity, that is

[0074] m_atack i ←Attacker.paraphrase(m_attack i )

[0075] where the paraphrase method enhances the input attack data through prompt engineering.

[0076] S2.3: For each piece of benign request data, use the attacker model to construct a complex scenario for it to hide the true intention and randomly add noise symbols to generate attack data corresponding to the benign request data.

[0077] In the specific implementation process, it is achieved through the following methods:

[0078] b_attack i =Attacker.construct_scene(q′ i )

[0079] where the construct_scene method also uses prompt engineering to construct a complex context scenario for the input benign request, and b_attack i represents the attack data generated by the benign request.

[0080] In S3, generate N replies for each piece of attack data, evaluate each reply, and construct positive and negative sample pairs based on the evaluation results, which can be achieved through the following methods:

[0081] Generate N different reply data for each piece of attack data with a high-diversity generation setting;

[0082] Use the designed evaluation function to score the N reply data respectively. Among them, the reply with the highest score is the positive sample, and the reply with the lowest score is the negative sample. When setting the critical value during evaluation, when the score difference between the positive and negative samples is greater than the critical value, retain the attack data, positive sample, and negative sample as one piece of data, otherwise re-enter the original request data into the model to generate attack data and positive and negative sample data.

[0083] Specifically, 1) The model generates N reply data response i for each piece of attack data attack i ={r1,r2,…,r N}, which is used to select the highest-score reply and the lowest-score reply for subsequent evaluation.

[0084] 2) The present invention designs an evaluation function that is lightweight and can provide a high-fine-grained reward signal, evaluating each response at the sentence level and the token level. At this time, for the response i Obtain the scores score of each of the N responses i ={s1, s2, …, s N}.

[0085] 3) For the response i Select the responses pos with the highest and lowest scores i = response(argmax(score i ))), and the response neg with the lowest score i = response(argmin(score i ))). Construct all positive and negative sample pairs Pairs = {{attack1, pos1, neg1}, {attack2, pos2, neg2}, …, {attack n , pos n , neg n}}.

[0086] 4) Use an optimization algorithm to fine-tune the model, making it tend to generate pos i given attack i , and move away from neg i} to achieve the effect of enhancing the model's security inference ability.

[0087] The present invention enhances the security performance of the model based on enhancing the model's thinking ability. The present invention will be explained in detail below with reference to the accompanying drawings. I. Combine the attached Figure 1 Introduce the overall process of specifically implementing the present invention

[0088] Step 1. Collect data in the field of generative model security on websites such as Huggingface and github, including malicious request data of various types and a comparable amount of benign request data. Input the request data into the model in two times. The first time, use "prompt engineering" to guide the model to generate a predefined security inference path {s1, s2, s3}, where s1: the initial judgment of the request, analyzing the rationality of the request; s2: analyzing whether the response to the request contains possible harmful content; s3: making a judgment on whether to respond to the request based on s1, s2 and the overall tone of the request; the second time, directly input the request to obtain the direct response data of the request.

[0089] Step 2. Use a string-matching-based method to filter out instruction data where the thought path conflicts with the direct response semantics, such as: data where the thought path determines that the request needs to be rejected but the direct response does not reject, and data where the thought path determines that an answer can be given but the response rejects. Concatenate the remaining data in a fixed format to obtain instruction data. Formally, this format consists of three parts: the "thinking part" assesses the security of the input; the "decision part" makes a final decision on whether to respond to the request; the "response part" presents the final output. Use the instruction data to fine-tune the model so that it obtains initial thinking ability. At this time, the optimization objective can be described as

[0090] L(x 1:n ,y)=-logp πθ (y|x 1:n )

[0091] where x 1:n represents the input instruction, and y represents the thought chain data for the response to the instruction.

[0092] Step 3: Use a generative model as the attacker model. Also use the request data collected in Step 1, and collect attack template data within the domain. For malicious requests, randomly select an attack template data, and hide the malicious request in it to form initial attack data; for benign requests, use prompt engineering to make the attacker model construct a complex scenario for the request and hide the request in it to form initial attack data. The attacker model randomly selects some sentences from the initial attack data for restatement and then inserts meaningless noise into any sentence to ensure data diversity, resulting in attack data.

[0093] Step 4: For each attack data, the target model generates multiple responses. To select the best and worst responses among the multiple responses, the present invention designs an evaluation function. At the sentence level, use a word frequency statistics-based method to score the model's ability to analyze the true intention of the input in the "thinking part"; at the token level, extract the generation probabilities of key tokens with format information and decision information, and score this token according to the request category and the token probability. Weight the sum of the sentence-level score and the token-level score as the overall score of the response, and select the responses with the highest and lowest scores as positive and negative sample pairs, and use this data to strengthen the thinking ability of the model. The optimization objective can be described as

[0094]

[0095] where x 1:n represents the input sequence of the model, y pos represents the positive sample, y neg represents the negative sample, σ represents the sigmoid function, β is a training hyperparameter, π represents the target model being trained, π refThe representative reference model is used to limit the training step size to mitigate reward hacking.

[0096] II. Combination Figure 2 Introduce the process of specifically constructing positive and negative sample data

[0097] Step 1. In the black-box attack on the generative model, a common strategy is to construct a context scenario to conceal the true purpose of the input. Therefore, when generating the initial attack data, for malicious requests, randomly select an attack template and embed the request to hide the request; for benign requests, use prompt engineering to make the attacker model construct a complex scenario for the request to hide the request. The attacker model randomly selects some statements of the initial attack data, paraphrases these statements into a seemingly harmless form, and finally inserts meaningless characters as noise in the random fields. At this time, the attack data is obtained.

[0098] Step 2. When the model with initial thinking ability faces a weak attack, its reply has a high probability of being a correct reply, while when facing a strong attack, its reply has a high probability of being an incorrect reply. To sample different paths under a given state simultaneously, for each attack data, the target model generates N different reply data with a high diversity of generation settings (e.g., temperature = 2.0; top_p = 1.0; top_k = 1).

[0099] Step 3. Use the designed evaluation function to score each of the N reply data. Since the subsequent fine-tuning goal is to increase the probability of the model generating positive samples and decrease the probability of generating negative samples, in order to ensure the consistency of the optimization direction, there should preferably be an obvious gap between the highest-score reply (positive sample) and the lowest-score reply (negative sample). Set a critical value during the evaluation. When the score difference between the positive and negative samples is greater than the critical value, retain the attack data, positive sample, and negative sample as one piece of data. Otherwise, repeat Steps 1 to 3 to regenerate the attack data and positive and negative sample data from the original request data.

[0100] III. Combination Appendix Figure 3 Introduce the process of using the evaluation function to score the model's replies

[0101] Step 1. Take each reply generated by the model for the attack data and the original request data included in the attack data as the input of the function. Among them, the model's reply contains text information and the probability distribution corresponding to each token during generation. Next, the evaluation function scores this reply at the sentence and token levels.

[0102] Step 2. At the sentence level, the attacker model mentioned above needs to hide the true intention of the request, while here it is necessary to evaluate whether the model can truly infer the true intention of the input during the inference process. Here, the "thinking part" text in the reply (designated as D) needs to be intercepted and compared with the original request data in the attack data (designated as q) for similarity. The higher the similarity, the more the model can discover the true intention of the input, so the more correct the inference process of the model and the higher the score, and vice versa. The present invention uses a sliding window plus TF-IDF strategy for similarity calculation. Calculate the document length l of the original request, and divide D into multiple sub-document sets {d1, d2, …, d n} with a length of l and an overlapping area of h. Calculate the similarity between any sub-document d l and q, count the vocabulary sizes of d l and q, and calculate the TF-IDF value of each word in the vocabulary for each of the two documents. At this time, two semantic vectors v(d l ) and v(q) with a length equal to the vocabulary size are obtained. The similarity calculation is s i = cos(v(d l ), v(q)), and take the sentence-level evaluation score S = max(s1, s2, …, s n ) as the overall similarity at the sentence level.

[0103] Step 3. At the token level, in order to provide a fine-grained reward signal, the evaluation function evaluates the "key tokens" containing format information or decision information. When the generative model generates text, the generation of the next token depends on all the previous tokens. Therefore, the probability distribution on the vocabulary when a token is generated contains the information state of the previous context.

[0104] For tokens containing format information, if the model outputs the corresponding format token at the corresponding position, the token is regarded as correct, otherwise it is incorrect; for tokens containing decision information, when the input is a malicious / benign request, if the model's decision is to "reject" / "reply" to the request, the token is regarded as correct, otherwise it is incorrect. The following scores are given to these key tokens respectively:

[0105]

[0106] Among them, i represents the token actually generated by the model at the position of the "key token", t represents the correct token in the above rules, t iRepresents the score obtained for this key token. If the correct token is generated (i.e., i = t), the larger the probability p(i), the more "confident" the model's output of the correct token is considered, and the higher the score at this time. However, if the generation of the "key token" is incorrect (i.e., i ≠ t), then at the corresponding position, the probability of the remaining tokens other than the correct token is viewed, that is, the size of 1 - p(t). At this time, the meaning of the corresponding token is the "confidence" of the model in making mistakes. Therefore, the larger the probability, the greater the "error" of the previously generated content, and the evaluation score of this token is t i = -1 * (1 - p(t))) = p(t) - 1. Obtain the overall token-level score T = Σt i .

[0107] Step 4. Sum the evaluation scores of the two levels to obtain the evaluation score of the entire reply by the evaluation function Score = S + T.

[0108] Embodiment 2

[0109] Based on the same inventive concept, this embodiment discloses a device for enhancing the security capabilities of a generative model based on chain-of-thought reinforcement. Please refer to Figure 4 , including:

[0110] An initial security reasoning ability acquisition module 101, configured to use prompt engineering to synthesize instruction data containing predefined reasoning paths to fine-tune the generative model, so that the generative model obtains initial security reasoning ability;

[0111] An attack data generation module 102, configured to generate attack data based on benign and malicious request data, using attack templates in the field and the introduced attacker model;

[0112] An evaluation and reinforcement module 103, configured to generate N replies for each piece of attack data, evaluate each reply, construct positive and negative sample pairs based on the evaluation results, and input the constructed positive and negative sample pairs into the generative model to strengthen the thinking ability of the model.

[0113] Since the device introduced in Embodiment 2 of the present invention is the device adopted for implementing the method for enhancing the security capabilities of a generative model based on chain-of-thought reinforcement in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device adopted in the method of Embodiment 1 of the present invention falls within the scope of protection of the present invention.

[0114] Embodiment 3

[0115] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in Embodiment 1.

[0116] Since the computer-readable storage medium introduced in the third embodiment of the present invention is the computer-readable storage medium used in the method for enhancing the security capabilities of a generative model based on thought chain reinforcement in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium, so it will not be elaborated here. Any computer-readable storage medium used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0117] Embodiment 4

[0118] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for enhancing the security capabilities of a generative model based on thought chain reinforcement described in Embodiment 1.

[0119] Since the device introduced in the fourth embodiment of the present invention is the computer device used in the method for enhancing the security capabilities of a generative model based on thought chain reinforcement in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of this computer device, so it will not be elaborated here. Any computer device used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0120] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0121] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0122] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A method for enhancing the security capabilities of a generative model based on chain-of-thought reinforcement, characterized in that Including: Using prompt engineering to synthesize instruction data containing predefined inference paths to fine-tune a generative model, enabling the generative model to obtain initial secure inference capabilities; Based on benign and malicious request data, using attack templates within the domain and an introduced attacker model to generate attack data; Generating N replies for each piece of attack data, evaluating each reply, constructing positive and negative sample pairs based on the evaluation results, and inputting the constructed positive and negative sample pairs into the generative model to enhance the model's thinking ability.

2. The method for enhancing the security capability of the generative model based on thought chain reinforcement according to claim 1, wherein, Using prompt engineering to synthesize instruction data containing predefined inference paths to fine-tune a generative model, including: Collecting benign and malicious request data within the domain, inputting each request data into the generative model, and guiding the model to generate predefined inference paths during the analysis process through prompt engineering; Inputting each request data into the generative model again and directly generating a corresponding reply; Based on the predefined inference paths and the generated replies, obtaining instruction data synthesized using prompt engineering that contains predefined inference paths; Using the said instruction data to perform supervised fine-tuning on the initial generative model.

3. The method for enhancing the security capability of the generative model based on thought chain reinforcement according to claim 2, wherein Based on the predefined inference paths and the generated replies, obtaining instruction data synthesized using prompt engineering that contains predefined inference paths, including: Using a string matching-based method to match the inference paths corresponding to each request data and the directly generated replies, eliminating data with semantic conflicts between the inference paths and the directly generated replies, and splicing the remaining inference paths and directly generated replies to obtain instruction data synthesized using prompt engineering that contains predefined inference paths, as formatted thought chain instruction fine-tuning data.

4. The method for enhancing the security capability of the generative model based on thought chain reinforcement according to claim 3, wherein The fine-tuned model can generate three parts of content for each input request, namely, a thinking part for security assessment of the input, a decision part for making a final decision on whether to respond to the request, and a reply part for presenting the final output.

5. The method for enhancing the security capability of the generative model based on the reinforcement of the thought chain according to claim 1, wherein, Based on benign and malicious request data, using attack templates within the domain and an introduced attacker model to generate attack data, including: Collecting attack templates within the domain and introducing an attacker model; For each piece of malicious request data, combining it with a randomly selected attack template to generate initial attack data, and paraphrasing randomly selected parts of each initial attack data and randomly inserting noise symbols through the attacker model to generate attack data corresponding to the malicious request data; For each piece of benign request data, using the attacker model to construct a complex scenario to hide the true intention and randomly adding noise symbols to generate attack data corresponding to the benign request data.

6. The method for enhancing the security capability of a generative model based on thought chain reinforcement according to claim 4, wherein Generating N reply data for each piece of attack data, evaluating each reply data, and constructing positive and negative sample pairs based on the evaluation results, including: Generating N different reply data for each piece of attack data with a high-diversity generation setting; Score the N reply data respectively using the designed evaluation function. Among them, the reply with the highest score value is the positive sample, and the reply with the lowest score value is the negative sample. Set a critical value during evaluation. When the score difference between the positive and negative samples is greater than the critical value, retain the attack data, positive sample, and negative sample as one piece of data. Otherwise, re-enter the original request data into the model to generate attack data and positive and negative sample data.

7. The method for improving the security capability of the generative model based on thought chain reinforcement according to claim 6, wherein Score the N reply data respectively using the designed evaluation function, including: Use each reply generated by the generative model for the attack data and the original request data included in the attack data as the input of the evaluation function, where the reply contains text information and the probability distribution corresponding to the generation of each token; At the sentence level, calculate the similarity between the text of the thinking part in the reply and the original request data in the attack data to obtain the sentence-level score; At the token level, for tokens containing format information, score the tokens according to whether the model outputs the corresponding format tokens at the corresponding positions. For tokens containing decision information, score the tokens according to whether the decision of the model is consistent with the input request data; Sum the evaluation scores at the sentence level and the flat boundary level to obtain the evaluation score of the evaluation function for the entire reply.

8. An apparatus for enhancing the security capabilities of a generative model based on chain-of-thought reinforcement, characterized in that, Including: An initial security reasoning ability acquisition module for fine-tuning the generative model using instruction data synthesized by prompt engineering that contains predefined reasoning paths, so that the generative model obtains initial security reasoning ability; An attack data generation module for generating attack data based on benign and malicious request data, using attack templates within the domain and the introduced attacker model; An evaluation and reinforcement module for generating N replies for each attack data, evaluating each reply, constructing positive and negative sample pairs based on the evaluation results, and inputting the constructed positive and negative sample pairs into the generative model to strengthen the thinking ability of the model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for enhancing the security ability of the generative model based on thought chain reinforcement as described in any one of claims 1 to 7.

10. A computer device, characterized in that, Including: Including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for enhancing the security ability of the generative model based on thought chain reinforcement as described in any one of claims 1 to 7.

Images