Safety guardrail enhancement method and device for text generation large model and computer equipment

Abstract

The invention relates to a safety guardrail enhancement method and device for a text generation large model and computer equipment, and the method comprises the steps: training a predefined combined guardrail based on a preset training data set, and the training process comprises the steps: through each guardrail model in the combined guardrail, carrying out the training of the combined guardrail; identifying a sample text generation prompt in the training data set to obtain an output result corresponding to each guardrail model; dynamically adjusting the weight corresponding to each guardrail model according to each output result; processing an input text generation prompt through the trained combined guardrail to obtain a decision result; the decision result is determined by the processing result of each guardrail model for the text generation prompt and the weight corresponding to each guardrail model after training. According to the method and the device, the problem that a large model security fence is insufficient in defense capability and cannot effectively cope with diversified jailbreak attacks is solved, and the defense capability of the large model security fence is improved so as to effectively cope with the diversified jailbreak attacks.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and computer equipment for enhancing safety guardrails of large text-generated models. Background Art

[0002] At present, guardrail technology plays a vital role in the security protection of large language models. For example, large text generation models are used in literary creation, copywriting, and strategy formulation. Guardrail defense mechanisms are usually used to identify and respond to various attacks to prevent harmful content output and ensure information security. However, when facing various prompt word jailbreak attacks, the guardrail defense capabilities of large language models are insufficient, that is, the existing guardrail defense mechanism only performs well in specific areas. When the attack form changes or cross-domain attacks occur, its defense capability decreases significantly and cannot effectively respond to diverse jailbreak attacks.

[0003] There is currently no effective solution to the problem that large-model security guardrails in related technologies are insufficient in defense capabilities and cannot effectively respond to diverse jailbreak attacks. Summary of the invention

[0004] In this embodiment, a method, device and computer equipment for enhancing the security fence of a large text-generated model are provided to solve the problem that the large model security fence in the related art has insufficient defense capability and cannot effectively deal with various jailbreak attacks.

[0005] In a first aspect, a method for enhancing a safety guardrail of a large text-generated model is provided in this embodiment, and the method comprises:

[0006] Based on a preset training data set, a predefined combined guardrail is trained; the combined guardrail is trained by: using each guardrail model in the combined guardrail, a sample text generation prompt in the training data set is identified to obtain an output result corresponding to each guardrail model; and according to each output result, a weight corresponding to each guardrail model is dynamically adjusted;

[0007] The input text generation prompt is processed by the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of the text generation prompt by each guardrail model and the corresponding weight of each guardrail model after training.

[0008] In some of the embodiments, dynamically adjusting the weight corresponding to each guardrail model according to each of the output results includes:

[0009] Determining a target guardrail model among the guardrail models;

[0010] Compare the output result corresponding to each guardrail model with the output result corresponding to the target guardrail model;

[0011] According to the comparison result, the weight corresponding to the guardrail model is dynamically adjusted.

[0012] In some of the embodiments, dynamically adjusting the weight corresponding to the guardrail model according to the comparison result includes:

[0013] Get predefined ensemble learning models;

[0014] Processing the comparison result based on the decision function defined by the ensemble learning model;

[0015] According to the processing result, the weight corresponding to the guardrail model is dynamically adjusted.

[0016] In some of the embodiments, the decision result is determined by the processing result of each guardrail model on the text generation prompt and the weight corresponding to each guardrail model, including:

[0017] Determine the evaluation value of the combined guardrail for the text generation prompt based on the processing result of each guardrail model on the text generation prompt and the weight corresponding to each guardrail model;

[0018] Determining whether the evaluation value is greater than or equal to a preset threshold;

[0019] According to the judgment result, the decision result of the combined guardrail for generating a prompt for the text is output.

[0020] In some of the embodiments, after training the combined guardrail based on the preset training data set, the method further includes:

[0021] Transform the original attack sample to obtain the target attack sample;

[0022] Through the target attack samples, a simulated attack is carried out on the trained combined guardrail to obtain a protection test result of the combined guardrail.

[0023] In some embodiments, converting the original attack sample to obtain the target attack sample includes:

[0024] Determining a target scenario corresponding to the original attack sample;

[0025] Based on the target scenario, the original attack sample is transformed through a predefined scenario construction function to obtain the corresponding target attack sample.

[0026] In some embodiments, converting the original attack sample to obtain the target attack sample includes:

[0027] Based on the gradient descent principle, the original attack sample is updated as a whole to obtain the corresponding target attack sample.

[0028] In a second aspect, in this embodiment, a device for enhancing a safety guardrail by generating a large model from text is provided, the device comprising:

[0029] A training module is used to train a predefined combined guardrail based on a preset training data set; the combined guardrail is trained by: using each guardrail model in the combined guardrail, identifying the sample text generation prompt in the training data set, and obtaining an output result corresponding to each guardrail model; and dynamically adjusting the weight corresponding to each guardrail model according to each output result;

[0030] A processing module is used to process the input text generation prompt through the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training.

[0031] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for enhancing the safety guardrail of a large model generated from text as described in the first aspect above is implemented.

[0032] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the method for enhancing the safety guardrail of a large model generated from text as described in the first aspect above is implemented.

[0033] Compared with the related art, the security guardrail enhancement method, device and computer equipment of the text generation large model provided in the present embodiment trains a predefined combined guardrail based on a preset training data set; the training of the combined guardrail is: through each guardrail model in the combined guardrail, the sample text generation prompt in the training data set is identified to obtain the output result corresponding to each guardrail model; according to each output result, the weight corresponding to each guardrail model is dynamically adjusted; the input text generation prompt is processed by the trained combined guardrail to obtain the corresponding decision result; the decision result is determined by the processing result of the text generation prompt by each guardrail model and the weight corresponding to each guardrail model after training, which solves the problem that the defense capability of the large model security guardrail is insufficient and cannot effectively respond to various jailbreak attacks, and realizes the improvement of the defense capability of the large model security guardrail, so that it can effectively respond to various jailbreak attacks.

[0034] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 This is a hardware structure block diagram of a terminal device of a method for enhancing a safety guardrail of a large model generated by text provided in an embodiment of the present application;

[0037] Figure 2 This is a flow chart of a method for enhancing a safety guardrail of a large text model provided in an embodiment of the present application;

[0038] Figure 3 is a flowchart of a method for enhancing a safety guardrail of a large model generated from text provided in another embodiment of the present application;

[0039] Figure 4 It is a flowchart of a method for enhancing a safety guardrail of a large model generated by text provided in a preferred embodiment of the present application;

[0040] Figure 5 It is a structural block diagram of a safety guardrail enhancement device for generating a large text model provided in an embodiment of the present application.

[0041] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, training module; 20, processing module. DETAILED DESCRIPTION

[0042] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 : is a hardware structure block diagram of the terminal of the method for enhancing the safety guardrail of the text-generated large model of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0045] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for enhancing the safety guardrail of the large model generated by the text in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0047] In this embodiment, a method for enhancing the safety guardrail of a large text-generated model is provided. Figure 2 Flow chart of the method for enhancing the safety guardrail of the large model generated by text in this embodiment. Figure 2 As shown, the process includes the following steps:

[0048] Step S210, based on a preset training data set, training the predefined combined guardrail; the combined guardrail is trained by: using each guardrail model in the combined guardrail, identifying the sample text generation prompt in the training data set, and obtaining the output result corresponding to each guardrail model; according to each output result, dynamically adjusting the weight corresponding to each guardrail model;

[0049] Step S220, the input text generation prompt is processed by the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training.

[0050] Specifically, a training data set is obtained, the training data set includes sample text generation prompts of different categories, and sample text generation prompts of different categories include attack prompts, and a predefined combined guardrail is trained based on the training data set. Among them, the predefined combined guardrail includes multiple guardrail models, and the guardrail is a security protection mechanism for detecting and intercepting harmful content, such as Llama Guard3, NeMo Guardrails, Guardrails AI, etc., which are not limited here. It should be noted that the decision result of the combined guardrail for text generation prompts is determined by the processing results of each guardrail model on the text generation prompts, and the weight corresponding to each guardrail model after training, that is, the weighted sum of the processing results of each guardrail model.

[0051] When training the combined guardrail, the sample text generation prompt in the training data set is input into the combined guardrail, and the sample text generation prompt is recognized by each guardrail model in the combined guardrail to obtain the output result corresponding to each guardrail model, which is specifically expressed as follows:

[0052]

[0053] In formula (1), G i (x) represents the output of the i-th guardrail model; x represents the input sample text generation prompt. When the output of the guardrail model is that x is harmful content, the output G i (x) = 1, when the guardrail model outputs that x is not harmful content, then output G i (x) = 0, thereby ensuring that each guardrail model can independently evaluate the input content and give a preliminary classification result.

[0054] Then, for the n guardrail models in the combined guardrail, set the weight ω corresponding to each guardrail model i , the weight satisfies the non-negative property (ω i ≥0) and normalization It is used to indicate the relative importance of the guardrail model in the final decision of the combined guardrail for the input, and dynamically adjusts the weight corresponding to the guardrail model based on the output results of each guardrail model. Among them, the recognition accuracy of each guardrail model can be evaluated according to the recognition results of the guardrail model for each sample text and the preset labels of the sample text generation prompt, and the weight corresponding to the guardrail model can be dynamically adjusted to improve the decision accuracy of the combined guardrail; or, according to the recognition ability of each guardrail model or the characteristics of the specific recognition scene, the target guardrail model is selected from the combined guardrail, the output results corresponding to each guardrail model are compared with the output results corresponding to the target guardrail model, the comparison results are processed based on the decision function defined by the integrated learning model, and the weight corresponding to the guardrail model is dynamically adjusted according to the processing results, so as to adaptively adjust the importance of each guardrail, so that the trained combined guardrail can effectively respond to diversified jailbreak attacks while achieving the best recognition effect in different scenarios. Exemplarily, based on the recognition capabilities of each guardrail model, Llama Guard3 with the strongest recognition capability is selected as the target guardrail model. Taking Llama Guard3 as the target guardrail model as an example, the weight adjustment process for generating prompts based on each sample text is described. The output results corresponding to each guardrail model are compared with the output results corresponding to Llama Guard3. Based on the decision function defined by the ensemble learning model, the comparison results are processed, and according to the processing results, the weights corresponding to the guardrail models are dynamically adjusted, as shown below:

[0055]

[0056] In formula (2), represents a decision function, which is used to determine the weight of the guardrail model after dynamic adjustment; represents the output result of the j-th guardrail model generating prompts for the i-th sample text; L i G represents the preset label corresponding to the prompt generated by the i-th sample text; j = Guard3 means that the output result corresponding to the j-th guardrail model is the same as Llama Guard3; G j ≠Guard3 means that the output result corresponding to the jth guardrail model is different from Llama Guard3. Next, according to the output result of each guardrail model for generating prompts for the sample text and the adjusted weight, the evaluation value of the combined guardrail for generating prompts for the sample text is determined, which is specifically expressed as follows:

[0057]

[0058] In formula (3), G represents the evaluation value of the combined guardrail for generating prompts for the i-th sample text; jIt represents the output result corresponding to the jth guardrail model. After that, it is judged whether the evaluation value is greater than or equal to the preset threshold value. According to the judgment result, the decision result of the combined guardrail for the text generation prompt is output, which is specifically expressed as follows:

[0059]

[0060] In formula (4), represents the decision result of the current combined guardrail for the sample text generation prompt; θ represents the preset threshold. When the evaluation value is greater than or equal to the preset threshold, the decision result is that the current sample text generates a prompt that is harmful content (unsafe); otherwise, the decision result is that the current sample text generates a prompt that is safe content (safe).

[0061] Furthermore, in actual applications, the input text generation prompt is processed by the trained combined guardrail to obtain the corresponding decision result. Among them, based on the processing results of each guardrail model on the text generation prompt and the weight of each guardrail model after training, the evaluation value of the combined guardrail for the text generation prompt is determined, that is, the weighted sum of the processing results of each guardrail model is calculated as the evaluation value, and then it is judged whether the evaluation value is greater than or equal to the preset threshold. According to the judgment result, the decision result of the combined guardrail for the text generation prompt is output. Among them, when it is detected that the evaluation value is greater than or equal to the preset threshold, the output decision result is that the current input text generation prompt is harmful content, otherwise, the output decision result is that the current input text generation prompt is safe content, so as to achieve accurate recognition of the text generation prompt and avoid the large text generation model from responding to harmful text generation prompts.

[0062] At present, guardrail technology plays a vital role in the security protection of large language models. For example, large text generation models are used in literary creation, copywriting, and strategy formulation. Guardrail defense mechanisms are usually used to identify and respond to various attacks to prevent harmful content output and ensure information security. However, when facing various prompt word jailbreak attacks, the guardrail defense capabilities of large language models are insufficient, that is, the existing guardrail defense mechanism only performs well in specific areas. When the attack form changes or cross-domain attacks occur, its defense capability decreases significantly and cannot effectively respond to diverse jailbreak attacks.

[0063] Compared with the prior art, the present application trains the predefined combined guardrail based on a preset training data set; the combined guardrail is trained as follows: through each guardrail model in the combined guardrail, the sample text generation prompt in the training data set is identified to obtain the output result corresponding to each guardrail model; according to each output result, the weight corresponding to each guardrail model is dynamically adjusted; the input text generation prompt is processed by the trained combined guardrail to obtain the corresponding decision result; the decision result is determined by the processing result of the text generation prompt by each guardrail model and the weight corresponding to each guardrail model after training. Based on this, by constructing a combined guardrail containing multiple guardrail models and adaptively adjusting the importance of each guardrail in the combined guardrail, the judgment results of different guardrails are organically combined, so that the combined guardrail can achieve the best recognition effect in different scenarios, solves the problem that the large model security guardrail has insufficient defense capability and cannot effectively respond to various jailbreak attacks, and realizes the improvement of the defense capability of the large model security guardrail, thereby improving the protection accuracy and protection efficiency, and can effectively respond to various jailbreak attacks, and avoid the large text generation model responding to output harmful content.

[0064] In some embodiments, dynamically adjusting the weight corresponding to each guardrail model according to each output result in step S210 includes the following steps:

[0065] Step S211, determining a target guardrail model among the guardrail models;

[0066] Step S212, comparing the output result corresponding to each guardrail model with the output result corresponding to the target guardrail model;

[0067] Step S213: dynamically adjust the weight corresponding to the guardrail model according to the comparison result.

[0068] Specifically, according to the recognition capability of each guardrail model or the characteristics of a specific recognition scenario, a target guardrail model is selected from the combined guardrails. For example, according to the recognition capability of each guardrail model, Llama Guard3 with the strongest recognition capability is selected as the target guardrail model. This embodiment does not specifically limit the method for selecting the target guardrail model.

[0069] Furthermore, the output result corresponding to each guardrail model is compared with the output result corresponding to the target guardrail model to determine whether the output result corresponding to the guardrail model is the same as the output result corresponding to the target guardrail model, and then the weight corresponding to the guardrail model is dynamically adjusted according to the determination result. The decision function defined by the integrated learning model can be used to output the weight adjustment value corresponding to different determination results.

[0070] Through this embodiment, the target guardrail model among each guardrail model is determined, and the output results corresponding to each guardrail model are compared with the output results corresponding to the target guardrail model. Then, according to the comparison results, the weights corresponding to the guardrail models are dynamically adjusted, thereby achieving weight adjustment of each guardrail model, which helps to improve the accuracy of guardrail judgment.

[0071] In some embodiments, dynamically adjusting the weight corresponding to the guardrail model according to the comparison result in step S213 includes the following steps:

[0072] Get predefined ensemble learning models;

[0073] Processing the comparison results based on the decision function defined by the ensemble learning model;

[0074] According to the processing results, the weight corresponding to the guardrail model is dynamically adjusted.

[0075] Specifically, the output results corresponding to each guardrail model are compared with the output results corresponding to the target guardrail model, and then the comparison results are processed based on the decision function defined by the integrated learning model, and the weights corresponding to the guardrail models are dynamically adjusted according to the processing results. The integrated learning model is used to determine whether the output results corresponding to the guardrail model are the same as the output results corresponding to the target guardrail model, and whether the output results corresponding to the guardrail model are the same as the preset labels of the sample text generation prompts, and the weights of the guardrail models are adjusted according to the judgment results.

[0076] It should be noted that the training process of the above ensemble learning model can use a cross entropy loss function to measure the difference between the model prediction and the actual label. This embodiment does not specifically limit the loss function used. Taking the cross entropy loss function as an example, it is specifically expressed as follows:

[0077]

[0078] In formula (5), L CE represents the cross entropy loss function of the model; p(x i ) represents the actual label; q(x i ) represents the probability predicted by the model. Based on this, the loss function helps the model minimize the prediction error during the training process, and the loss function helps the model minimize the prediction error during the training process and improves the accuracy of the model.

[0079] Through this embodiment, a predefined integrated learning model is obtained, the comparison result is processed based on the decision function defined by the integrated learning model, and the weight corresponding to the guardrail model is dynamically adjusted according to the processing result. In this way, through the coordinated optimization of the integrated learning model and multiple guardrails, the guardrail has stronger robustness and accuracy, ensuring the maximization of the protection capability.

[0080] In some of the embodiments, Figure 3 As shown, the decision result in step S220 is determined by the processing result of each guardrail model on the text generation prompt and the weight corresponding to each guardrail model, including the following steps:

[0081] Step S221, determining the evaluation value of the combined guardrail for the text generation prompt based on the processing result of each guardrail model for the text generation prompt and the weight corresponding to each guardrail model;

[0082] Step S222, determining whether the evaluation value is greater than or equal to a preset threshold;

[0083] Step S223: outputting the decision result of the combined guardrail for the text generation prompt according to the judgment result.

[0084] Specifically, the trained combined guardrail processes the actual input text generation prompt to obtain the corresponding decision result. Based on the processing result of each guardrail model on the text generation prompt and the weight of each guardrail model after training, the evaluation value of the combined guardrail for the text generation prompt is determined, that is, the weighted sum of the processing results of each guardrail model is calculated as the evaluation value.

[0085] Afterwards, it is determined whether the evaluation value is greater than or equal to the preset threshold, and based on the determination result, the decision result of the combined guardrail for the text generation prompt is output. If the evaluation value is greater than or equal to the preset threshold, the current text generation prompt is output as harmful content; otherwise, the current text generation prompt is output as safe content.

[0086] Through this embodiment, based on the processing results of each guardrail model for text generation prompts and the weight corresponding to each guardrail model, the evaluation value of the combined guardrail for text generation prompts is determined, and it is judged whether the evaluation value is greater than or equal to the preset threshold. According to the judgment result, the decision result of the combined guardrail for text generation prompts is output, so as to accurately identify the input text generation prompts, which helps to achieve precise security defense of the guardrail.

[0087] In some of the embodiments, after the combined guardrail is trained based on the preset training data set, the following steps are also included:

[0088] Transform the original attack sample to obtain the target attack sample;

[0089] Through the target attack samples, the trained combined guardrail is simulated and the protection test results of the combined guardrail are obtained.

[0090] Specifically, the original attack sample is transformed to obtain the target attack sample. The original attack sample can be transformed based on the selected target scenario through a predefined scenario construction function to obtain the corresponding target attack sample, or the original attack sample can be updated as a whole based on the gradient descent principle to obtain the corresponding target attack sample, which is not limited here.

[0091] Afterwards, the trained combined guardrail is simulated attacked by the converted target attack samples to obtain the protection test results of the combined guardrail. It should be noted that the protection test results of the combined guardrail can be determined based on the recognition accuracy of the combined guardrail for each target attack sample. For example, for 100 target attack samples, the combined guardrail identifies and intercepts 97 target attack samples, that is, the interception success rate of the combined guardrail after training is 97%, indicating that the current defense capability of the combined guardrail is relatively strong. This is used as the protection test result of the combined guardrail, and a single guardrail model with strong recognition capability is used to identify the above 100 target attack samples. The current defense level of the combined guardrail is further evaluated based on the recognition and interception results of the single guardrail model. At the same time, the current vulnerabilities of the combined guardrail can be analyzed and repaired based on the target attack samples that have not been successfully intercepted.

[0092] In addition, when the combined guardrail includes a black-box guardrail model, a deep learning network corresponding to the black-box guardrail model can be constructed, and the network structure of the deep learning network can be adjusted according to the specific requirements of the agent guardrail model task. Here, the output of the deep learning network is assumed to be D(x), and the output layer of the network is designed as a linear layer, aiming to predict multiple harmful labels corresponding to each training sample. The output layer is expressed as D(x) = Wx + b, where W represents the weight matrix and b represents the bias vector. An equal amount of normal samples and harmful samples are obtained, and they are respectively input into the black-box guardrail model for query operations to obtain a score list for each sample under different harmful categories, and a corresponding training data set is constructed. Set f(x) as the prediction function of the black-box guardrail model. For each sample x, a corresponding score list {f(x)1, f(x)2, ..., f(x) n}, n is the number of harmful categories. The deep learning network is trained based on the training data set, and the trained deep learning network is used as the proxy guardrail model. The proxy guardrail model is used to simulate the evaluation behavior of the black box guardrail model on the test sample. The proxy guardrail model can accurately simulate the behavior logic of the black box guardrail model, which helps to identify potential vulnerabilities of the guardrail and repair them in time, thereby improving the protection capability of the guardrail.

[0093] Through this embodiment, the original attack samples are transformed to obtain target attack samples, and the trained combined guardrail is subjected to simulated attacks through the target attack samples to obtain the protection test results of the combined guardrail, thereby accurately testing the defense capability of the combined guardrail. At the same time, it helps to promptly discover potential vulnerabilities of the combined guardrail and repair them, so as to improve the defense capability of the combined guardrail.

[0094] In some of the embodiments, converting the original attack sample to obtain the target attack sample includes the following steps:

[0095] Determine the target scenario corresponding to the original attack sample;

[0096] Based on the target scenario, the original attack sample is transformed through the predefined scenario construction function to obtain the corresponding target attack sample.

[0097] Specifically, a reasonable scenario is selected as the corresponding target scenario for the original attack sample from the predefined scenario library, and the original attack sample is embedded into the target scenario through the predefined scenario construction function. For example, the original attack sample is converted into role-playing or other metaphorical expressions, etc., which is specifically expressed as follows:

[0098] x * =S(x,L) (6)

[0099] In formula (6), x represents the original attack sample; x * represents the target attack sample; S represents the scenario construction function; L represents the predefined scenario library. Based on this, the original attack sample is converted into a reasonable scenario-based attack sample.

[0100] Through this embodiment, the target scenario corresponding to the original attack sample is determined, and based on the target scenario, the original attack sample is transformed through a predefined scenario construction function to obtain the corresponding target attack sample, thereby realizing the transformation of the original attack sample, which helps to test the real defense capability of the combined guardrail, thereby timely discovering the potential weaknesses of the current guardrail.

[0101] In some of the embodiments, converting the original attack sample to obtain the target attack sample includes the following steps:

[0102] Based on the principle of gradient descent, the original attack sample is updated as a whole to obtain the corresponding target attack sample.

[0103] Specifically, for the guardrail model, the original attack sample is updated as a whole based on the gradient descent principle to obtain the corresponding target attack sample, which is specifically expressed as follows:

[0104]

[0105] In formula (7), x new represents the updated target attack sample; x old represents the original attack sample; η represents the learning rate; f(x) represents the safety score function of the guardrail model.

[0106] Through this embodiment, the original attack sample is updated with the entire text based on the gradient descent principle to obtain the corresponding target attack sample, thereby realizing the transformation of the original attack sample, which helps to accurately test the defense capability of the combined guardrail.

[0107] The present embodiment is described and illustrated below through preferred embodiments.

[0108] Figure 4 Flow chart of the method for enhancing the safety guardrail of the large model generated by text in the preferred embodiment. Figure 4 As shown, the safety guardrail enhancement method for generating a large model of the text includes the following steps:

[0109] Step S410, training a predefined combined guardrail based on a preset training data set;

[0110] Step S420, the training of the above-mentioned combined guardrail is as follows: by using each guardrail model in the combined guardrail, the sample text generation prompt in the training data set is identified to obtain the output result corresponding to each guardrail model; the target guardrail model in each guardrail model is determined, and the output result corresponding to each guardrail model is compared with the output result corresponding to the target guardrail model; the comparison result is processed based on the decision function defined by the integrated learning model, and the weight corresponding to the guardrail model is dynamically adjusted according to the processing result;

[0111] Step S430, the input text generation prompt is processed by the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training.

[0112] Through this embodiment, the predefined combined guardrail is trained based on a preset training data set; the training of the combined guardrail is as follows: through each guardrail model in the combined guardrail, the sample text generation prompt in the training data set is identified to obtain the output result corresponding to each guardrail model; the target guardrail model in each guardrail model is determined, and the output result corresponding to each guardrail model is compared with the output result corresponding to the target guardrail model; the comparison result is processed based on the decision function defined by the integrated learning model, and the weight corresponding to the guardrail model is dynamically adjusted according to the processing result, so that the weight is dynamically adjusted through the integrated learning model, so that the combined guardrail has stronger adaptability and accuracy, ensuring the maximization of protection capability.

[0113] Furthermore, in actual applications, the trained combined guardrail is used to process the input text generation prompt to obtain the corresponding decision result. The decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training, thereby solving the problem that the large model security guardrail has insufficient defense capability and cannot effectively respond to various jailbreak attacks. The defense capability of the large model security guardrail is improved, which can effectively respond to various jailbreak attacks.

[0114] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0115] In this embodiment, a device for enhancing the safety guardrail of a large text-generated model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0116] Figure 5 : is a structural block diagram of the safety guardrail enhancement device of the text generation large model of this embodiment, such as Figure 5 As shown, the device comprises:

[0117] The training module 10 is used to train the predefined combined guardrail based on the preset training data set; the combined guardrail is trained by: using each guardrail model in the combined guardrail, identifying the sample text generation prompt in the training data set, and obtaining the output result corresponding to each guardrail model; according to each output result, dynamically adjusting the weight corresponding to each guardrail model;

[0118] The processing module 20 is used to process the input text generation prompt through the trained combined guardrail to obtain the corresponding decision result; the decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training.

[0119] Through the device provided in this embodiment, the predefined combined guardrail is trained based on a preset training data set; the training of the combined guardrail is: through each guardrail model in the combined guardrail, the sample text generation prompt in the training data set is identified to obtain the output result corresponding to each guardrail model; according to each output result, the weight corresponding to each guardrail model is dynamically adjusted; the input text generation prompt is processed by the trained combined guardrail to obtain the corresponding decision result; the decision result is determined by the processing result of the text generation prompt by each guardrail model and the weight corresponding to each guardrail model after training, which solves the problem that the defense capability of the large model security guardrail is insufficient and cannot effectively respond to various jailbreak attacks, and realizes the improvement of the defense capability of the large model security guardrail, so that it can effectively respond to various jailbreak attacks.

[0120] In some of the embodiments, the training module 10 is also used to determine a target guardrail model among the guardrail models; compare the output results corresponding to each guardrail model with the output results corresponding to the target guardrail model; and dynamically adjust the weights corresponding to the guardrail models based on the comparison results.

[0121] In some of the embodiments, the training module 10 is also used to obtain a predefined integrated learning model; process the comparison result based on the decision function defined by the integrated learning model; and dynamically adjust the weight corresponding to the guardrail model according to the processing result.

[0122] In some of the embodiments, the processing module 20 is also used to determine the evaluation value of the combined guardrail for generating prompts for the text based on the processing results of each guardrail model for generating prompts for the text and the weight corresponding to each guardrail model; determine whether the evaluation value is greater than or equal to a preset threshold; and output the decision result of the combined guardrail for generating prompts for the text based on the judgment result.

[0123] In some of these embodiments, Figure 5 On the basis of, the device also includes a testing module, which is used to transform the original attack samples to obtain target attack samples; through the target attack samples, a simulated attack is performed on the trained combined guardrail to obtain a protection test result of the combined guardrail.

[0124] In some of the embodiments, the test module is further used to determine a target scenario corresponding to the original attack sample; based on the target scenario, the original attack sample is transformed by a predefined scenario construction function to obtain a corresponding target attack sample.

[0125] In some of the embodiments, the above-mentioned test module is also used to perform overall text update on the original attack sample based on the gradient descent principle to obtain the corresponding target attack sample.

[0126] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0127] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0128] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0129] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0130] S1, based on a preset training data set, training a predefined combined guardrail; the combined guardrail training is as follows: using each guardrail model in the combined guardrail, identifying the sample text generation prompt in the training data set, and obtaining the output result corresponding to each guardrail model; according to each output result, dynamically adjusting the weight corresponding to each guardrail model;

[0131] S2, processing the input text generation prompts through the trained combined guardrail to obtain the corresponding decision results; the decision results are determined by the processing results of each guardrail model on the text generation prompts and the corresponding weights of each guardrail model after training.

[0132] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0133] In addition, in combination with the method for enhancing the safety fence of a large model generated from text provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the methods for enhancing the safety fence of a large model generated from text in the above embodiments is implemented.

[0134] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0135] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0136] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0137] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for enhancing safety guardrails by generating a large model from text, characterized in that: The method comprises: Based on a preset training data set, a predefined combined guardrail is trained; the combined guardrail is trained by: using each guardrail model in the combined guardrail, a sample text generation prompt in the training data set is identified to obtain an output result corresponding to each guardrail model; and according to each output result, a weight corresponding to each guardrail model is dynamically adjusted; The input text generation prompt is processed by the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of the text generation prompt by each guardrail model and the corresponding weight of each guardrail model after training.

2. The method for enhancing the safety guardrail of a large text-generated model according to claim 1, characterized in that: The dynamically adjusting the weight corresponding to each guardrail model according to each of the output results includes: Determining a target guardrail model among the guardrail models; Compare the output result corresponding to each guardrail model with the output result corresponding to the target guardrail model; According to the comparison result, the weight corresponding to the guardrail model is dynamically adjusted.

3. The method for enhancing the safety guardrail of a large text-generated model according to claim 2, characterized in that: The dynamically adjusting the weight corresponding to the guardrail model according to the comparison result includes: Get predefined ensemble learning models; Processing the comparison result based on the decision function defined by the ensemble learning model; According to the processing result, the weight corresponding to the guardrail model is dynamically adjusted.

4. The method for enhancing the safety guardrail of a large text-generated model according to claim 1, characterized in that: The decision result is determined by the processing result of each guardrail model on the text generation prompt and the weight corresponding to each guardrail model, including: Determine the evaluation value of the combined guardrail for the text generation prompt based on the processing result of each guardrail model on the text generation prompt and the weight corresponding to each guardrail model; Determining whether the evaluation value is greater than or equal to a preset threshold; According to the judgment result, the decision result of the combined guardrail for generating a prompt for the text is output.

5. The method for enhancing the safety guardrail of a large text-generated model according to claim 1, characterized in that: After the combined guardrail is trained based on the preset training data set, the method further includes: Transform the original attack sample to obtain the target attack sample; Through the target attack samples, a simulated attack is carried out on the trained combined guardrail to obtain a protection test result of the combined guardrail.

6. The method for enhancing the safety guardrail of a large text-generated model according to claim 5, characterized in that: The converting of the original attack sample to obtain the target attack sample includes: Determining a target scenario corresponding to the original attack sample; Based on the target scenario, the original attack sample is transformed through a predefined scenario construction function to obtain the corresponding target attack sample.

7. The method for enhancing the safety guardrail of a large text-generated model according to claim 5, characterized in that: The converting of the original attack sample to obtain the target attack sample includes: Based on the gradient descent principle, the original attack sample is updated as a whole to obtain the corresponding target attack sample.

8. A safety guardrail enhancement device for generating a large model from text, characterized in that: The device comprises: A training module is used to train a predefined combined guardrail based on a preset training data set; the combined guardrail is trained by: using each guardrail model in the combined guardrail, identifying the sample text generation prompt in the training data set, and obtaining an output result corresponding to each guardrail model; and dynamically adjusting the weight corresponding to each guardrail model according to each output result; A processing module is used to process the input text generation prompt through the trained combined guardrail to obtain a corresponding decision result; the decision result is determined by the processing result of each guardrail model on the text generation prompt and the corresponding weight of each guardrail model after training.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the method for enhancing the safety guardrail of a large model generated from text according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for enhancing the safety guardrail of a large model generated from text according to any one of claims 1 to 7 are implemented.

Images