Question and answer model training method and device, question and answer method and device and readable storage medium

Abstract

The invention relates to a question and answer model training method and device, a question and answer method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that a training set is constructed, and cue words of a first category with a known attack category label and cue words of a second category with an unknown attack category label are arranged in the training set; training an initial reward model according to the cue word to obtain a reward model which is used for outputting a reward signal; inputting the cue word into the large language model, outputting to obtain predicted reply data, and inputting the predicted reply data and the cue word into a reward model to obtain a reward signal for the predicted reply data; and determining an updating strategy of the large language model according to the reward signal of the predicted reply data, and updating the large language model based on the updating strategy to obtain a trained question and answer model. By adopting the method, the recognition precision of the question and answer model for known attack types can be improved, and the omission ratio for unknown attack types can be reduced.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a question-answering model training method and device, a question-answering method and device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. Large language models currently have a profound impact on economic development and social progress.

[0003] However, at the same time, it also creates security risks such as the complication of network security attacks and forged information. Prompt words are the bridge between users and large language models. Attackers may manipulate large language models to execute their intentions through carefully designed malicious prompt words, leading to sensitive information leakage, inappropriate content generation, system function abuse and other problems.

[0004] In related technologies, suspected malicious or non-compliant content is mainly discovered based on rules and pattern matching, and intercepted or released after manual review to identify malicious prompt words and filter malicious content. However, the content security issues brought about by the application of large language models are more complicated. Traditional methods cannot guarantee both the missed detection rate and the false detection rate at the same time, resulting in the inability to guarantee the security of the question and answer content. Summary of the invention

[0005] Based on this, it is necessary to provide a training method and device for a question-and-answer model, a question-and-answer method and device, a computer device, a computer-readable storage medium, and a computer program product that can improve the security of question-and-answer content in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for training a question-answering model, the method comprising:

[0007] Constructing a training set, wherein the training set includes prompt words of a first category and prompt words of a second category, wherein the prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels;

[0008] An initial reward model is trained according to the prompt word to obtain a trained reward model, wherein the reward model is used to output a reward signal, wherein the reward signal includes a positive reward signal or a negative reward signal, wherein the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word;

[0009] Inputting the prompt word into a large language model to obtain predicted response data as output, and inputting the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data;

[0010] An update strategy for the large language model is determined according to a reward signal for the predicted reply data, and the large language model is updated based on the update strategy to obtain a trained question-answering model.

[0011] In one embodiment, the step of training the initial reward model according to the prompt word to obtain the trained reward model includes:

[0012] constructing positive sample pairs and negative sample pairs based on the prompt words of the first category, wherein the positive sample pairs include prompt words corresponding to the same attack category, and the negative sample pairs include prompt words corresponding to different attack categories;

[0013] An initial reward model is trained based on the positive sample pairs, the negative sample pairs, and each of the prompt words to obtain a trained reward model.

[0014] In one embodiment, the training of the initial reward model based on the positive sample pairs, the negative sample pairs, and each of the prompt words to obtain the trained reward model includes:

[0015] Input the prompt word into the initial reward model to obtain the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category;

[0016] Determining a first loss value according to the embedding vector of the prompt word in each of the sample pairs;

[0017] Determine a second loss value based on the energy data of the prompt word for each known attack category;

[0018] Obtaining a joint loss value of the initial reward model according to the first loss value and the second loss value;

[0019] The initial reward model is trained according to the joint loss value to obtain a trained reward model.

[0020] In one embodiment, determining the first loss value according to the embedding vector of the prompt word in each of the sample pairs includes:

[0021] Substituting the embedding vector of the prompt word in each of the sample pairs into the contrast loss function to obtain a first loss value;

[0022] The contrast loss function is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs.

[0023] In one embodiment, determining the second loss value based on the energy data of each known attack category of the prompt word includes:

[0024] Substituting the energy data of the prompt word for each known attack category into the energy function to obtain the energy value of the prompt word;

[0025] Determining a second loss value according to the energy value of each prompt word and the energy value threshold;

[0026] The energy function is used to maximize the energy value of the prompt word of the unknown attack category.

[0027] In a second aspect, the present application provides a question-answering method, the method comprising:

[0028] In response to the questioning operation, obtaining a prompt word;

[0029] Inputting the prompt word into the question-answering model to obtain response data for the prompt word;

[0030] Wherein, the question-answering model is trained using any of the question-answering model training methods described above.

[0031] In a third aspect, the present application also provides a training device for a question-answering model, the device comprising:

[0032] A construction module, used to construct a training set, wherein the training set includes prompt words of a first category and prompt words of a second category, wherein the prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels;

[0033] A first training module is used to train an initial reward model according to the prompt word to obtain a trained reward model, wherein the reward model is used to output a reward signal, wherein the reward signal includes a positive reward signal or a negative reward signal, wherein the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word;

[0034] A processing module, used for inputting the prompt word into a large language model, outputting predicted response data, and inputting the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data;

[0035] The second training module is used to determine the update strategy of the large language model according to the reward signal of the predicted reply data, and update the large language model based on the update strategy to obtain a trained question-answering model.

[0036] In one embodiment, the first training module is specifically used for:

[0037] constructing positive sample pairs and negative sample pairs based on the prompt words of the first category, wherein the positive sample pairs include prompt words corresponding to the same attack category, and the negative sample pairs include prompt words corresponding to different attack categories;

[0038] An initial reward model is trained based on the positive sample pairs, the negative sample pairs, and each of the prompt words to obtain a trained reward model.

[0039] In one embodiment, the first training module is further used to:

[0040] Input the prompt word into the initial reward model to obtain the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category;

[0041] Determining a first loss value according to the embedding vector of the prompt word in each of the sample pairs;

[0042] Determine a second loss value based on the energy data of the prompt word for each known attack category;

[0043] Obtaining a joint loss value of the initial reward model according to the first loss value and the second loss value;

[0044] The initial reward model is trained according to the joint loss value to obtain a trained reward model.

[0045] In one embodiment, the first training module is further used to:

[0046] Substituting the embedding vector of the prompt word in each of the sample pairs into the contrast loss function to obtain a first loss value;

[0047] The contrast loss function is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs.

[0048] In one embodiment, the first training module is further used to:

[0049] Substituting the energy data of the prompt word for each known attack category into the energy function to obtain the energy value of the prompt word;

[0050] Determining a second loss value according to the energy value of each prompt word and the energy value threshold;

[0051] The energy function is used to maximize the energy value of the prompt word of the unknown attack category.

[0052] In a fourth aspect, the present application further provides a question-answering device, the device comprising:

[0053] An acquisition module, used for acquiring a prompt word in response to a question operation;

[0054] A question-answering module, used for inputting the prompt word into a question-answering model to obtain response data for the prompt word;

[0055] Wherein, the question-answering model is trained using any of the question-answering model training methods described above.

[0056] In a fifth aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the aforementioned question-answering model training methods or question-answering methods when executing the computer program.

[0057] In a sixth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned question-answering model training methods or question-answering methods.

[0058] In a seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the aforementioned question-answering model training methods or question-answering methods.

[0059] The training method and device of the question-answering model, the question-answering method and device, the computer device, the computer-readable storage medium and the computer program product. The training method of the question-answering model can first construct a training set, which includes prompt words of the first category and prompt words of the second category. The sample data of the first category has a known attack category label, and the prompt words of the second category have an unknown attack category label. The initial reward model is trained according to the prompt words to obtain a trained reward model. The reward model is used to output a reward signal. The reward signal includes a positive reward signal or a negative reward signal. The positive reward signal is used to describe the consistency of the predicted attack category and the attack category label corresponding to the prompt word. The negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word. The prompt word is input into the large language model, and the output is obtained. The predicted reply data, and the predicted reply data and the prompt word are input into the reward model to obtain a reward signal for the predicted reply data; the update strategy of the large language model is determined according to the reward signal of the predicted reply data, and the large language model is updated based on the update strategy to obtain a trained question-answering model. By using the question-answering model training method and apparatus, question-answering method and apparatus, computer equipment, computer-readable storage medium, and computer program product provided in the embodiments of the present application, in the process of constructing a data set, prompt words of known attack categories and prompt words of unknown attack categories are constructed, and after obtaining a reward model based on training of the training set, the reward signal output by the reward model is used to guide the training of the large language model, thereby obtaining a question-answering model that can accurately identify known attack categories and detect unknown attack categories, and then protect malicious prompt words, which can reduce the missed detection rate and false detection rate of the question-answering model, improve the compliance of the question-answering content output by the question-answering model, and thus improve network information security. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A flowchart of a method for training a question-answering model in one embodiment;

[0062] Figure 2 is a flow chart of step 104 in one embodiment;

[0063] Figure 3 is a flow chart of step 204 in one embodiment;

[0064] Figure 4is a flow chart of step 306 in one embodiment;

[0065] Figure 5 A schematic diagram of a flow chart of a question-answering method in one embodiment;

[0066] Figure 6 is a schematic diagram of a training method for a question-answering model in one embodiment;

[0067] Figure 7 is a structural block diagram of a training device for a question-answering model in one embodiment;

[0068] Figure 8 is a structural block diagram of a question-answering device in an embodiment;

[0069] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] In one embodiment, Figure 1 As shown, a training method for a question-answering model is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0072] Step 102 : construct a training set, which includes prompt words of a first category and prompt words of a second category. The prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels.

[0073] In the embodiment of the present application, a data set containing prompt words of known attack categories and unknown attack categories can be collected and created as a training set. Exemplarily, assuming that the known attack types include attack type A, attack type B, and attack type C, corresponding prompt words can be collected and created for each known attack category and annotated with the corresponding known attack type. In addition, prompt words for unknown attack categories are collected and created, and prompt words for unknown attack types can be annotated as unknown attack categories, or they can be left unannotated, which is not specifically limited in the embodiment of the present application.

[0074] Exemplarily, offensive prompt words can be widely collected through channels such as the Internet and social media, and manual and semi-automatic annotations can be performed to ensure the accuracy and diversity of the data. The annotation method of prompt words is not specifically limited in the embodiment of the present application. After collecting some prompt words, in order to enhance the generalization ability of the model, prompt words corresponding to known attack categories and prompt words corresponding to unknown attack categories can also be further created through data mining, genetic algorithms and synthesis methods. For example: new prompt words with specific offensive labels and new prompt words without offensive labels can be generated using a small parameter large model to supplement real data and improve the generalization ability of the model; or, the prompt words initially collected are used as a population to generate new prompt words through crossover and mutation operations, for example: new prompt words are generated through crossover (combining the features of different prompt words) and mutation (randomly changing part of the content of the prompt word) operations. Use multiple trained large language models to respond to the newly generated prompt words, observe whether the output meets the purpose of the attack, manually evaluate the aggressiveness of the prompt words, and determine the degree of its impact. Prompt words with high fitness (i.e., those that can effectively trigger bad outputs) are retained to the next generation, while those that do not meet the standards (i.e., those that have little or no effect on the model) are eliminated. Through multiple generations of iterations, the prompt words are continuously optimized to make them more aggressive, so that the generated data covers a variety of attack methods, thereby comprehensively testing the robustness of the question-answering model, testing the performance of the question-answering model in different attack scenarios, and improving the question-answering model accordingly to enhance its ability to resist attacks.

[0075] Step 104, train an initial reward model based on the sample data to obtain a trained reward model, the reward model is used to output a reward signal, the reward signal includes a positive reward signal or a negative reward signal, the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word.

[0076] In the embodiment of the present application, the reward model is a machine learning model for outputting a reward signal according to input data to evaluate the quality of the data. Exemplarily, the parameters of the reward model can be initialized using a pre-trained text embedding model (such as BERT, RoBERTa, etc.) to obtain an initial reward model, and the initial reward model can be trained using a prompt word. Through a machine learning algorithm, the initial reward model learns how to output a corresponding reward signal according to the attack category label of the prompt word. After the training is completed, a reward model that can output a reward signal can be obtained.

[0077] The reward signal includes a positive reward signal and a negative reward signal. For example, when the predicted attack category of the reply data of the large language model is consistent with the attack category label corresponding to the prompt word, a positive reward signal is output; or, when the predicted attack category of the reply data of the large language model is inconsistent with the attack category label corresponding to the prompt word, a negative reward signal is output.

[0078] For example, suppose there is a prompt word "how to make a bomb" and its attack category is labeled "violence". If the response data of the large language model is "It is detected that your question involves violent factors. Please adjust the question and submit it again", its predicted attack category is consistent with the attack category label of the prompt word, then the reward model outputs a positive reward signal; or, if the response data of the large language model is "The steps of planting flowers and plants include..., let's change the topic", then its predicted attack category is inconsistent with the attack category label of the prompt word, and the reward model outputs a negative reward signal.

[0079] Step 106, input the prompt word into the large language model, output the predicted response data, and input the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data.

[0080] In the embodiment of the present application, after the reward model is trained, the reward model can be used to guide the training of the large language model. For example, the prompt word can be input into the large language model, and the large language model will output the predicted response data for the prompt word. The predicted response data and the prompt word are input into the reward model, and the reward model will output a reward signal for the predicted response data.

[0081] Exemplarily, still taking the above example, assuming that the prompt word is "how to make a bomb", and its attack category is labeled as "violence", if the predicted reply data output by the large language model is "your question is detected to involve violent factors, please adjust the question and resubmit", the predicted reply data and the prompt word are input into the reward model, and the predicted attack category of the predicted reply data identified by the reward model is "violence", and the attack category corresponding to the prompt word is labeled as "violence", then it is determined that the predicted attack category of the predicted reply data is consistent with the attack category label of the prompt word, and the reward model can output a positive reward signal; or, if the reply data of the large language model is "the steps of planting flowers and plants include..., let's change the topic", the predicted attack category of the predicted reply data identified by the reward model is "non-attack category", and the attack category corresponding to the prompt word is labeled as "violence", it is determined that the predicted attack category of the predicted reply data is inconsistent with the attack category label of the prompt word, then the reward model can output a negative reward signal.

[0082] Step 108, determining an update strategy for the large language model according to the reward signal of the predicted reply data, and updating the large language model based on the update strategy to obtain a trained question-answering model.

[0083] In an embodiment of the present application, after obtaining a reward signal for predicting reply data, an update strategy for the large language model can be determined based on the reward signal, and then the large language model can be updated based on the update strategy to obtain a trained question-answering model.

[0084] For example, the advantage function can be calculated based on the reward signal of the predicted reply data, and the advantage function can be used to measure the advantages and disadvantages of the current strategy (new strategy) relative to the baseline strategy (old strategy). The PPO (Proximal Policy Optimization) algorithm is combined with the advantage function to determine the update strategy of the large language model. The objective function of the PPO algorithm aims to maximize the expected return while limiting the amplitude of the strategy update to prevent strategy collapse. The loss function of the large language model is shown in formula (1):

[0085] Formula (I)

[0086] in, is the probability ratio of the new and old strategies, indicating that in state Next, the new strategy takes action The ratio of the probability of taking the same action with the old policy. is a clipping function, To cut the threshold, limit the amplitude of the policy update, prevent the policy from crashing, and ensure exist and between. is the advantage function, is the expected value.

[0087] In multiple rounds, the large language model interacts with the environment, generates replies and obtains reward signals. Based on the obtained reward signals, the advantage function is calculated, and the parameters of the policy model are updated using the PPO algorithm. The quality and security of the generated content of the model are evaluated on the training set and the validation set. Pay attention to indicators such as the change in the loss function, the diversity, correctness and security of the generated content to ensure that the performance of the model is gradually improved. After completing the model fine-tuning, the model inference is accelerated by the model distillation and quantization methods to reduce time and computing costs. For example: by transferring the knowledge of the large model to the small model, the small model can approximate the performance of the large model. The specific steps include: first training a high-performance large model (teacher model), using the output of the teacher model as a guide, training a small student model, so that the student model is close to the teacher model in performance, and converting the model's weights and activation values ​​from floating point numbers to low-precision integers (such as 8-bit integers), thereby reducing the model's storage and computing requirements. Common quantization methods include uniform quantization and non-uniform quantization. Quantization can significantly reduce the model's inference time and computing resource consumption, thereby obtaining a trained question-answering model.

[0088] The training method of the above-mentioned question-answering model can first construct a training set, which includes prompt words of the first category and prompt words of the second category. The sample data of the first category has a known attack category label, and the prompt words of the second category have an unknown attack category label. The initial reward model is trained according to the prompt words to obtain a trained reward model. The reward model is used to output a reward signal. The reward signal includes a positive reward signal or a negative reward signal. The positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word. The prompt word is input into the large language model, and the output is obtained by predicting the reply data, and the predicted reply data and the prompt word are input into the reward model to obtain a reward signal for the predicted reply data; the update strategy of the large language model is determined according to the reward signal of the predicted reply data, and the large language model is updated based on the update strategy to obtain the trained question-answering model. By adopting the training method of the question-answering model provided in the embodiment of the present application, prompt words of known attack categories and prompt words of unknown attack categories are constructed in the process of constructing a data set, and after obtaining a reward model based on training of the training set, the reward signal output by the reward model is used to guide the training of the large language model, thereby obtaining a question-answering model that can accurately identify known attack categories and detect unknown attack categories, and then protect malicious prompt words, which can reduce the missed detection rate and false detection rate of the question-answering model, improve the compliance of the question-answering content output by the question-answering model, and thus improve network information security.

[0089] In an exemplary embodiment, referring to Figure 2As shown, in step 104, the initial reward model is trained according to the prompt word to obtain the trained reward model, including the following steps 202 to 204, wherein:

[0090] Step 202, constructing positive sample pairs and negative sample pairs based on the prompt words of the first category, wherein the positive sample pairs include prompt words corresponding to the same attack category, and the negative sample pairs include prompt words corresponding to different attack categories;

[0091] Step 204 , training an initial reward model based on the positive sample pairs, the negative sample pairs, and each prompt word to obtain a trained reward model.

[0092] In the embodiment of the present application, positive and negative sample pairs can be constructed based on the prompt words of the first category, aiming to enhance the model's ability to distinguish samples of known categories and ensure that samples of unknown categories maintain a large distance in the embedding space. Exemplarily, two samples corresponding to the same known attack category can be randomly selected from the prompt words of the first category as positive sample pairs, hoping that their embedded representations are close in space, and two samples from different known attack categories can be selected as negative sample pairs, hoping that their embedded representations are far apart in space.

[0093] The initial reward model is trained based on positive sample pairs and negative sample pairs. The goal is to make the embedding vectors of prompt words corresponding to the same attack category as close as possible in space, and the embedding vectors of prompt words corresponding to different attack categories as far as possible in space, so as to improve the model's recognition accuracy for known attack categories.

[0094] At the same time, the initial reward model is trained based on the prompt words of known attack categories and the prompt words of unknown attack categories. The goal is to make the prompt words of known attack categories have a good distinction from the prompt words of unknown attack categories, so that the model can accurately detect the prompt words of unknown attack categories, thereby improving the model's detection accuracy for the prompt words of unknown attack categories.

[0095] In an exemplary embodiment, referring to Figure 3 As shown, in step 204, the initial reward model is trained based on the positive sample pairs, the negative sample pairs, and each prompt word to obtain the trained reward model, which may include the following steps 302 to 310, wherein:

[0096] Step 302, input the prompt word into the initial reward model to obtain the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category;

[0097] Step 304, determining a first loss value according to the embedding vector of the prompt word in each sample pair;

[0098] Step 306, determining a second loss value based on the energy data of the prompt word for each known attack category;

[0099] Step 308, obtaining a joint loss value of the initial reward model according to the first loss value and the second loss value;

[0100] Step 310: train the initial reward model according to the joint loss value to obtain a trained reward model.

[0101] In an embodiment of the present application, the prompt words in the training set can be input into the initial reward model, and the initial reward model can output the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category, and the energy data is used to represent the probability that the prompt word belongs to a known attack category.

[0102] For each sample pair in each positive sample pair and each negative sample pair, a first loss value may be determined according to a distance between embedding vectors of two prompt words in the sample pair. In an exemplary embodiment, determining the first loss value according to the embedding vectors of the prompt words in each sample pair may include:

[0103] Substitute the embedding vector of the prompt word in each sample pair into the contrast loss function to obtain the first loss value;

[0104] Among them, the contrastive loss function is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs.

[0105] In the embodiment of the present application, a contrast loss function may be predefined, which is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs, as shown in the following formula (II):

[0106] Formula (II)

[0107] in, represents the contrast loss function, represents the total number of samples of the prompt word, Indicates prompt words The embedding vector of Indicates prompt words The embedding vector of . Indicates prompt words The embedding vector of The distance of the embedding vector, Indicates prompt words Attack category labeling, Prompt word Attack category labeling, is the preset distance threshold.

[0108] Substituting the prompt word band in each sample pair into the above contrast loss function, the first loss value can be obtained.

[0109] For any prompt word, its corresponding energy value can be determined based on the energy data of the prompt word for each known attack category, wherein the energy value can be used to characterize the uncertainty of the prompt word. An energy value can be determined as an energy value threshold based on the distribution of energy values ​​of prompt words of known attack categories, and energy values ​​above the energy value threshold represent unknown attack categories.

[0110] In an exemplary embodiment, referring to Figure 4 As shown, in step 306, determining the second loss value based on the energy data of each known attack category of the prompt word may include the following steps 402 to 404, wherein:

[0111] Step 402, substituting the energy data of the prompt word for each known attack category into the energy function to obtain the energy value of the prompt word;

[0112] Step 404, determining a second loss value according to the energy value of each prompt word and the energy value threshold;

[0113] The energy function is used to maximize the energy value of the prompt word of the unknown attack category.

[0114] In the embodiment of the present application, an energy function may be predefined to calculate the energy value of the input prompt word. The higher the energy value, the higher the uncertainty of the corresponding attack category, which is helpful to identify offensive prompt words of unknown categories. The energy function may introduce an adjustment temperature coefficient to control the output confidence. The energy function is shown in the following formula (III):

[0115] Formula (III)

[0116] in, Indicates the energy value of the prompt word, Indicates the adjustment temperature coefficient, represents the total number of known attack categories, Indicates prompt words Energy data for the k-th known attack class.

[0117] After obtaining the energy value of each prompt word, the second loss value can be determined based on the energy value of each prompt word and the energy value threshold. The second loss value can be determined by referring to formula (IV):

[0118] Formula (IV)

[0119] in, represents the second loss value, Indicates prompt words The energy value, Indicates the energy value threshold.

[0120] After obtaining the first loss value and the second loss value, a joint loss value of the initial reward model can be obtained according to the first loss value and the second loss value, and the initial reward model can be trained based on the joint loss value to obtain a trained reward model.

[0121] Among them, the determination method of the joint loss value can refer to formula (V):

[0122] Formula (V)

[0123] in, represents the joint loss value, represents the first loss value, is the preset weight, Represents the second loss value.

[0124] In the embodiment of the present application, the contrast loss function and the energy function can be combined into a joint loss function to train the reward model. The training goal is to optimize the model parameters so that it can accurately distinguish offensive prompt words of known and unknown categories, and provide an effective reward signal for the training of the large language model.

[0125] In this way, the reward model can be trained using the joint loss function. Exemplarily, the process of training the reward model based on the joint loss value may include: initializing model parameters, including initializing the parameters of the reward model using a pre-trained text embedding model. Adopting the Adam optimizer. Iteratively minimizing the joint loss function on the training data. Forward propagation, including calculating the embedding representation, energy value and loss function value for each batch of samples; back propagation, including calculating the gradient of the loss function to the model parameters. Update the model parameters according to the optimization algorithm based on the loss function value. Evaluate the model performance on the validation set and adjust the hyperparameters. Ensure that the model has high classification accuracy for known categories and high detection rate for unknown categories.

[0126] In one embodiment, Figure 5 As shown, a question-answering method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 502 and 504, wherein:

[0127] Step 502, in response to the questioning operation, obtaining a prompt word;

[0128] Step 504, input the prompt word into the question-answering model to obtain response data for the prompt word;

[0129] Among them, the question-answering model is trained using the question-answering model training method provided in the aforementioned embodiment.

[0130] In an embodiment of the present application, after the question-answering model is trained using the training method of the question-answering model provided in the aforementioned embodiment, in the subsequent application process, the user can interact with the question-answering model by asking questions such as inputting prompt words. After the terminal obtains the prompt word input by the user, the prompt word can be input into the question-answering model to obtain reply data of the prompt word. Since the question-answering model strengthens the recognition ability of prompt words of known attack types and the detection ability of prompt words of unknown attack types during the training process, it can recognize prompt words of known attack types and detect prompt words of unknown attack types, and then output safe and compliant reply data.

[0131] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below through specific examples.

[0132] For example, refer to Figure 6 As shown, the training process of the question-answering model in the embodiment of the present application includes the following steps:

[0133] Step 1: Data preparation. Collect known malicious prompt words (including various injection attacks, jailbreak attacks, etc.) and annotate them. In order to enhance the generalization ability of the model, create an unknown category of offensive prompt word datasets through data mining, genetic algorithms and synthesis methods. Exemplarily, step 1 includes the following steps 1.1 to 1.3, where:

[0134] Step 1.1: Data collection and labeling. We collected offensive words through the Internet, social media and other channels, and performed manual and semi-automatic labeling, covering multiple categories such as gender and race, to ensure data accuracy and diversity.

[0135] Step 1.2: Data synthesis: Use a large model with small parameters to generate new samples with specific offensive labels to supplement real data and improve the generalization ability of the model.

[0136] Step 1.3, genetic algorithm. First, collect the initial attack prompt words as the population, and generate new prompt words through crossover and mutation operations. Fitness evaluation is based on the feedback of multiple large models, combined with human feedback to judge the impact of the generated prompt words on the model, so as to identify more aggressive prompts. Prompt words with high fitness are retained to the next generation, and those that do not meet the standards are eliminated to continuously evolve more challenging attack data. Through multiple generations of iterations, the generated data covers a variety of attack methods.

[0137] Step 2, construct positive and negative sample pairs. Contrastive learning training aims to enhance the model's ability to distinguish samples of known categories and ensure that samples of unknown categories maintain a large distance in the embedding space. In the embodiment of the present application, sample pairs are constructed based on known data, including: constructing positive sample pairs (positive examples), randomly selecting two samples from the same known category, hoping that their embedded representations are close in space; constructing negative sample pairs (counterexamples), selecting two samples from different known categories, hoping that their embedded representations are far apart in space.

[0138] Step 3: Calculate sample embeddings. Use a pre-trained text embedding model (such as BERT, RoBERTa, etc.) to convert the prompt words into embedding vectors to prepare for subsequent loss calculations.

[0139] Step 4. Define the contrast loss function: The contrast loss function is used to measure the distance between sample pairs, maximize the similarity of samples of the same type, minimize the similarity of samples of different types, and enhance the model's ability to distinguish. The contrast loss function refers to the above formula (II).

[0140] Step 5. Define the energy function. Define the energy function to calculate the energy value of the input prompt word. The higher the energy value, the higher the uncertainty, which helps to identify offensive prompt words of unknown categories. Introduce the adjustment temperature coefficient to control the output confidence; based on the energy value distribution of known category samples on the validation set, select a threshold that can effectively distinguish known and unknown categories; add energy regularization terms in training to ensure that the energy value of known category samples is lower than the threshold. The energy function refers to the above formula (III).

[0141] Step 6: Train the reward model. The contrast loss function and the energy function are combined into a joint loss function (refer to the above formulas (IV and V)) to train the reward model. The training goal is to optimize the model parameters so that it can accurately distinguish between offensive prompt words of known and unknown categories, and provide effective reward signals for the training of the large language model. Step 6 includes the following steps 6.1 to 6.4, where:

[0142] Step 6.1. Initialize model parameters. Use a pre-trained text embedding model (such as BERT, RoBERTa, etc.) to initialize the parameters of the reward model.

[0143] Step 6.2: Select the optimizer and use the Adam optimizer.

[0144] Step 6.3, iterative training, iteratively minimize the joint loss function on the training data:

[0145] Forward propagation: For each batch of samples, calculate the embedding representation, energy value and loss function value.

[0146] Backpropagation: Calculate the gradient of the loss function with respect to the model parameters.

[0147] Parameter update: Update model parameters according to the optimization algorithm.

[0148] Step 6.4: Verification and Adjustment: Evaluate the model performance on the validation set and adjust the hyperparameters to ensure that the model has high classification accuracy for known categories and high detection rate for unknown categories.

[0149] Step 7: Reinforcement learning to train the large language model. Use the trained reward model to provide a reward signal for the large language model, and train the large language model through reinforcement learning method (proximal policy optimization algorithm PPO) to enable it to generate high-quality responses that meet safety requirements. Step 7 includes the following steps 7.1 to 7.4, where:

[0150] Step 7.1. Select the algorithm and use the Proximal Policy Optimization (PPO) algorithm as the reinforcement learning algorithm.

[0151] Step 7.2, get environmental feedback, including: the interaction process, the large language model generates a response based on the input prompt word. Reward signal design, use the trained reward model to evaluate the generated response and give positive or negative reward signals. Positive rewards, for high-quality, correctly classified responses generated, give positive rewards. Negative rewards, for responses that are not successfully classified or that mistakenly put unknown categories into known categories, give negative rewards.

[0152] Step 7.3, update the policy model. Calculate the advantage function. Based on the reward signal, calculate the advantage function to measure the advantages and disadvantages of the current policy relative to the benchmark policy. Update the policy. Use the PPO algorithm to update the parameters of the policy model and optimize the objective function, as shown in formula (1).

[0153] Step 7.4, training iteration, using the proximal policy optimization algorithm (PPO) as the reinforcement learning algorithm.

[0154] Repeat the above process in multiple rounds to continuously improve the generation strategy of large language models. Monitor the training progress and evaluate the performance of the model on the training set and validation set to ensure that the quality and security of the generated content are gradually improved.

[0155] Step 8: Model acceleration: After fine-tuning the model, model distillation and quantization are used to accelerate model reasoning and reduce time and computing costs.

[0156] The training method of the question-answering model provided in the embodiment of the present application solves the problem that the related technology lacks the ability to effectively identify and defend against unknown category attacks when dealing with malicious prompt words and jailbreak attacks. By introducing the energy function and contrastive learning method, a reward model is constructed that can simultaneously identify known and unknown category offensive prompt words. Combined with reinforcement learning training, the large language model can ensure the high quality of the content while meeting security and compliance requirements when generating replies. It has the following beneficial effects:

[0157] 1. Determine unknown offensive prompt words by energy value. This includes calculating the energy value of the input prompt word and setting an energy threshold to distinguish known and unknown categories. A higher energy value indicates that the uncertainty of the input prompt word is higher, so that potential unknown offensive prompt words can be identified. Relative related technologies mainly rely on predefined rules or adversarial training to identify known offensive prompt words, which is difficult to deal with new or mutated attacks. In the embodiment of the present application, the energy function module is introduced to significantly enhance the recognition ability of unknown offensive prompt words. The setting of the energy threshold can ensure that the energy value of the known category sample is lower than the threshold, and the unrecognized offensive prompt words are marked as unknown categories by high energy values, which effectively reduces the risk of unknown attack content being mistakenly generated.

[0158] 2. Expand the data set based on data synthesis and genetic algorithms. Including in the data preparation module, based on the initial data set, using data synthesis, data enhancement and genetic algorithms to generate diversified offensive prompt word data, including attack samples of unknown categories. Compared with traditional manual data annotation, data synthesis and genetic algorithms greatly enrich the diversity and coverage of data, and improve the model's ability to identify new and complex attack content. This module reduces the reliance on manual annotation, improves data preparation efficiency, and increases the generalization of the model, so that it can maintain a stable detection effect when facing attack prompt words in various forms of expression.

[0159] 3. Optimize the embedding space distribution of known category prompt words. This includes optimizing the embedding of known category prompt words through contrastive learning methods, so that prompt words of the same category are clustered in the embedding space and different categories are separated, thereby improving the model's ability to distinguish known categories. Compared with traditional methods, it is difficult to effectively distinguish samples of different categories in the embedding space when processing offensive prompt words. In the embodiments of the present application, contrastive learning is used to enable the model to accurately distinguish offensive prompt words of different categories in the embedding space, further improving the recognition accuracy of known category samples and reducing the misjudgment rate. In addition, the optimized embedding space can help the model accelerate convergence speed and reduce computational overhead during training.

[0160] 4 Based on the PPO algorithm combined with a trained reward model. Including in the generative model reinforcement learning module, the generative model is trained using PPO (proximal policy optimization algorithm), but the traditional reward function is innovatively replaced by a reward model trained by contrastive learning and energy function optimization to provide more accurate positive and negative reward signals. The reward model ensures that the model refuses to respond and provides a reasonable explanation when it identifies offensive prompt words by evaluating the generated content. Unlike the related art that directly uses the rule-defined reward function, the embodiment of the present application uses a trained reward model to significantly improve the accuracy and effectiveness of the reward signal. Through this design, the model can intelligently refuse to generate and give a reasonable explanation when faced with offensive prompt words. This method not only improves the security of the generated content, but also enhances the robustness of the model to malicious inputs. At the same time, by optimizing the generation strategy through PPO, the generation process of the model is more stable, avoiding the risk of strategy collapse during training.

[0161] In summary, the training method of the question-answering model provided in the embodiment of the present application can improve security and generalization ability, that is, enhance the model's ability to recognize offensive prompt words of unknown categories and reduce potential security risks. It can reduce manual dependence, generate diversified attack samples through data synthesis and genetic algorithms, reduce dependence on manually labeled data, and improve data preparation efficiency. The model performance can be optimized, and the accuracy and reliability of the reward model can be improved by using contrastive learning and energy functions to provide effective reward signals for the training of large language models. The generation quality can be improved: through reinforcement learning methods, the generation strategy of the large language model can be optimized to ensure that the reply content meets user needs and complies with safety regulations. The deployment cost can be reduced, and the model distillation and quantization technology can be used to accelerate the model reasoning speed and reduce hardware resource consumption and time costs.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a training device for a question-answering model and a question-answering device for the question-answering method for implementing the training method of the question-answering model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the training device and question-answering device embodiments of one or more question-answering models provided below can be found in the above limitations on the training method and question-answering method of the question-answering model, and will not be repeated here.

[0164] In an exemplary embodiment, Figure 7 As shown, a training device for a question-answering model is provided, including: a construction module 702, a first training module 704, a processing module 706 and a second training module 708, wherein:

[0165] A construction module 702 is used to construct a training set, wherein the training set includes prompt words of a first category and prompt words of a second category, wherein the prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels;

[0166] A first training module 704 is used to train an initial reward model according to the prompt word to obtain a trained reward model, wherein the reward model is used to output a reward signal, wherein the reward signal includes a positive reward signal or a negative reward signal, wherein the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word;

[0167] A processing module 706 is used to input the prompt word into the large language model, output the predicted response data, and input the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data;

[0168] The second training module 708 is used to determine the update strategy of the large language model according to the reward signal of the predicted reply data, and update the large language model based on the update strategy to obtain a trained question-answering model.

[0169] In one embodiment, the first training module is specifically used for:

[0170] constructing positive sample pairs and negative sample pairs based on the prompt words of the first category, wherein the positive sample pairs include prompt words corresponding to the same attack category, and the negative sample pairs include prompt words corresponding to different attack categories;

[0171] An initial reward model is trained based on the positive sample pairs, the negative sample pairs, and each of the prompt words to obtain a trained reward model.

[0172] In one embodiment, the first training module is further used to:

[0173] Input the prompt word into the initial reward model to obtain the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category;

[0174] Determining a first loss value according to the embedding vector of the prompt word in each of the sample pairs;

[0175] Determine a second loss value based on the energy data of the prompt word for each known attack category;

[0176] Obtaining a joint loss value of the initial reward model according to the first loss value and the second loss value;

[0177] The initial reward model is trained according to the joint loss value to obtain a trained reward model.

[0178] In one embodiment, the first training module is further used to:

[0179] Substituting the embedding vector of the prompt word in each of the sample pairs into the contrast loss function to obtain a first loss value;

[0180] The contrast loss function is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs.

[0181] In one embodiment, the first training module is further used to:

[0182] Substituting the energy data of the prompt word for each known attack category into the energy function to obtain the energy value of the prompt word;

[0183] Determining a second loss value according to the energy value of each prompt word and the energy value threshold;

[0184] The energy function is used to maximize the energy value of the prompt word of the unknown attack category.

[0185] By using the training device of the question-answering model provided in the embodiment of the present application, prompt words of known attack categories and prompt words of unknown attack categories are constructed in the process of constructing a data set, and after obtaining a reward model based on training of the training set, the reward signal output by the reward model is used to guide the training of the large language model, thereby obtaining a question-answering model that can accurately identify known attack categories and detect unknown attack categories, and then protect malicious prompt words, which can reduce the missed detection rate and false detection rate of the question-answering model, improve the compliance of the question-answering content output by the question-answering model, and thus improve network information security.

[0186] In an exemplary embodiment, Figure 8 As shown, a question-answering device is provided, including: an acquisition module 802 and a question-answering module 804, wherein:

[0187] An acquisition module 802 is used to acquire a prompt word in response to a question operation;

[0188] The question-answering module 804 is used to input the prompt word into the question-answering model to obtain response data for the prompt word; wherein the question-answering model is trained using any of the training methods for the question-answering model described above.

[0189] By using the question-and-answer device provided in the embodiment of the present application, prompt words for known attack categories and prompt words for unknown attack categories are constructed in the process of constructing a data set. After obtaining a reward model based on training of the training set, the reward signal output by the reward model is used to guide the training of the large language model, thereby obtaining a question-and-answer model that can accurately identify known attack categories and detect unknown attack categories, and then protect malicious prompt words, which can reduce the missed detection rate and false detection rate of the question-and-answer model, improve the compliance of the question-and-answer content output by the question-and-answer model, and thus improve network information security.

[0190] The training device of the question-answering model and each module in the question-answering device can be implemented in whole or in part by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to the modules above.

[0191] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a training method or a question-answering method of a question-answering model is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0192] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0193] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0194] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0195] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0197] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0198] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0199] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a question-answering model, characterized in that: The method comprises: Constructing a training set, wherein the training set includes prompt words of a first category and prompt words of a second category, wherein the prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels; An initial reward model is trained according to the prompt word to obtain a trained reward model, wherein the reward model is used to output a reward signal, wherein the reward signal includes a positive reward signal or a negative reward signal, wherein the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word; Inputting the prompt word into a large language model to obtain predicted response data, and inputting the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data; An update strategy for the large language model is determined according to a reward signal for the predicted reply data, and the large language model is updated based on the update strategy to obtain a trained question-answering model.

2. The method according to claim 1, characterized in that The step of training the initial reward model according to the prompt word to obtain the trained reward model includes: constructing positive sample pairs and negative sample pairs based on the prompt words of the first category, wherein the positive sample pairs include prompt words corresponding to the same attack category, and the negative sample pairs include prompt words corresponding to different attack categories; An initial reward model is trained based on the positive sample pairs, the negative sample pairs, and each of the prompt words to obtain a trained reward model.

3. The method according to claim 2, characterized in that The initial reward model is trained based on the positive sample pair, the negative sample pair, and each of the prompt words to obtain a trained reward model, including: Input the prompt word into the initial reward model to obtain the embedding vector corresponding to the prompt word and the energy data of the prompt word for each known attack category; Determining a first loss value according to the embedding vector of the prompt word in each of the sample pairs; Determine a second loss value based on the energy data of the prompt word for each known attack category; Obtaining a joint loss value of the initial reward model according to the first loss value and the second loss value; The initial reward model is trained according to the joint loss value to obtain a trained reward model.

4. The method according to claim 3, characterized in that The determining of the first loss value according to the embedding vector of the prompt word in each of the sample pairs includes: Substituting the embedding vector of the prompt word in each of the sample pairs into the contrast loss function to obtain a first loss value; The contrast loss function is used to maximize the similarity of the embedding vectors of the prompt words in the positive sample pairs and minimize the similarity of the embedding vectors of the prompt words in the negative sample pairs.

5. The method according to claim 3, characterized in that: The determining of the second loss value based on the energy data of each known attack category of the prompt word comprises: Substituting the energy data of the prompt word for each known attack category into the energy function to obtain the energy value of the prompt word; Determining a second loss value according to the energy value of each prompt word and the energy value threshold; The energy function is used to maximize the energy value of the prompt word of the unknown attack category.

6. A question-answering method, characterized in that: The method comprises: In response to the questioning operation, obtaining a prompt word; Inputting the prompt word into the question-answering model to obtain response data for the prompt word; Wherein, the question-answering model is trained using the question-answering model training method described in any one of claims 1 to 5.

7. A training device for a question-answering model, characterized in that: The device comprises: A construction module, used to construct a training set, wherein the training set includes prompt words of a first category and prompt words of a second category, wherein the prompt words of the first category have known attack category labels, and the prompt words of the second category have unknown attack category labels; A first training module is used to train an initial reward model according to the prompt word to obtain a trained reward model, wherein the reward model is used to output a reward signal, wherein the reward signal includes a positive reward signal or a negative reward signal, wherein the positive reward signal is used to describe the consistency between the predicted attack category and the attack category label corresponding to the prompt word, and the negative reward signal is used to describe the difference between the predicted attack category and the attack category label of the prompt word; A processing module, used for inputting the prompt word into a large language model, outputting predicted response data, and inputting the predicted response data and the prompt word into the reward model to obtain a reward signal for the predicted response data; The second training module is used to determine the update strategy of the large language model according to the reward signal of the predicted reply data, and update the large language model based on the update strategy to obtain a trained question-answering model.

8. A question-answering device, characterized in that: The device comprises: An acquisition module, used for acquiring a prompt word in response to a question operation; A question-answering module, used for inputting the prompt word into a question-answering model to obtain response data for the prompt word; Wherein, the question-answering model is trained using the question-answering model training method described in any one of claims 1 to 5.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images