Model iteration training method, sample expansion method, equipment, medium and product
By splitting and iterative training of the target dataset, using the reply evaluation model to identify and generate Q&A samples that comply with security specifications, the problem of content security and compliance risks in natural language processing in deep learning models is solved, and the accuracy and robustness of the model in complex scenarios is improved.
Patent Information
- Application Number
- CN202510531050.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-05
AI Technical Summary
Deep learning and large-scale generative models face content security and compliance risks in the fields of natural language processing and intelligent question-and-answer. High-quality security labeling data sets are scarce and expensive to obtain, resulting in insufficient accuracy and robustness of the model in complex scenarios.
By splitting the target data set into a training set and a verification set, the training language model is initially trained, the replies evaluation model is used to identify violation content and generate Q&A samples that comply with security specifications, and iterative training is used to enhance the model's understanding and response capabilities in complex security scenarios.
It significantly improves the accuracy and robustness of the model in diverse scenarios, enhances the coverage and compliance response capabilities of complex security scenarios, and achieves high-quality data expansion and diversity coverage.
Smart Images

Figure CN120430410A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of this specification relate to the field of model technology, and in particular, to a model iterative training method, a sample expansion method, an electronic device, a computer-readable medium, and a computer program product. Background Art
[0002] With the rapid adoption of deep learning and large-scale generative models in natural language processing and intelligent question answering, they have demonstrated strong potential in scenarios such as customer service, content creation, and knowledge retrieval. However, in practical applications, these models often face content security and compliance risks. High-quality, security-labeled datasets are scarce and expensive to obtain, and relevant data often has privacy protection restrictions, making it difficult to circulate on a large scale. This makes it difficult for models to fully learn various potential violations or sensitive scenarios, resulting in insufficient accuracy and robustness when handling complex scenarios. Summary of the Invention
[0003] In view of this, one or more embodiments of this specification provide a model iterative training method, a sample expansion method, an electronic device, a computer-readable medium, and a computer program product.
[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:
[0005] According to a first aspect of one or more embodiments of this specification, a model iterative training method is proposed, including:
[0006] Obtaining a target dataset and splitting the target dataset into a training set and a validation set; the target dataset includes multiple questions and answers that meet preset safety specifications;
[0007] Training a target language model based on the training set to obtain a trained target language model; using the questions in the training set as model input and the answers to the questions as supervision labels;
[0008] generating responses to be evaluated for questions in the validation set using the target language model;
[0009] Using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification, and if a conclusion is reached that it does not comply, further outputting analysis information describing the illegal content in the response to be evaluated;
[0010] Using the trained sample generation model, generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information;
[0011] The target dataset is updated based on at least one of the question-answer pair samples for the next iterative training of the target language model.
[0012] The following beneficial effects may be included: the model iterative training method provided in this embodiment first splits the existing target data set into a training set and a validation set, uses the training set to perform preliminary training on the language model to be trained to obtain the target language model, and then allows the target language model to output responses to be evaluated for questions in the existing limited validation set, and then the response evaluation model identifies and outputs analysis information of illegal content in the responses to be evaluated, and then uses the sample generation model to automatically generate new, security-compliant question-answer pair samples based on this analysis information, and finally merges these question-answer pair samples with the target training set for the next round of training. This cycle is repeated, effectively overcoming the problems of scarce training data and insufficient coverage. The target language model continuously "learns" from the errors it generates, gradually enhancing its understanding and response capabilities for complex security scenarios, and significantly improving the accuracy, robustness, and coverage of the final model in a variety of scenarios.
[0013] According to a second aspect of one or more embodiments of this specification, a sample expansion method is proposed, including:
[0014] Obtaining a target dataset and splitting the target dataset into a training set and a validation set; the target dataset includes multiple questions and answers that meet preset safety specifications;
[0015] Generate responses to be evaluated for questions in the validation set using the target language model trained using the training set;
[0016] Using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification, and if a conclusion is reached that it does not comply, further outputting analysis information describing the illegal content in the response to be evaluated;
[0017] The trained sample generation model is used to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information, so as to expand the target data set.
[0018] This can have the following beneficial effects: The sample expansion method described above can automatically identify security deficiencies in the target language model when handling specific questions, generate targeted analysis information based on the response evaluation model, and then use the sample generation model to reference this analysis information to synthesize new question-answer pairs that meet preset security specifications, effectively supplementing the types of questions that were under-covered or risky in the original dataset. This method not only improves the target dataset's coverage in terms of diversity and security, but also achieves high-quality data expansion, providing more representative and robust training samples for subsequent language model training, ultimately enabling the model to achieve higher accuracy and compliance response capabilities in complex security scenarios.
[0019] According to a third aspect of the embodiments of this specification, there is provided an electronic device, including:
[0020] processor;
[0021] a memory for storing processor-executable instructions;
[0022] When the processor executes the executable instructions, it is used to implement the method described in the first aspect or the second aspect.
[0023] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.
[0024] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect or the second aspect when executed by a processor.
[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flowchart of a model iterative training method provided by an exemplary embodiment.
[0027] Figure 2 is another flowchart of a model iterative training method provided by an exemplary embodiment.
[0028] Figure 3 It is a flowchart of evaluating a response to be evaluated provided by an exemplary embodiment.
[0029] Figure 4 It is a flowchart of generating at least one question-answer pair sample provided by an exemplary embodiment.
[0030] Figure 5This is a flowchart of a sample expansion method provided by an exemplary embodiment.
[0031] Figure 6 It is a structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION
[0032] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.
[0033] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0034] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.
[0035] Here are some explanations of the terms mentioned in this manual:
[0036] 1. Language models, such as the Large Language Model (LLM) and the BERT model, are AI models based on deep learning technology, particularly those trained using large corpora. They are designed to understand and generate text similar to human language and possess strong natural language understanding and generation capabilities. The goal of language models is to leverage natural language processing capabilities to enable a variety of applications, such as text generation, translation, summarization, question-answering, and dialogue systems, thereby improving the efficiency and automation of human-computer interaction.
[0037] 2. Prompts are tips, instructions, or instructions. In the field of artificial intelligence (AI), prompts are used to guide and direct language models to produce specific outputs. Prompts can be text snippets or questions that inspire language models to think and generate relevant content.
[0038] The embodiments of this specification provide a model iterative training method, a sample expansion method, an apparatus, an electronic device, a computer-readable medium, and a computer program product.
[0039] In some embodiments, see Figure 1 as well as Figure 2 The following is an exemplary description of the model iterative training method. The method can be performed by electronic devices, including but not limited to physical servers, server clusters, cloud servers, smart phones / mobile phones, tablet computers, personal digital assistants (PDAs), laptop computers, and desktop computers. The model iterative training method includes:
[0040] In S101, a target data set is obtained and split into a training set and a validation set; the target data set includes multiple questions and answers that meet preset security specifications.
[0041] For example, during the first training, the target dataset can be pre-set, for example, the developer selects existing data from various security scenarios to build a high-quality target dataset that covers various typical security scenarios and edge cases.
[0042] Exemplarily, after obtaining the target data set, the electronic device can randomly split the target data set into a training set and a verification set, or it can split the target data set into a training set and a verification set according to a preset ratio. For example, the preset ratio includes {training set: verification value = 7:3 or 6:4 or 5:5}, but is not limited to this. The ratio can be set according to the actual application scenario.
[0043] For example, the preset security specifications can be set according to specific security scenarios, and this embodiment does not impose any restrictions on this.
[0044] For example, in the phishing email protection scenario, the preset security specifications include but are not limited to: ① Any answers requiring users to provide account numbers, passwords, or verification codes are prohibited; ② Wording containing typical phishing keywords (such as "Verify account now" / "Update payment information") is prohibited.
[0045] For example, in content review scenarios, preset safety standards include but are not limited to: ① prohibiting any description that encourages or glorifies violent acts; ② prohibiting the use of insulting, discriminatory, or hateful words against specific groups; ③ requiring that when sensitive topics arise, a "neutral" or "discouragement" orientation must be added to the answer.
[0046] For example, in child-friendly scenarios, preset safety regulations include but are not limited to: ① Any adult content or innuendo is strictly prohibited; ② The language must be simple and clear, and the sentences must encourage positivity; ③ It is prohibited to mention dangerous games, extreme sports, and other dangerous behaviors that may cause children to imitate.
[0047] In S102, the language model to be trained is trained based on the training set to obtain a trained target language model; the questions in the training set are used as model inputs and the answers to the questions are used as supervision labels.
[0048] Exemplarily, the language model to be trained can be any pre-trained language model, and this embodiment imposes no limitation on this. The electronic device uses questions from the training set as input and supervises the responses to the questions to perform supervised learning on the language model to be trained, thereby enabling the model to master preliminary question-answer generation capabilities and safety specifications.
[0049] In one possible implementation, the electronic device inputs each question in the target training set into the language model to be trained. The language model generates a corresponding predicted response and compares it with the corresponding response in the training set that is known to comply with pre-set safety standards. The accuracy and standard deviation of the model output are evaluated by calculating the error between the two (for example, using metrics such as cross-entropy loss, KL divergence, and mean squared error). Subsequently, with minimizing the error between the predicted response and the response that complies with pre-set safety standards as the training objective, the model's trainable parameters are automatically adjusted using a backpropagation algorithm to improve the model's performance in subsequent response generation. Each parameter adjustment is based on the principle of "how far the prediction is from the safety compliance target," helping the model gradually learn how to respond accurately and compliantly in specific security scenarios. This training process is typically iterative. With increasing training rounds, the error between the responses generated by the model and the expected safety-compliant responses will continue to decrease until the model converges, resulting in a target language model that can consistently generate high-quality, safety-compliant responses.
[0050] In another possible implementation, in order to improve the training efficiency of the language model to be trained, the electronic device can adopt parameter efficient fine-tuning technology such as Low-Rank Adaptation (LoRA). Specifically, the electronic device selects at least part of the model parameters in the language model to be trained as the basis matrix, such as selecting the weight matrix in the substructure such as the attention layer and the feedforward network, and determines the trainable low-rank matrix corresponding to each basis matrix, and then supervises and fine-tunes the language model to be trained based on the training set; wherein, in the supervised fine-tuning process, each basis matrix remains unchanged, and the parameters of the low-rank matrix corresponding to each basis matrix are updated, and the combination of the updated low-rank matrix corresponding to each basis matrix and the basis matrix is determined as the updated model parameter. Since this training method does not need to update the original large-scale parameters, it only needs to train a small number of low-rank parameters, which significantly reduces the amount of computation and storage costs.
[0051] In S103 , the target language model is used to generate responses to be evaluated for questions in the validation set.
[0052] For example, after obtaining a trained target language model, the electronic device can input questions in the verification set into the target language model, and the target language model generates responses to the questions to be evaluated, so as to examine the output performance of the target language model on the verification set.
[0053] In S104, the trained response evaluation model is used to evaluate whether the response to be evaluated complies with the preset security specifications, and when a conclusion of non-compliance is reached, analysis information describing the illegal content in the response to be evaluated is further output.
[0054] The response evaluation model may be any language model or other type of model with evaluation function, and this embodiment does not impose any restrictions on this.
[0055] This step feeds the response to be evaluated into the response evaluation model to determine whether it complies with pre-set security specifications. If the response evaluation model determines that the response to be evaluated complies with the pre-set security specifications, it outputs a compliance conclusion. If the response evaluation model determines that it does not comply, it further outputs analytical information describing the violations in the response to be evaluated. This allows for the refinement of black-box "violations" into actionable analytical information, facilitating subsequent targeted corrections. It also captures model risks in an automated and structured manner, effectively replacing manual review.
[0056] For example, multiple assessment prompt templates can be pre-configured for different safety scenarios. Each assessment prompt template corresponds to a safety scenario and includes preset safety standards related to that safety scenario. For example, a safety scenario template involving children's content includes safety standards such as "age appropriateness" and "educational orientation"; a template involving financial content emphasizes safety standards such as "false advertising" and "investment misleading."
[0057] See also Figure 3 , for each answer to be evaluated in the verification set, the electronic device first identifies the security scenario to which the question belongs, so as to select an evaluation prompt template that matches the security scenario to which the question belongs from multiple pre-configured evaluation prompt templates; for example, if the question is "How to choose online content for children?", the electronic device will select an evaluation prompt template involving a children's content safety scenario. The electronic device then embeds the answer to be evaluated in the selected evaluation prompt template to obtain an evaluation prompt word; the evaluation prompt word is then input into the answer evaluation model, so that the answer evaluation model evaluates whether each answer to be evaluated meets the preset safety specifications, outputs the evaluation result (such as the conclusion of compliance or non-compliance), and further outputs analysis information used to describe the illegal content in the answer to be evaluated when a non-compliance conclusion is reached. Guided by the evaluation prompt word, the answer evaluation model can output a clearer violation analysis structure (such as violation type, core keywords, and scope of impact), which is helpful for subsequent question and answer rewriting and manual review and interpretation.
[0058] Exemplarily, the analysis information includes at least one of the following: ① Keywords that do not comply with preset safety specifications in the response to be evaluated; such keywords refer to specific words or phrases in the response to be evaluated that directly or indirectly violate preset safety specifications. As the core trigger point of the violation content, they can help electronic devices quickly locate the problematic content and facilitate subsequent correction, replacement or blocking. ② Violation factors involved in the response to be evaluated; violation factors refer to the potential or explicit risk categories reflected in the response content, reflecting in which aspect the content violates the safety specifications or causes safety hazards. By identifying violation factors, the nature of the violation of the response can be classified and judged, providing a structured basis for subsequent security scenario assessment and model repair. ③ Violation summary content for the response to be evaluated; the violation summary content is a comprehensive description or inductive analysis of the part of the response that does not comply with the preset safety specifications, usually given in natural language. The summary content provides a highly explanatory and contextually complete violation description for manual review or model correction, which is helpful for further risk modeling or reinforcement learning.
[0059] In S105, the trained sample generation model is used to generate at least one question-answer pair sample that can avoid the illegal content described in the analysis information.
[0060] The sample generation model can be any language model or other type of model with data synthesis function, and this embodiment does not impose any restrictions on this. For example, the sample generation model and the response evaluation model can be the same language model, or they can be different language models.
[0061] In this step, the electronic device can input the analysis information into the sample generation model to instruct the sample generation model to generate at least one question-and-answer pair sample that eliminates the illegal content described in the analysis information. The question-and-answer sample includes a question and an answer. Without increasing the cost of manual annotation, the analysis information is used to automatically expand the sample set that meets security standards, continuously introducing new, diverse, and safe and compliant training samples. This also targets weak scenarios in the target language model and improves the relevance of the dataset.
[0062] Exemplarily, multiple sample generation prompt templates for different security scenarios can be pre-configured, each sample generation prompt template corresponds to a security scenario, and each sample generation prompt template includes but is not limited to at least one of the following: ① background description related to the corresponding security scenario; ② multiple reasoning steps arranged in a chain of thought for instructing the sample generation model to generate question-answer pairs for samples; ③ negative examples or identified violation fragments related to the corresponding security scenario, used to enhance the model's sensitivity to security regulations; wherein, negative examples include questions belonging to the security scenario, answers that do not comply with the preset security regulations of the security scenario, and analysis information for the answer; ④ location parameters for embedding analysis information to ensure that the model can obtain and avoid the violation content described in the analysis information.
[0063] In an optional implementation, the multiple reasoning steps include a first reasoning step, a second reasoning step, and a solution step. (1) The first reasoning step is used to instruct the sample generation model to refer to the analysis information of the input, describe a security scenario, and list at least one violation content that can appear in the security scenario; thereby allowing the model to understand the risk space in the context. (2) The second reasoning step is used to instruct the sample generation model to analyze the cause of at least one violation content and provide at least one preventive measure; exemplary causes include missing information, misleading wording, and beyond reasonable speculation; exemplary preventive measures include adding neutral statements, introducing factual support, clarifying scope limitations, emphasizing that it does not constitute a suggestion, etc., so that the subsequent generated content has a logical basis for avoidance. (3) The solution step is used to instruct the sample generation model to generate a question-and-answer pair sample that meets the preset security specifications of the security scenario based on the reasoning results of the first reasoning step and the reasoning results of the second reasoning step. This embodiment divides the reasoning task into multiple steps, which can effectively control the model generation process, guide the model to collaboratively reason in the two dimensions of semantic understanding and security avoidance, improve the practicality and compliance of the samples, and combine analytical information with the step-by-step reasoning chain to enhance the sample generation model's sensitivity to security risks and accurate understanding of compliance goals, avoiding semantic deviation or excessive avoidance.
[0064] For example, Table 1 shows a sample generation prompt template.
[0065] Table 1
[0066]
[0067]
[0068] See also Figure 4After obtaining the non-compliance conclusion and analysis information for a response to be evaluated, the electronic device can select a sample generation prompt template that matches the security scenario to which the response refers, from among multiple pre-configured sample generation prompt templates, based on the question indicated by the response to be evaluated. Multiple sample generation prompt templates cover different security scenarios, and dynamic template selection helps the model gain a more detailed understanding of "which risks should be avoided in which security scenarios," thereby improving security robustness. The electronic device then embeds the analysis information into the selected sample generation prompt template to obtain a sample generation prompt word. Of course, information such as the question indicated by the analysis information, the response to be evaluated, and the conclusion that it does not comply with the preset security specification can also be embedded into the selected sample generation prompt template to provide more contextual information to the sample generation model, but this embodiment does not impose any restrictions on this. The sample generation prompt word is then input into the sample generation model, allowing the sample generation model to generate at least one question-and-answer pair sample that can avoid the illegal content described in the analysis information. By using sample generation prompt templates that closely match specific security scenarios and guiding the sample generation model to avoid specific violations, this embodiment can significantly reduce the probability of illegal content in generated samples and improve content quality.
[0069] In S106 , the target dataset is updated based on the at least one question-answer pair sample for the next iterative training of the target language model.
[0070] This step uses the newly generated question-answer pair samples to update the target dataset, thereby expanding the number of samples. A new round of iteration can be initiated based on the updated target dataset to continuously enhance the model's security performance.
[0071] In one possible implementation, the electronic device may merge at least one question-answer pair sample and a target data set to obtain a target data set for the next iterative training, and the language model to be trained in the next iterative training is the target language model trained this time.
[0072] In another possible implementation, after the sample generation model generates multiple question-answer pair samples, the electronic device does not directly use all of them to expand the target data set, but further uses the reply evaluation model to evaluate whether the reply samples in each question-answer pair sample comply with the preset security specifications to obtain an evaluation result; specifically, the electronic device can embed the reply in each question-answer pair sample into an evaluation prompt template that matches the security scenario to which it belongs, form an evaluation prompt word, and then input it into the reply evaluation model, so that the reply evaluation model can output an evaluation result of whether it complies with the preset security specifications.
[0073] The electronic device can then filter the question-and-answer pairs based on the evaluation results, retaining only those pairs that meet the conclusion and using them to update the target dataset, while removing those that do not. This effectively improves the quality and security of the data used for subsequent model training, prevents potentially illegal content from being mistakenly included in the training corpus, and reduces the likelihood of introducing risks during model training.
[0074] Exemplarily, the electronic device can merge the target data set and the question-answer pair samples that meet the conclusions in the evaluation results to obtain a merged set; then, the electronic device can deduplicate at least two question-answer pairs in the merged set whose similarity is greater than a preset threshold based on the similarity between the question-answer pairs. Specifically, a vectorized coding model can be used to encode each question-answer pair into a semantic vector, and the similarity between the semantic vectors of any two question-answer pairs is calculated. If the similarity between two question-answer pairs is greater than a preset threshold (for example, 0.95 or 0.9), they are considered semantically redundant, and a representative sample is retained from them, and the remaining duplicate samples are removed, thereby completing the deduplication process. The data set after the above deduplication operation is regarded as a new target data set and will be used for the next round of iterative training of the target language model. In this embodiment, deduplication after merging can effectively avoid the accumulation of invalid information, save storage space and training resources, and ensure that each piece of data has incremental value in training.
[0075] The model iterative training method provided in this embodiment first splits the existing target data set into a training set and a validation set, uses the training set to perform preliminary training on the language model to be trained to obtain the target language model, then allows the target language model to output responses to be evaluated for questions in the existing limited validation set, and then the response evaluation model identifies and outputs analysis information of illegal content in the responses to be evaluated, and then uses the sample generation model to automatically generate new, security-compliant question-answer pair samples based on this analysis information, and finally merges these question-answer pair samples with the target training set for the next round of training. This cycle repeats, not only continuously mining and supplementing diverse security scenario data from the model's own output, but also timely adjusting the sample distribution according to actual risks, significantly improving the model's coverage, accuracy, and robustness in complex security environments.
[0076] The above process describes the steps required to perform an iterative training process and can be regarded as a complete iterative training task.
[0077] For example, iterative training tasks are executed periodically at preset intervals. As security requirements evolve, the model periodically introduces newly generated, high-quality question-and-answer samples, continuously optimizing its judgment boundaries and language generation capabilities, maintaining the model's "freshness" and effectiveness. During periodic training, if new samples include feedback from the latest security threats or attack patterns, the model can quickly complete "knowledge injection," allowing it to adapt more quickly to new security scenarios.
[0078] For example, the iterative training task may stop executing when a preset iteration end condition is met. The preset iteration end condition includes at least one of the following:
[0079] ① The number of iterations reaches a preset number, such as limiting it to 50 or 60 rounds of iterations to prevent model overfitting or excessive consumption of system resources.
[0080] ② The preset evaluation indicators of the target language model meet the requirements. For example, if the accuracy, security compliance rate or other evaluation indicators of the target language model on the validation set reach the expected threshold (such as ≥98%), the training target is considered to have been achieved.
[0081] ③ If the number of samples in the target dataset exceeds the preset number, such as the cumulative sample volume exceeds the set capacity (such as 1 million), the iteration process can be terminated to prevent data redundancy from causing slow training or excessive pressure on memory resources.
[0082] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.
[0083] In some embodiments, see Figure 5 , this specification also provides a sample expansion method, including:
[0084] In S501 , a target data set is obtained and split into a training set and a validation set; the target data set includes multiple questions and answers that meet preset security specifications.
[0085] In S502 , the target language model trained by the training set is used to generate responses to questions in the validation set.
[0086] In S503, the trained response evaluation model is used to evaluate whether the response to be evaluated complies with the preset security specifications, and when a conclusion of non-compliance is reached, analysis information describing the illegal content in the response to be evaluated is further output.
[0087] In S504, the trained sample generation model is used to generate at least one question-answer pair sample that can avoid the illegal content described in the analysis information, so as to expand the target data set.
[0088] In this embodiment, the sample expansion method described above can automatically discover the security deficiencies of the target language model when handling specific questions, generate targeted analysis information based on the response evaluation model, and then use the sample generation model to reference the analysis information to synthesize new question-answer pair samples that meet preset security specifications, effectively supplementing the types of questions that are insufficiently covered or risky in the original dataset. This method not only improves the coverage of the target dataset in terms of diversity and security, but also achieves high-quality data expansion, thereby providing more representative and robust training samples for subsequent language model training, ultimately enabling the model to have higher accuracy and compliance response capabilities in complex security scenarios.
[0089] In one implementation, a trained response evaluation model is used to evaluate whether the response to be evaluated complies with preset security specifications, and when a conclusion of non-compliance is reached, analysis information for describing the illegal content in the response to be evaluated is further output, including: for the response to be evaluated for each question in the verification set, an evaluation prompt template that matches the security scenario to which the question belongs is selected from multiple pre-configured evaluation prompt templates; each evaluation prompt template corresponds to a security scenario and contains preset security specifications related to the security scenario; the response to the question to be evaluated is embedded in the selected evaluation prompt template to obtain an evaluation prompt word; the evaluation prompt word is input into the response evaluation model, so that the response evaluation model evaluates whether each response to be evaluated complies with the preset security specifications, and when a conclusion of non-compliance is reached, analysis information for describing the illegal content in the response to be evaluated is further output.
[0090] In one implementation, the analysis information includes at least one of the following: keywords in the response to be evaluated that do not comply with preset security specifications, violation factors involved in the response to be evaluated, and a summary of violations for the response to be evaluated.
[0091] In one implementation, a trained sample generation model is used to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information, including: based on the question pointed to by the answer to be evaluated for which there is analysis information, a sample generation prompt template that matches the security scenario to which the question belongs is selected from multiple pre-configured sample generation prompt templates; each sample generation prompt template corresponds to a security scenario; the analysis information is embedded in the selected sample generation prompt template to obtain a sample generation prompt word; the sample generation prompt word is input into the sample generation model, so that the sample generation model generates at least one question-answer pair sample that can circumvent the illegal content described in the analysis information.
[0092] Exemplarily, each sample generation prompt template includes: multiple reasoning steps arranged in a thought chain manner for instructing the sample generation model to generate question-answer pair samples, and / or negative examples related to the security scenario corresponding to the sample generation prompt template; wherein, the negative examples include questions belonging to the security scenario, answers that do not comply with the preset security specifications of the security scenario, and analysis information for the answers.
[0093] Exemplarily, the multiple reasoning steps include a first reasoning step, a second reasoning step and a solution step; the first reasoning step is used to instruct the sample generation model to refer to the analysis information of the input, describe a security scenario and list at least one illegal content that can appear in the security scenario; the second reasoning step is used to instruct the sample generation model to analyze the cause of at least one illegal content and provide at least one preventive measure; the solution step is used to instruct the sample generation model to generate a question-answer pair sample that meets the preset security specifications of the security scenario based on the reasoning results of the first reasoning step and the reasoning results of the second reasoning step.
[0094] In one implementation, the method further includes: using a response evaluation model to evaluate whether the response samples in each question-answer pair sample comply with preset security specifications to obtain an evaluation result; and updating the target data set based on the question-answer pair samples that contain the response samples that meet the conclusion in the evaluation results.
[0095] In one implementation, based on the fact that the evaluation results contain question-answer pair samples that meet the conclusion, the target dataset is updated, including: merging the target dataset and the question-answer pair samples whose evaluation results contain meet the conclusion to obtain a merged set; deduplicating at least two question-answer pairs in the merged set whose similarity is greater than a preset threshold to obtain a target dataset for the next iterative training of the target language model.
[0096] The various technical features in the above embodiments are specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.
[0097] In some embodiments, an embodiment of this specification further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the above methods by running the executable instructions.
[0098] Figure 6 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 6At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0099] In some embodiments, the model iteration training device can be applied to Figure 6 The device shown in the figure is used to implement the technical solution of this specification. The model iterative training device may include:
[0100] The dataset splitting module is used to obtain a target dataset and split it into a training set and a validation set; the target dataset includes multiple questions and answers that meet preset security specifications.
[0101] The training module is used to train the target language model based on the training set to obtain a trained target language model; the questions in the training set serve as the model input and the answers to the questions serve as supervision labels.
[0102] The response generation module is used to generate responses to be evaluated for questions in the validation set using the target language model.
[0103] The evaluation module is used to use the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specifications, and when a conclusion of non-compliance is reached, further output analysis information describing the illegal content in the response to be evaluated.
[0104] The sample generation module is used to use the trained sample generation model to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information.
[0105] The dataset updating module is used to update the target dataset based on at least one question-answer pair sample for the next iterative training of the target language model.
[0106] In some embodiments, the sample expansion device can be used in Figure 6 The device shown in the figure is used to implement the technical solution of this specification. The sample expansion device may include:
[0107] The dataset splitting module is used to obtain a target dataset and split it into a training set and a validation set; the target dataset includes multiple questions and answers that meet preset security specifications.
[0108] The response generation module is used to generate responses to be evaluated for questions in the validation set using the target language model trained with the training set.
[0109] The evaluation module is used to use the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specifications, and when a conclusion of non-compliance is reached, further output analysis information describing the illegal content in the response to be evaluated.
[0110] The sample generation module is used to use the trained sample generation model to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information, so as to expand the target data set.
[0111] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0112] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.
[0113] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.
[0114] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0115] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.
[0116] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.
Claims
1. A model iterative training method, comprising: Obtain a target dataset and split the target dataset into a training set and a validation set; The target data set includes a plurality of questions and answers that meet preset security specifications; Training a target language model based on the training set to obtain a trained target language model; using the questions in the training set as model input and the answers to the questions as supervision labels; generating responses to be evaluated for questions in the validation set using the target language model; Using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification, and if a conclusion is drawn that it does not comply, further outputting analysis information describing the illegal content in the response to be evaluated; Using the trained sample generation model, generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information; The target dataset is updated based on at least one of the question-answer pair samples for the next iterative training of the target language model.
2. The method according to claim 1, wherein training the language model to be trained based on the training set comprises: Selecting at least some model parameters in the language model to be trained as basis matrices, and determining low-rank matrices corresponding to each basis matrix; The language model to be trained is supervised fine-tuned based on the training set; wherein, during the supervised fine-tuning process, the parameters of the low-rank matrices corresponding to each basis matrix are updated, and the combination result of the updated low-rank matrices corresponding to each basis matrix and the basis matrix is determined as the updated model parameters.
3. The method according to claim 1, wherein the step of using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification, and when a conclusion of non-compliance is reached, further outputting analysis information describing the non-compliant content in the response to be evaluated, comprises: For each answer to be evaluated in the validation set, select an evaluation prompt template that matches the security scenario to which the question belongs from a plurality of pre-configured evaluation prompt templates; each evaluation prompt template corresponds to a security scenario and includes preset security specifications related to the security scenario; Embedding the answer to be evaluated in the question into the selected evaluation prompt template to obtain an evaluation prompt word; The evaluation prompt words are input into the response evaluation model so that the response evaluation model can evaluate whether each of the responses to be evaluated complies with the preset safety specifications, and when a conclusion of non-compliance is reached, further output analysis information for describing the illegal content in the responses to be evaluated.
4. According to the method according to claim 1, the analysis information includes at least one of the following: keywords in the response to be evaluated that do not comply with the preset security specifications, violation factors involved in the response to be evaluated, and a summary of the violations for the response to be evaluated.
5. The method according to claim 1, wherein the step of using a trained sample generation model to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information comprises: Based on the question pointed to by the to-be-evaluated answer in the analysis information, a sample generation prompt template matching the security scenario to which the question belongs is selected from a plurality of pre-configured sample generation prompt templates; Each of the sample generation prompt templates corresponds to a safety scenario; Embedding the analysis information into a selected sample generation prompt template to obtain a sample generation prompt word; The sample generation prompt word is input into the sample generation model so that the sample generation model generates at least one question-answer pair sample that can circumvent the illegal content described in the analysis information.
6. The method according to claim 5, wherein generating a prompt template for each sample comprises: Multiple reasoning steps arranged in a chain of thought for instructing the sample generation model to generate a question-answer pair sample, and / or negative examples related to a security scenario corresponding to the sample generation prompt template; The negative examples include questions belonging to the security scenario, answers that do not comply with the preset security specifications of the security scenario, and analysis information for the answers.
7. The method according to claim 6, wherein the plurality of reasoning steps comprises a first reasoning step, a second reasoning step, and a solution step; The first reasoning step is used to instruct the sample generation model to refer to the analysis information of the input, describe a security scenario and list at least one violation content that may appear in the security scenario; The second reasoning step is used to instruct the sample generation model to analyze the cause of the at least one illegal content and provide at least one preventive measure; The answering step is used to instruct the sample generation model to generate a question-answer pair sample that meets the preset security specifications of the security scenario based on the reasoning result of the first reasoning step and the reasoning result of the second reasoning step.
8. The method according to claim 1, wherein updating the target dataset based on at least one of the question-answer pair samples comprises: Using the answer evaluation model to evaluate whether the answer sample in each of the question-answer pair samples complies with the preset security specification, and obtaining an evaluation result; The target dataset is updated based on the evaluation results containing question-answer pair samples that meet the conclusion.
9. The method according to claim 8, wherein the step of updating the target dataset based on the evaluation results containing question-answer pair samples that meet the conclusion comprises: Merging the target data set and the question-answer pair samples that meet the conclusion in the evaluation results to obtain a merged set; At least two question-answer pairs in the merged set whose similarity is greater than a preset threshold are deduplicated to obtain a target data set for the next iterative training of the target language model.
10. A sample expansion method, comprising: Obtain a target dataset and split the target dataset into a training set and a validation set; The target data set includes a plurality of questions and answers that meet preset security specifications; Generate responses to be evaluated for questions in the validation set using the target language model trained using the training set; Using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification, and if a conclusion is drawn that it does not comply, further outputting analysis information describing the illegal content in the response to be evaluated; The trained sample generation model is used to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information, so as to expand the target data set.
11. The method according to claim 10, wherein the step of using the trained response evaluation model to evaluate whether the response to be evaluated complies with the preset security specification and, when a conclusion of non-compliance is reached, further outputting analysis information describing the non-compliant content in the response to be evaluated comprises: For each answer to be evaluated in the validation set, select an evaluation prompt template that matches the security scenario to which the question belongs from a plurality of pre-configured evaluation prompt templates; each evaluation prompt template corresponds to a security scenario and includes preset security specifications related to the security scenario; Embedding the answer to be evaluated in the question into the selected evaluation prompt template to obtain an evaluation prompt word; The evaluation prompt words are input into the response evaluation model so that the response evaluation model can evaluate whether each of the responses to be evaluated complies with the preset safety specifications, and when a conclusion of non-compliance is reached, further output analysis information for describing the illegal content in the responses to be evaluated.
12. The method according to claim 10, wherein the step of using a trained sample generation model to generate at least one question-answer pair sample that can circumvent the illegal content described in the analysis information comprises: Based on the question pointed to by the to-be-evaluated answer in the analysis information, a sample generation prompt template matching the security scenario to which the question belongs is selected from a plurality of pre-configured sample generation prompt templates; Each of the sample generation prompt templates corresponds to a safety scenario; Embedding the analysis information into a selected sample generation prompt template to obtain a sample generation prompt word; The sample generation prompt word is input into the sample generation model so that the sample generation model generates at least one question-answer pair sample that can circumvent the illegal content described in the analysis information.
13. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 12 by running the executable instructions.
14. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
15. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.
