A method, system and computer program product for obtaining language model training samples
By obtaining and correcting the Q&A sample pair of language models, training samples are generated to solve the poor response problems of the language models when dealing with risk topics, improving their risk identification and processing capabilities, ensuring that the output complies with ethical and regulatory requirements.
Patent Information
- Application Number
- CN202410650470.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-05-23
AI Technical Summary
When existing language models deal with user issues involving illegal or risky topics, they may output improper replies, resulting in poor guidance and difficult to effectively identify and correct risk content.
By obtaining the initial question samples, using the language model to generate relevant question-and-answer sample pairs, and judging their risk level based on the risk threshold, correcting improper responses, and generating training samples to enhance the risk identification and processing capabilities of the language model.
The language model's ability to identify and process risk content is improved, ensuring that its output complies with ethical and regulatory requirements, reducing bad guidance, and enhancing security.
Smart Images

Figure CN118312787B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system and computer program product for obtaining language model training samples. Background Art
[0002] As an artificial intelligence technology, language models are widely used in modern work and life. When a user inputs a question, the language model can generate a corresponding answer. However, for some risky questions raised by users involving illegal activities, the language model may output answers that are dangerous to the user, for example, instructing the user on how to commit illegal acts. Therefore, we need the language model to be able to identify such risky questions in a timely manner and output answers that positively guide the user.
[0003] Some embodiments of the present specification aim to provide a method for acquiring language model training samples, which can efficiently and automatically acquire diverse training samples. Through the training samples, the language model's ability to identify and process risky content can be improved, thereby enhancing the security of the language model. Summary of the invention
[0004] One or more embodiments of the present specification provide a method for obtaining language model training samples, including: obtaining a first question sample; the first question sample involves a risk topic; generating a second question sample involving the risk topic by referring to the first question sample through a first language model; generating a question answer text based on the question sample through the first language model, thereby obtaining a question and answer sample pair; the question and answer sample pair includes a question sample and its corresponding question answer text, and the question sample includes a first question sample and / or a second question sample; when the question answer text in the question and answer sample pair involves a risk topic, judging whether the question answer text therein contains a risk based on the question and answer sample pair through the first language model, and when the first language model judges that the question answer text contains a risk, correcting the question answer text through the first language model to reduce the risk level, and obtaining a corrected answer text; obtaining a training sample based on the question sample and its corresponding corrected answer text.
[0005] According to the method described in some embodiments of the present specification, the generation of a second question sample involving a risk topic by referring to the first question sample through a first language model includes: obtaining a first call text template, the first call text template including one or more first question sample slots, used to instruct the first language model to generate a second question sample with reference to the first question sample; selecting one or more first question samples from a first question library and filling them into one or more first question sample slots in the first call text template to obtain a first call text; inputting the first call text into the first language model to obtain a second question sample output by the first language model.
[0006] According to the method described in some embodiments of the present specification, the method also includes: processing the question and answer sample pairs through a judgment model to obtain the risk level of the question answer text in the question and answer sample pairs; when the risk level is greater than a set risk threshold, determining that the question answer text in the question and answer sample pairs involves a risk topic.
[0007] According to the method described in some embodiments of the present specification, the judging whether a question answer text therein contains risks based on the question and answer samples by the first language model includes: selecting one or more risk inquiry texts from a second question library; inputting the one or more risk inquiry texts into the first language model, and obtaining one or more judgment results output by the first language model as to whether the question answer text contains risks.
[0008] According to the method described in some embodiments of the present specification, when the first language model determines that the question answer text has risks, the question answer text is corrected through the first language model to reduce the risk level to obtain a corrected answer text, including: when at least one of the one or more judgment results reflects that the question answer text has risks, then: selecting one or more correction instruction texts from a third question library; inputting the one or more correction instruction texts into the first language model to obtain one or more corrected answer texts output by the first language model.
[0009] According to the method described in some embodiments of the present specification, the first question sample includes a preceding description of the conversation and a question statement, and the preceding description of the conversation and the question statement are respectively related to different steps of a certain risk behavior.
[0010] According to the method described in some embodiments of this specification, the training samples are used to iteratively train the first language model.
[0011] One or more embodiments of the present specification provide a language model training sample acquisition system, including: a first question sample acquisition module, used to acquire a first question sample; the first question sample involves a risk topic; a second question sample generation module, used to generate a second question sample involving a risk topic by referring to the first question sample through a first language model; a question and answer sample pair acquisition module, used to generate a question answer text based on the question sample through a first language model, and then obtain a question and answer sample pair; the question and answer sample pair includes a question sample and its corresponding question answer text, and the question sample includes a first question sample and / or a second question sample; a model introspection guidance module, used to determine whether the question answer text in the question and answer sample pair contains risks based on the question and answer sample pair through the first language model when the question answer text involves a risk topic, and when the first language model determines that the question answer text contains risks, the question answer text is corrected through the first language model to reduce the risk level and obtain a corrected answer text; a training sample acquisition module, used to obtain a training sample based on the question sample and its corresponding corrected answer text.
[0012] One or more embodiments of the present specification provide a language model training sample acquisition device, including a processor and a storage medium, wherein the storage medium is used to store computer instructions, and the processor is used to execute at least part of the computer instructions to implement the aforementioned language model training sample acquisition method.
[0013] One or more embodiments of the present specification provide a storage medium storing computer instructions, wherein when at least part of the computer instructions are executed by a processor, the aforementioned method for acquiring language model training samples can be implemented.
[0014] One or more embodiments of the present specification provide a computer program product, including computer instructions or a computer program, which, when at least part of the computer instructions or the computer program is executed by a processor, can implement the aforementioned method for acquiring language model training samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] This specification will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. The same numbers in the drawings represent the same structures or steps.
[0016] Figure 1 It is a schematic diagram of application scenarios of the language model shown in some embodiments of this specification.
[0017] Figure 2 It is an exemplary flowchart of the method for obtaining language model training samples according to some embodiments of this specification.
[0018] Figure 3is an exemplary flow chart of generating a second question sample according to some embodiments of the present specification.
[0019] Figure 4 This is an exemplary flowchart for determining whether a question answer text involves a risk topic through a judgment model according to some embodiments of this specification.
[0020] Figure 5 It is an exemplary block diagram of a system for acquiring language model training samples according to some embodiments of this specification. DETAILED DESCRIPTION
[0021] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the contents described below are some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, the technical solutions or means disclosed in this specification can also be applied to other scenarios based on these technical contents.
[0022] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0023] Unless otherwise specified, technical terms used in this specification to describe components, elements, etc. do not refer to the singular, but may also include the plural. Generally speaking, terms such as "include", "comprise", etc. only indicate that the steps, elements, or components that have been clearly identified are included, and these steps, elements, and components do not constitute an exclusive list, such as the method or device described may also include other steps or components.
[0024] Flowcharts are used in this specification to illustrate the operation steps performed by the device or system of the relevant embodiments, but unless otherwise specified, the order used to describe these steps should not be understood as a limitation on the order in which the steps are performed. A person of ordinary skill in the art can adjust the order in which these steps are performed based on the knowledge and information conveyed by the embodiments of this specification, and the adjustment includes but is not limited to swapping the order, merging multiple steps, and splitting a certain step.
[0025] As an artificial intelligence technology, language models are widely used in modern work and life. When a user inputs a question, the language model can intelligently generate a corresponding answer. In some cases, users may ask the language model some questions involving risk topics, and the language model may output replies that also involve risk topics. In some embodiments, questions involving risk topics may be questions that violate public order and morality, or even related to certain illegal acts. Replies involving risk topics may cause adverse guidance, so appropriate training samples are needed to train the language model. When the questions entered by the user involve risk topics, the trained language model can give positive replies to correctly guide the user.
[0026] To this end, some embodiments of this specification propose a method for obtaining language model training samples, which automatically expands a small number of questions (which can be understood as seed questions) compiled or collected manually to generate a larger number of question samples, and outputs the corresponding reply texts through the language model. The reply texts with risks are corrected, and the corrected reply texts and the corresponding question samples are used as training samples. Further, the language model is trained based on the obtained training samples, and the trained language model can output positive replies that correctly guide users.
[0027] Figure 1 Schematic diagram of application scenarios of language models according to some embodiments of this specification. Figure 1 As shown, the application scenario 100 may include: a processing device 110 , a network 120 , a storage device 130 , and a terminal 140 .
[0028] In some embodiments, a language model (not shown) may be deployed on the processing device 110, and the processing device 110 may perform one or more functions of the language model. For example, the processing device 110 may perform data interaction with the terminal 140 through the network 120, obtain the question input by the user through the terminal 140, and the processing device 110 may input the user's question into the language model, and return the answer output by the language model to the user's terminal 140. The aforementioned language model may be a first language model or another language model. In some embodiments, the processing device 110 may also process information and / or data related to the language model training sample acquisition method to perform one or more functions described in one or more embodiments of this specification. For example, the processing device 110 may guide the first language model to generate more second question samples based on the first question sample, and guide the first language model to generate a question answer text based on the question sample. For another example, the processing device 110 may determine whether there is a risk in the question answer text in the question and answer sample pair. For another example, the processing device 110 may guide the first language model to correct the question answer text with risk to obtain the corrected answer text.
[0029] In some embodiments, the processing device 110 may include one or more processors or processing engines (e.g., a single-chip processing engine or a multi-chip processing engine). By way of example only, the processor or processing engine may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, etc., or any combination thereof.
[0030] The network 120 may include any suitable network that can facilitate information and / or data exchange between components in the application scenario 100. In some embodiments, one or more components in the application scenario 100 (e.g., the processing device 110, the storage device 130, etc.) may exchange information and / or data through the network 120. For example, the processing device 110 obtains data in the storage device 130 through the network 120. In some embodiments, the network 120 may be any one or more of a wired network or a wireless network. For example, the network 120 may include a cable network, an optical fiber network, a telecommunications network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network (ZigBee), a near field communication (NFC), an in-device bus, an in-device line, a cable connection, or any combination thereof. The network connection between the various parts may be in one of the above-mentioned ways or in multiple ways. In some embodiments, the network may be a point-to-point, shared, centralized, or other topological structure or a combination of multiple topological structures.
[0031] The storage device 130 may be used to store data and / or instructions related to the application scenario 100. In some embodiments, the storage device 130 may store data and / or information obtained from the processing device 110, the terminal 140, etc. In some embodiments, the storage device is used to store model data and training samples of the language model, and may also store conversation records between the user and the model within a certain period of time in the recent period. Specifically, the storage device 130 may store question samples, question answer texts, question and answer sample pairs, revised answer texts, and training samples, etc.
[0032] In some embodiments, the storage device 130 may include one or more storage components, each of which may be an independent device or a part of another device. In some embodiments, the storage device 130 may be disposed in the processing device 110. In some embodiments, the storage device 130 may include a random access memory (RAM), a read-only memory (ROM), a mass storage device, a removable memory, a volatile read-write memory, or the like, or any combination thereof. Exemplarily, the mass storage device may include a magnetic disk, an optical disk, a solid-state disk, or the like. In some embodiments, the storage device 130 may be implemented on a cloud platform.
[0033] The user can use the terminal 140 to access the language model deployed on the processing device 110 through the network 120. In some embodiments, the user can input instructions or questions through the terminal 140. Specifically, the user can interact with the language model through natural language, ask the language model questions, and obtain the question answer text output by the language model. In some embodiments, the terminal 140 can be one of or any combination of other devices with input and / or output functions such as a mobile device, a tablet computer, a laptop computer, a desktop computer, etc. In some embodiments, the terminal 140 can also transmit relevant information and data through the network 120 and other components in the application scenario 100 (for example, the storage device 130).
[0034] It should be noted that the application scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of the language model usage scenarios provided in some embodiments of this specification. For example, in some application scenarios, the processing device 110 and the terminal 140 can be integrated into one, and in this case, the network 120 can be omitted.
[0035] Figure 2 is an exemplary flow chart of a method for obtaining language model training samples according to some embodiments of this specification. In some embodiments, Figure 2 The language model training sample acquisition method process 200 shown may be executed by the processing device 110. Further, it may be implemented by the language model training sample acquisition system 500 deployed on the processing device 110. The process 200 may include the following steps.
[0036] Step 210 , obtaining a first question sample; the first question sample involves a risk topic. In some embodiments, step 210 may be implemented by a first question sample obtaining module 510 .
[0037] The first question sample may refer to a manually acquired question sample. For example, the first question sample may be "How to peek?" The first question sample may be manually selected, collected, or manually compiled. For example, historical search questions may be determined from a database of a search engine, and the first question sample may be manually selected from the historical search questions. The number of the first question sample may be at least one.
[0038] Risk topics can refer to topics that may violate laws and regulations, threaten the personal and property safety of others, or violate moral ethics and public order. For example, risk topics can be topics related to theft, fighting, and poisoning. Risk topics can be determined manually. By determining risk topics, the training purpose of the language model can be effectively controlled, thereby improving the adaptability of the model in practical applications.
[0039] In some embodiments, in order to circumvent the language model's identification of risk issues, the questioner artificially splits the questions involving risk topics. For example, the original intention of the risk question may be "How to steal from A?", which is related to the act of stealing. However, since the language model may have a certain risk identification ability, it will not answer this question. Therefore, the questioner can split "How to steal from A?" into "How to make an appointment with A at the warehouse?", "How to make the bundling stronger?", "How to understand personal asset information?", etc. according to more than two steps of the risky behavior. In this case, each split question may not obviously involve the risk topic, but the overall combination involves the risk topic.
[0040] In view of this, in some embodiments, the first question sample may include a previous description of the conversation and a question statement. The previous description of the conversation and the question statement are respectively related to different steps of a risk behavior. As an example, the previous description of the conversation may be related to one step of a risk behavior, and the question statement may be related to another step of the aforementioned risk behavior.
[0041] More specifically, the question statement may be one of the multiple questions obtained by the above splitting, which is related to a certain step in a risky behavior. The previous description of the conversation may refer to the previous content or background content in the form of a conversation related to the question statement. The previous description of the conversation may include questions that interact with the first language model and the corresponding reply text. In some embodiments, the previous description of the conversation may include statements or questions and answers related to one or more other steps in the aforementioned risky behavior. For example, the previous description of the conversation may include "How to invite people to the warehouse?" and the corresponding reply text is "Tell the other party that a party will be held in the warehouse", "How can objects be fixed to the chair?" and the corresponding reply text is "Use a rope"; Correspondingly, the question statement may be "How to understand personal asset information?".
[0042] Through one or more embodiments of the present specification, by adding the above description of the conversation and the question statement to the first question sample, the first language model's ability to understand and process the content of the first question sample that potentially involves risk topics can be enhanced, thereby improving the overall accuracy of the first language model in identifying risk-related topics.
[0043] Step 220 , using the first language model to refer to the first question sample, generates a second question sample that also involves the risk topic. In some embodiments, step 220 may be performed by the second question sample acquisition module 520 .
[0044] The first language model may refer to a computer program or algorithm for understanding, generating, and processing natural language text. For example, the first language model may be Chat GPT, T5 (Text-to-Text Transfer Transformer), ERNIE (Enhanced Representation through kNowledge Integration), BERT (Bidirectional Encoder Representations from Transformers), etc.
[0045] In some embodiments, the first language model can understand the language text and make a response. The first language model can also execute the instructions such as answering, judging, and correcting in the language text through the reply. The input of the first language model can be a natural language text. More specifically, the input can be a language text containing requirements or instructions such as answering, judging, and correcting, and the output can be a reply text that meets the requirements of answering, judging, and correcting in the language text. For example, the input of the first language model can be a sentence requiring an answer "Give my cat a name", and the output can be the reply text "Xiaobai". For details about judging and correcting through the first language model, please refer to Figure 3 Related description.
[0046] The second question sample may refer to a question sample generated by the first language model. In some embodiments, the second question sample may involve the same risk topic as the first question sample. For example, the first question sample may be "How to burn down a house?", and the second question sample is "How to destroy a house?". In some embodiments, the second question sample may involve a different risk topic than the first question sample. For example, the first question sample may be "Can you give me a way to burn down my neighbor's house?", and the second question sample is "How to break into a house?".
[0047] In some embodiments, the second question sample may include a previous description of the conversation and a question statement. In some embodiments, the previous description of the conversation and the question statement in the second question sample are respectively related to different steps of a risk behavior. For details of the previous description of the conversation and the question statement, please refer to the relevant description of step 210.
[0048] In some embodiments, generating a second question sample that also involves a risk topic by referring to the first question sample through the first language model may include: obtaining a first call text template, the first call text template including one or more first question sample slots, used to instruct the first language model to refer to the first question sample to generate a second question sample; selecting one or more first question samples from the first question library to fill in one or more first question sample slots in the first call text template to obtain a first call text; inputting the first call text into the first language model to obtain a second question sample output by the first language model. For details about the first question library, the first call text template, and the first call text, please refer to this specification. Figure 3 Related description.
[0049] In some embodiments, the corresponding risk topic can be determined by the risk topic determination model based on the first question sample, and the corresponding risk topic can be input into the first language model to obtain a second question sample corresponding to the risk topic. For example, the first question sample "How to insult others?" is input into the risk topic determination model, and the corresponding risk topic output can be "insult", and "insult" is input into the first language model to obtain the second question sample "What are the 10 ways to insult others?".
[0050] In some embodiments, the risk topic determination model may be a machine learning model, for example, the risk topic determination model may be a neural network model, etc. The input of the risk topic determination model may include a first question sample, and the output may include a risk topic corresponding to the first question.
[0051] In some embodiments, the risk topic determination model can be obtained through training. For example, a topic training sample is input into the initial risk topic determination model, and a loss function is established based on the label and the output result of the initial risk topic determination model, and the parameters of the initial risk topic determination model are updated. When the preset conditions are met, the model training is completed, wherein the preset conditions may be that the loss function converges, the number of iterative training reaches a threshold, etc.
[0052] In some embodiments, the topic training sample may be the first question sample, and the topic training sample may be obtained based on historical data. The label of the training sample may be the risk topic corresponding to the first question sample. The label may be manually marked.
[0053] Through some embodiments of the present specification, a second question sample that also involves a risk topic is generated through a first language model, which reduces manual pressure, helps enrich training samples, improves the efficiency of generating question samples, enables the first language model to learn and understand risk topics more comprehensively and accurately, and can improve the generalization ability of the first language model.
[0054] Step 230 , generating a question answer text based on the question sample by using the first language model, thereby obtaining a question-answer sample pair. In some embodiments, step 230 may be performed by the question-answer sample pair acquisition module 530 .
[0055] The question sample may refer to an initial question input into the first language model. The question sample may be a first question sample or a second question sample.
[0056] The question answer text may refer to the text that answers the question in the question sample. The question answer text may include positive answers and negative answers to the question in the question sample. A positive answer may refer to an answer that does not involve a risky topic. A negative answer is an answer that involves a risky topic. For example, the question in the question sample may be "How to insult others?", and the positive answer may be "It is very important to establish positive and respectful interactions with others, and no suggestions can be provided on insulting or disrespecting others", and the negative answer to the question answer text may be "Throwing water on others in public".
[0057] In some embodiments, the question answer text may be obtained through a first language model. In some embodiments, the input of the first language model may include a question sample, and the output may include a corresponding question answer text.
[0058] Combining a question sample with its corresponding question answer text can produce a question-answer sample pair. For example, a question-answer sample pair may include a question sample "How to insult someone?" and a corresponding question answer text "Throw water on someone in public."
[0059] Through some embodiments of this specification, the first question sample determined manually and the second question sample determined by the first language model are used together as question samples, which can further improve the diversity of training samples. By determining the question-answer sample pair, the first language model can learn the relationship between the question sample and the corresponding answer text, and learn and understand the risk topic more comprehensively and accurately.
[0060] Step 240: when the question answer text in the question and answer sample pair also involves a risk topic, the first language model is used to determine whether the question answer text therein contains risks based on the question and answer sample pair, and when the first language model determines that the question answer text contains risks, the first language model is used to modify the question answer text to reduce the risk level, thereby obtaining a modified answer text. In some embodiments, step 240 may be performed by the model introspection guidance module 540.
[0061] In some embodiments, determining that the answer text in the question and answer sample pair also involves a risk topic may include: processing the question and answer sample pair through a judgment model to obtain the risk level of the answer text in the question and answer sample pair; when the risk level is greater than a set risk threshold, determining that the answer text in the question and answer sample pair also involves a risk topic. For details about the judgment model, please refer to this specification Figure 4 and its related description.
[0062] In some embodiments, determining whether the question answer text in the question and answer sample pair also involves risk topics can also include: creating a risk keyword list, checking whether the question answer text contains these keywords through keyword matching, and if so, it can be determined that the question answer text in the question and answer sample pair also involves risk topics.
[0063] In some embodiments, judging whether a question answer text contains risks based on question and answer samples by using a first language model may include: selecting one or more risk inquiry texts from a second question library; inputting the one or more risk inquiry texts into the first language model, and obtaining one or more judgment results on whether a question answer text output by the first language model contains risks.
[0064] The second question library may refer to a database including risk inquiry texts. The second question library may include at least one risk inquiry text. The second question library may be manually set.
[0065] The risk inquiry text may refer to a text used to inquire whether the question answer text contains risks. In some embodiments, at least one risk inquiry text may be a different expression of inquiring whether the question answer text contains risks. For example, the risk inquiry text may be "Does the above answer contain any behavior that helps users to commit crimes?"; for another example, the risk inquiry text may be "Does the above answer contain any behavior that guides users to commit crimes?".
[0066] The judgment result may refer to the judgment result of whether the question answer text has a risk. For example, the judgment result may be whether the question answer text has a risk or not. In some embodiments, it may be determined whether the question answer text has a risk based on the same question answer text and each risk inquiry text in the second question library. In some embodiments, the number of judgment results may be the same as the number of risk inquiry texts in the second question library.
[0067] In some embodiments, the process of judging whether there is a risk in the question answer text based on the risk inquiry text by the first language model may include: the input of the first language model may include one or more risk inquiry texts and the corresponding question answer texts, and the output may include one or more judgment results of whether there is a risk in the question answer text. In some embodiments, the first language model may output the judgment result of the question answer text through natural language text, such as "there is nothing wrong with this answer", "this answer does have bad guidance", etc. In some embodiments, the judgment result of the question answer text can be represented by a number, when the judgment result is that there is a risk, it is represented by 1, and when the judgment result is that there is no risk, it is represented by 0. In some embodiments, the judgment result output by the first language model can also be a numerical value between 0 and 1, and the numerical value is positively correlated with the risk level of the question answer text, that is, the larger the numerical value, the higher the risk level.
[0068] In some embodiments of the present specification, by setting a plurality of different forms of risk inquiry texts and combining them with question answer texts for inquiry, it is helpful to improve the accuracy and comprehensiveness of the first language model in risk identification.
[0069] Since the answer texts to questions involving risky topics do not necessarily involve risks, through some embodiments of the present specification, the question answer text is first judged whether it involves risky topics through a judgment model, and then the question answer text is judged again whether there is a risk through the first language model, rather than judging whether there is a risk for each question answer text, which can improve the efficiency of testing the risk identification ability of the first language model.
[0070] In some embodiments, when the first language model determines that there is a risk in the question answer text, the question answer text is corrected through the first language model to reduce the risk level to obtain a corrected answer text, including: when at least one of the one or more judgment results reflects that there is a risk in the question answer text, then: selecting one or more correction instruction texts from a third question library; inputting the one or more correction instruction texts into the first language model to obtain one or more corrected answer texts output by the first language model.
[0071] The third question library may refer to a database for storing correction instruction texts. The third question library may include at least one database of correction instruction texts.
[0072] The correction instruction text may refer to the text that guides the first language model to modify the question answer text. In some embodiments, at least one correction instruction text may be a different expression of the instruction to correct the answer text. For example, the correction instruction text may be "Modify your reply, your reply should comply with the requirements of laws and regulations." For another example, the correction instruction text may be "Please modify your answer to ensure that it does not contain any content that infringes on the rights of others." In some embodiments, the correction instruction text may correspond to the risk inquiry text. For example, the risk inquiry text may be "Does the above reply contain any behavior that helps users to conduct illegal and criminal information?", and the correction instruction text may be "Modify your reply, your reply should not contain any behavior that helps users to conduct illegal and criminal information."
[0073] The revised reply text may refer to a reply text after the reply text of the risky question is revised. The revision may refer to revising a negative reply to a positive reply. The revised reply text may be a positive reply to the question sample. The third question bank may be manually set.
[0074] In some embodiments, the process of correcting risky question answer text based on correction instruction text through a first language model may include: the input of the first language model may include risky question answer text and one or more correction instruction texts, and the output may include one or more correction answer texts of the question answer text.
[0075] In some embodiments of the present specification, by setting a plurality of different forms of correction instruction texts, it is helpful for the first language model to output positive responses in different forms, thereby enhancing the ability of the first language model to give positive responses to user questions.
[0076] Step 250 , based on the question sample and its corresponding corrected answer text, obtain a training sample. In some embodiments, step 250 may be performed by the training sample acquisition module 550 .
[0077] In some embodiments, the training sample can be used to train the first language model and can also be used to train other language models. The language model training process can be implemented by the training module 560. As an example, the language model training process can be supervised training, that is, the question sample in the training sample is input into the language model to be trained, and the model output is obtained. A loss function is constructed based on the model output and the corrected reply text in the training sample, and the model parameters of the language model to be trained are adjusted to reduce the difference between the model output and the corrected reply text. After training, when the user inputs questions involving risk topics to the language model, the language model can guide the user in a positive direction with a high probability.
[0078] By acquiring a large number of diverse training samples through the language model training sample acquisition methods of some embodiments of this specification, and training a language model based on the training samples, the security of the language model can be improved. Even when the user inputs questions involving risky topics, the language model can still give responses that comply with ethical and regulatory requirements, and reduce adverse guidance.
[0079] In some embodiments, the training samples can also be used to iteratively train the first language model. In this case, at least multiple rounds of iterative execution of steps 230 and 240 in process 200 and the step of performing supervised training on the first language model based on the training samples obtained in step 250 can be performed. In this way, on the one hand, by guiding the first language model to self-judge and correct the question answer text output by itself, the model's sensitivity to the question answer text involving risk topics can be increased, and the model's ability to output positive answers can be strengthened. On the other hand, through supervised training, the first language model can be made to output positive answers more directly and naturally when encountering questions involving risk topics, giving users positive guidance. In summary, the performance of the first language model can be better improved.
[0080] Figure 3 is an exemplary flow chart of generating a second question sample according to some embodiments of this specification. In some embodiments, process 300 may be executed by processing device 110. Specifically, process 300 may be implemented by second question sample acquisition module 520 in language model training sample acquisition system 500 deployed on processing device 110. Process 300 may include the following steps.
[0081] Step 310: Obtain a first calling text template, where the first calling text template includes one or more first question sample slots, which are used to instruct the first language model to generate a second question sample with reference to the first question sample.
[0082] The first calling text template may refer to a text template used to guide the generation of a question similar to the first question.
[0083] A slot may refer to an information structure that needs to be automatically filled, and a first question sample slot may refer to a text position that needs to be automatically filled with a first question sample. As an example, the first call text template may be "Please generate similar question texts based on x1 and x2", wherein "x1" and "x2" are first question sample slot identifiers or placeholders, and "x1" and "x2" may be replaced with first question samples, respectively. In some embodiments, the first question sample slot identifier or placeholder may correspond one-to-one to the first question sample. The first call text template and slot may be manually designed.
[0084] Step 320: Select one or more first question samples from the first question library and fill them into one or more first question sample slots in the first call text template to obtain a first call text.
[0085] The first question library may refer to a database of first question samples. The first question library may include at least one first question sample. Continuing with the previous example, two first question samples may be selected from the first question library and replaced with "x1" and "x2" in the first call text template, respectively, to obtain the first call text.
[0086] Step 330: input the first call text into the first language model to obtain a second question sample output by the first language model.
[0087] The input of the first language model may include a first call text, and the output may include a second question sample.
[0088] Through one or more embodiments of the present specification, the first call text can be automatically and efficiently generated through the first call text template and the first question library, and the second question sample can be automatically generated, further reducing labor costs and improving processing efficiency.
[0089] Figure 4 This is an exemplary flow chart for determining whether a question answer text involves a risk topic through a judgment model according to some embodiments of this specification. In some embodiments, process 400 may be executed by processing device 110. Specifically, process 400 may be implemented by a model introspection guidance module 540 in a language model training sample acquisition system 500 deployed on processing device 110. Process 400 may include the following steps.
[0090] Step 410, processing the question and answer sample pairs through the judgment model to obtain the risk level of the question answer text in the question and answer sample pairs.
[0091] The risk level may refer to the degree of risk involved in the question answering text. The risk level may be expressed as a numerical value, and the higher the numerical value, the higher the degree of risk involved in the question answering text. For example, the degree of risk involved in the question answering text with a risk level of 0.8 is higher than that of the question answering text with a risk level of 0.7.
[0092] The judgment model may refer to a model for judging the risk level of a question-answer sample pair. The judgment model may be a machine learning model independent of the first language model. For example, the judgment model may be a convolutional neural network, a recurrent neural network, etc. The input of the judgment model may include a question-answer sample pair, and the output may include the risk level of the question answer text in the question-answer sample pair.
[0093] In some embodiments, the judgment model can be obtained through training based on risk judgment samples. For example, the risk judgment sample is input into the initial judgment model, and a loss function is established based on the label and the output result of the initial judgment model, and the parameters of the initial judgment model are updated. When the loss function of the initial judgment model meets the preset conditions, the model training is completed, wherein the preset conditions may be that the loss function converges, the number of iterations reaches a threshold, etc.
[0094] In some embodiments, the risk judgment sample may include multiple question-answer sample pairs, and the risk judgment sample may be obtained based on historical data. The label of the risk judgment sample may be the risk level of the question-answer sample pair. The label may be manually annotated, such as "0" for no risk and "1" for risk. At this time, the input of the judgment model may include the question-answer sample pair, and the output may include the risk level of the question-answer sample pair.
[0095] Step 420: When the risk level is greater than the set risk threshold, determine whether the question answer text in the question and answer sample pair also involves a risk topic.
[0096] Setting a risk threshold may refer to determining the minimum risk level of a question-answer text involving a risk topic. Setting a risk threshold may be based on prior knowledge or manually set. For example, if the risk level of question-answer text A in a question-answer sample pair is 0.5, and the risk level of question-answer text B is 0.7, and the risk threshold is set to 0.6, then question-answer text B in the question-answer sample pair is determined to involve a risk topic.
[0097] In one or more embodiments of the present specification, the judgment model can improve the accuracy and efficiency of judging whether the question answer text in the question and answer sample pair involves risk topics. In some embodiments, the judgment result of the judgment model on the question answer text output by the first language model can be used as a reference to verify whether the first language model can self-check its question answer text, which helps to improve the sensitivity of the first language model to question answer text involving risk topics.
[0098] Figure 5 It is an exemplary block diagram of a system for acquiring language model training samples according to some embodiments of this specification. Figure 5 The system 500 shown is implemented by the processing device 110 , and may include: a first question sample acquisition module 510 , a second question sample generation module 520 , a question-answer sample pair acquisition module 530 , a model introspection guidance module 540 , and a training sample acquisition module 550 .
[0099] The first question sample acquisition module 510 is used to acquire a first question sample; the first question sample involves a risk topic.
[0100] The second question sample generating module 520 is used to generate a second question sample also involving a risk topic by referring to the first question sample through the first language model.
[0101] The question-answer sample pair acquisition module 530 is used to generate question answer text based on question samples through the first language model, and then obtain question-answer sample pairs; the question-answer sample pairs include question samples and their corresponding question answer texts, and the question samples include first question samples and / or second question samples.
[0102] The model introspection guidance module 540 is used to determine whether the question answer text in the question and answer sample pair contains risks through the first language model based on the question and answer sample pair when the question answer text also involves risk topics, and to correct the question answer text through the first language model to reduce the degree of risk when the first language model determines that the question answer text contains risks, thereby obtaining a corrected answer text.
[0103] The training sample acquisition module 550 is used to obtain training samples based on question samples and their corresponding corrected answer texts.
[0104] In some optional embodiments, the language model training sample acquisition system 500 may further include a training module 560. The training module 560 may be used for training a language model.
[0105] For more information about each module, see Figure 2~Figure 4 The relevant description of will not be repeated here. It should be understood that Figure 5The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or control codes contained in a processor, such as a disk, CD or DVD-ROM, etc., and such codes are provided in a carrier medium or a memory of a programmable device. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).
[0106] It should be noted that the above description of the system and its modules is only for convenience of description and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules without deviating from this principle to form a subsystem connected to other modules. Or some modules may be split to obtain more modules or multiple units under the module. Such variations are all within the scope of the disclosure of this specification.
[0107] Some embodiments of the present specification also provide a language model training sample acquisition device, including a processor and a storage medium, wherein the storage medium is used to store computer instructions, and the processor is used to execute at least part of the computer instructions to implement the language model training sample acquisition method shown in one or more embodiments of the present specification.
[0108] Some embodiments of the present specification also provide a storage medium storing computer instructions, which, when at least part of the computer instructions are executed by a processor, can implement the language model training sample acquisition method shown in one or more embodiments of the present specification.
[0109] One or more embodiments of the present specification also provide a computer program product, including computer instructions or a computer program. When at least part of the computer instructions or the computer program is executed by a processor, the language model training sample acquisition method shown in one or more embodiments of the present specification can be implemented.
[0110] In some embodiments, the processor may be a combination of one or more of the following processors: a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a programmable logic controller (PLC), a reduced instruction set computer (RISC), and a microprocessor.
[0111] In some embodiments, the storage medium may include a combination of one or more of the following: a mass storage, a removable storage, a volatile read-write memory, and a read-only memory (ROM). Exemplary mass storage may include a magnetic disk, an optical disk, a solid-state hard disk, and the like. Exemplary removable storage may include a flash disk, a floppy disk, an optical disk, a memory card, a compressed hard disk, a magnetic tape, and the like. Exemplary volatile read-write memory may include a random access memory (RAM). Exemplary random access memory may include a dynamic random access memory (DRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), a static random access memory (SRAM), a thyristor random access memory (T-RAM), and a zero capacitance memory (Z-RAM), and the like. Exemplary read-only memory may include a masked read-only memory (MROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), and a digital multifunction hard disk read-only memory, and the like.
[0112] The beneficial effects that may be brought by the embodiments of this specification include but are not limited to: 1) Automatically generating training samples through the first language model reduces labor costs and increases the diversity of training samples. 2) Training the language model through the training samples obtained in some embodiments of this specification helps to improve the safety of the language model, so that the output answers to questions are more in line with ethical and regulatory requirements, and reduce bad guidance. 3) Iterative training of the first language model based on the training samples can, on the one hand, increase the sensitivity of the model to the text of the answers to questions involving risk topics, and at the same time strengthen the ability of the model to output positive answers. On the other hand, through supervised training, the first language model can be more directly and naturally output positive answers when encountering questions involving the same risk topic, giving users positive guidance, thereby better improving the performance of the first language model. 4) By adding the above description of the dialogue and the question statement to the first question sample, the accuracy of the first language model in identifying risk topics can be further improved. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.
[0113] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are taught in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.
Claims
1. A method for obtaining language model training samples, characterized in that: include: Get the first question sample; The first sample of questions involves risk topics; Generate a second question sample involving a risk topic by referring to the first question sample through a first language model; Generate a question answer text based on the question sample through the first language model, thereby obtaining a question-answer sample pair; the question-answer sample pair includes the question sample and its corresponding question answer text, and the question sample includes the first question sample and / or the second question sample; determining whether the question answer text in the question and answer sample pair involves a risk topic; when the question answer text in the question and answer sample pair involves a risk topic, judging whether the question answer text therein contains risks based on the question and answer sample pair by using the first language model; and when the first language model judges that the question answer text contains risks, correcting the question answer text by using the first language model to reduce the risk level, thereby obtaining a corrected answer text; The training sample is obtained based on the question sample and its corresponding corrected answer text.
2. The method according to claim 1, characterized in that The step of generating a second question sample involving a risk topic by referring to the first question sample through the first language model includes: Acquire a first calling text template, where the first calling text template includes one or more first question sample slots, which are used to instruct the first language model to generate a second question sample with reference to the first question sample; Select one or more first question samples from the first question library and fill them into one or more first question sample slots in the first call text template to obtain a first call text; The first calling text is input into the first language model to obtain a second question sample output by the first language model.
3. The method according to claim 1, characterized in that Determining whether the question answer text in the question-answer sample pair involves a risk topic includes: Processing the question-answer sample pairs through a judgment model to obtain the risk level of the question answer text in the question-answer sample pairs; When the risk level is greater than a set risk threshold, it is determined that the question answer text in the question and answer sample pair involves a risk topic.
4. The method according to claim 1 or 3, characterized in that: The determining whether there is a risk in the question answer text based on the question and answer sample pair by the first language model includes: Select one or more risk inquiry texts from the second question bank; The one or more risk inquiry texts are input into the first language model to obtain one or more judgment results of whether the question answer text output by the first language model contains risks.
5. The method according to claim 4, characterized in that When the first language model determines that the question answer text has a risk, the question answer text is corrected by the first language model to reduce the risk level to obtain a corrected answer text, including: when at least one of the one or more judgment results reflects that the question answer text has a risk, then: selecting one or more correction instruction texts from a third question bank; The one or more correction instruction texts are input into the first language model to obtain one or more correction reply texts output by the first language model.
6. The method according to claim 1, characterized in that The first question sample includes a previous description of the conversation and a question statement, wherein the previous description of the conversation and the question statement are respectively related to different steps of a risky behavior.
7. The method according to claim 1, characterized in that The training samples are used to iteratively train the first language model.
8. A language model training sample acquisition system, characterized in that: include: A first question sample acquisition module, used to acquire a first question sample; The first sample of questions involves risk topics; A second question sample generating module, configured to generate a second question sample involving a risk topic by referring to the first question sample through a first language model; a question-answer sample pair acquisition module, configured to generate a question answer text based on a question sample through the first language model, thereby obtaining a question-answer sample pair; the question-answer sample pair includes the question sample and its corresponding question answer text, and the question sample includes the first question sample and / or the second question sample; a model introspection guidance module, for determining, by using the first language model and based on the question and answer sample pair, whether the question answer text in the question and answer sample pair contains risks when the question answer text in the question and answer sample pair involves a risk topic, and for correcting the question answer text by using the first language model to reduce the risk level when the first language model determines that the question answer text contains risks, thereby obtaining a corrected answer text; The training sample acquisition module is used to obtain the training sample based on the question sample and its corresponding corrected answer text.
9. A language model training sample acquisition device, comprising a processor and a storage medium, wherein the storage medium is used to store computer instructions, and the processor is used to execute at least part of the computer instructions to implement the method according to any one of claims 1 to 7.
10. A computer program product, comprising computer instructions or a computer program, wherein at least a part of the computer instructions or the computer program can implement the method according to any one of claims 1 to 7 when being executed by a processor.
