Method and apparatus for constructing pre-training model based on reinforcement learning, and electronic device

Abstract

A method, apparatus, and electronic device for constructing a pre-trained model based on reinforcement learning are disclosed. The pre-trained model includes a first pre-trained language model serving as the agent in reinforcement learning and a second pre-trained language model serving as the environment for reinforcement learning. The method includes: using prompt text generated by the first pre-trained language model as an action, using response text generated by the second pre-trained language model through reasoning on the prompt text as state feedback, and using the reward value corresponding to the prompt text calculated based on the risk detection result of the response text as the reward, with maximizing the reward value as the optimization objective, to perform reinforcement learning training on the first pre-trained language model; using the risk prompt text as a training sample and the expected response text as a sample label, to perform supervised optimization training on the second pre-trained language model, guiding it to generate a non-risk response text based on the input risk prompt text.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method, apparatus and electronic device for constructing a pre-trained model based on reinforcement learning. Background Technology

[0002] Pre-trained language models are widely used in various scenarios due to their powerful language understanding and generation capabilities. However, language models can also be misled into generating risky content. Therefore, it is necessary to identify input text that might lead language models to generate risky information in order to control the risks associated with language models. Summary of the Invention

[0003] This specification provides an embodiment of a method, apparatus, and electronic device for constructing a pre-trained model based on reinforcement learning.

[0004] According to a first aspect of the embodiments of this specification, a method for constructing a pre-trained model based on reinforcement learning is provided. The pre-trained model includes a first pre-trained language model serving as an agent in reinforcement learning and a second pre-trained language model serving as an environment for reinforcement learning. The method includes:

[0005] The first pre-trained language model generates a prompt text as the action corresponding to the reinforcement learning, the second pre-trained language model infers and generates a response text as the state feedback corresponding to the reinforcement learning, and the reward value calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning. The goal is to maximize the reward value. Reinforcement learning is then performed on the first pre-trained language model to guide it in generating risk prompt text. If the risk detection result of the response text indicates that the response text is risky, the reward value is a positive reward value.

[0006] During the reinforcement learning training of the first pre-trained language model, a risk warning text generated by the first pre-trained language model and a risk-free expected response text configured for the risk warning text are obtained; wherein, the risk warning text is the warning text corresponding to the response text with a calculated reward value of positive reward value;

[0007] The risk warning text is used as a training sample, and the expected response text configured for the risk warning text is used as a sample label. The second pre-trained language model is further subjected to supervised optimization training to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

[0008] Optionally, the step of using the prompt text generated by the first pre-trained language model as the action corresponding to the reinforcement learning, using the response text generated by the second pre-trained language model through inference on the input prompt text as the state feedback corresponding to the reinforcement learning, and using the reward value corresponding to the prompt text calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, with maximizing the reward value as the optimization objective, to perform reinforcement learning training on the first pre-trained language model includes:

[0009] The first pre-trained language model is trained iteratively with the following training process as the optimization objective: maximizing the reward value.

[0010] Obtain the prompt text generated by the first pre-trained language model, input the prompt text into the second pre-trained language model, and obtain the response text obtained by the second pre-trained language model inference on the prompt text;

[0011] A risk detection is performed on the response text to determine whether the response text is risky information, and a reward function is run to calculate the reward value corresponding to the prompt text based on the risk detection result.

[0012] The reinforcement learning algorithm is run to update the model parameters of the first pre-trained language model based on the reward value.

[0013] Optionally, the step of performing risk detection on the response text to determine whether the response text is risky information includes:

[0014] The response text is matched with risk information stored in a preset database;

[0015] If the response text successfully matches any risk information, the response text is determined to be risk information.

[0016] Optionally, if the risk detection result for the response text indicates that the response text is not risk information, the reward value is a negative reward value or a zero reward value.

[0017] Optionally, during the reinforcement learning training of the first pre-trained language model, the model parameters of the second pre-trained language model remain unchanged.

[0018] Optionally, the first pre-trained language model includes a question-and-answer language model, which is used to generate corresponding risk-free dialogue response text in response to the input dialogue initiation text.

[0019] According to a second aspect of the embodiments of this specification, an apparatus for constructing a pre-trained model based on reinforcement learning is provided. The pre-trained model includes a first pre-trained language model serving as an agent in reinforcement learning and a second pre-trained language model serving as an environment for reinforcement learning. The apparatus includes:

[0020] The reinforcement learning unit uses the prompt text generated by the first pre-trained language model as the action corresponding to the reinforcement learning, the response text generated by the second pre-trained language model through reasoning on the input prompt text as the state feedback corresponding to the reinforcement learning, and the reward value calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, with maximizing the reward value as the optimization objective. Reinforcement learning is then performed on the first pre-trained language model to guide it in generating risk prompt text; wherein, if the risk detection result of the response text indicates that the response text is risky information, the reward value is a positive reward value.

[0021] The text acquisition unit, during the reinforcement learning training of the first pre-trained language model, acquires a risk warning text generated by the first pre-trained language model, and a risk-free expected response text configured for the risk warning text; wherein, the risk warning text is the prompt text corresponding to the response text with a calculated reward value of positive reward value.

[0022] The model optimization unit uses the risk warning text as a training sample and the expected response text configured for the risk warning text as a sample label to further perform supervised optimization training on the second pre-trained language model, so as to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

[0023] Optionally, the reinforcement learning unit includes:

[0024] The first pre-trained language model is trained by iteratively executing the following sub-units with the optimization objective of maximizing the reward value:

[0025] The sub-unit is obtained by obtaining the prompt text generated by the first pre-trained language model, inputting the prompt text into the second pre-trained language model, and obtaining the response text obtained by the second pre-trained language model through reasoning on the prompt text.

[0026] The detection subunit performs risk detection on the response text to determine whether the response text is risk information, and runs a reward function to calculate the reward value corresponding to the prompt text based on the risk detection result.

[0027] Update the sub-unit and run the reinforcement learning algorithm to update the model parameters of the first pre-trained language model based on the reward value.

[0028] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0029] processor;

[0030] Memory used to store processor-executable instructions;

[0031] The processor is configured to construct any of the above-mentioned methods for pre-trained models based on reinforcement learning.

[0032] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the above-described methods for constructing a pre-trained model based on reinforcement learning.

[0033] This specification provides an embodiment of a scheme for constructing a pre-trained model based on reinforcement learning. On one hand, by introducing the concept of reinforcement learning, a first pre-trained language model and a second pre-trained language model can engage in adversarial training, thereby guiding the first pre-trained language model to generate risk warning text. On the other hand, by configuring non-risky expected response texts for the risk warning texts generated by the first pre-trained language model, and using these risk warning texts as training samples and the configured expected response texts as sample labels, supervised optimization training is performed on the second pre-trained language model. This further guides the second pre-trained language model to generate non-risky response texts based on the input risk warning texts. In this way, the generation of risky response texts by the second pre-trained language model can be avoided, thereby achieving risk control over the second pre-trained language model. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the reinforcement learning architecture provided in one embodiment of this specification;

[0035] Figure 2 This is a flowchart of a method for constructing a pre-trained model based on reinforcement learning, provided in one embodiment of this specification.

[0036] Figure 3 This is a system architecture diagram of a pre-trained language model based on the idea of ​​reinforcement learning, provided in one embodiment of this specification.

[0037] Figure 4 This is a hardware structure diagram of a device for constructing a pre-trained model based on reinforcement learning, provided in one embodiment of this specification.

[0038] Figure 5 This is a module of the apparatus for constructing a pre-trained model based on reinforcement learning, provided in one embodiment of this specification. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0040] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0042] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0043] Pre-trained language models (also known as large language models) are widely used in various scenarios due to their powerful language understanding and generation capabilities. However, in practical applications, these models may be misled into generating risky content; for example, content that violates ethical, legal, and social norms.

[0044] In related technologies, the current regulatory methods for pre-trained language models are usually done manually; for example, manual review is used to review and filter the content generated by the model using a set of rules.

[0045] However, manual review is not only inefficient and weak in supervision, but it can only supervise the risky content generated by the model after the fact. It cannot detect in advance those prompts that mislead the model to generate risky content. This may result in the user not getting the answer they want after inputting the prompt text into the model, because the content generated by the model is illegal, thus affecting the user experience.

[0046] Based on this, this specification proposes a technical solution that uses reinforcement learning to guide a pre-trained language model to generate warning text containing risky content in advance, and then optimizes and trains another pre-trained language model based on these warning texts, so as to guide the pre-trained language model to generate risk-free response text based on the risky warning text input by the user.

[0047] Please see Figure 1 , Figure 1 This is a system architecture diagram of reinforcement learning shown in this specification.

[0048] Reinforcement learning is a machine learning paradigm. Based on the idea of ​​reinforcement learning, it describes how an agent can maximize the rewards that can be obtained in a complex and uncertain environment.

[0049] In reinforcement learning, the agent continuously interacts with the environment, outputting an action as a decision based on the environment's state feedback. This decision is then fed back to the environment, which, based on the agent's decision, outputs the next state and provides the agent with a reward corresponding to the current decision by running a reward function. The agent's goal is to obtain as much reward as possible from the environment, prioritizing maximizing the reward obtained and continuously optimizing and adjusting the model's parameters.

[0050] In this specification, the following can be used Figure 1 The concept of reinforcement learning is shown to guide a pre-trained language model to generate warning text containing risky content in advance.

[0051] Please refer to the following: Figure 2 This document introduces an embodiment of a method for constructing a pre-trained model based on reinforcement learning, which can be applied to the aforementioned methods. Figure 1The reinforcement learning system shown includes the following method:

[0052] Step 210: The prompt text generated by the first pre-trained language model is used as the action corresponding to the reinforcement learning; the response text generated by the second pre-trained language model inference on the input prompt text is used as the state feedback corresponding to the reinforcement learning; and the reward value corresponding to the prompt text calculated based on the risk detection result of the response text is used as the reward corresponding to the reinforcement learning. The goal is to maximize the reward value. Reinforcement learning training is performed on the first pre-trained language model to guide it to generate risk prompt text. If the risk detection result of the response text indicates that the response text is risky information, the reward value is a positive reward value.

[0053] The first pre-trained language model can be used to generate prompt text. The second pre-trained language model can be used to infer the corresponding response text from the input prompt text.

[0054] In this description, a technical solution can be proposed that uses reinforcement learning to guide a first pre-trained language model to generate warning text containing risky content in advance, and then optimize and train a second pre-trained language model based on these warning texts, so as to guide the second pre-trained language model to generate risk-free response text based on the risk warning text input by the user.

[0055] Please see Figure 3 , Figure 3 This is a system architecture diagram of a pre-trained language model based on the idea of ​​reinforcement learning, as shown in this specification.

[0056] like Figure 3 As shown, when training the first pre-trained language model using reinforcement learning, the first pre-trained language model can serve as the reinforcement learning agent, and the prompt text generated by the first pre-trained language model can serve as the reinforcement learning action. The prompt text generated by the first pre-trained language model can then be input into the second pre-trained language model.

[0057] The aforementioned second pre-trained language model can serve as an environment for reinforcement learning. The response text generated by this second pre-trained language model, corresponding to the input prompt text, can serve as state feedback for reinforcement learning.

[0058] In addition, a reward function can be configured on the second pre-trained language model. After the second pre-trained language model generates response text corresponding to the input prompt text, the reward function can be run to calculate a corresponding reward value for the input prompt text, and the reward value is fed back to the first pre-trained language model.

[0059] The aforementioned return value can be further calculated after risk detection is performed on the response text to determine whether the response text is risky information.

[0060] For example, performing risk detection on the response text to determine whether the response text is risky information may include:

[0061] The response text is matched with risk information stored in a preset database;

[0062] If the response text successfully matches any risk information, the response text is determined to be risk information.

[0063] In implementation, the identified risk information is stored in a preset database, which is then used to determine whether the response text generated by the second pre-trained language model is risky.

[0064] For example, the text similarity between the response text and risk information stored in a preset database can be calculated. If the text similarity is greater than a threshold, the response text can be determined to be risk information; conversely, if the text similarity is less than or equal to the threshold, the response text can be determined to be either not risk information or non-risk information. The methods for calculating text similarity can include cosine similarity algorithms, Euclidean distance algorithms, etc.

[0065] In this specification, the above-mentioned reward function can output different reward values ​​for response texts containing risk information and those containing non-risk information. If the risk detection result for the response text indicates that the response text is risk information, the reward value is a positive reward value; if the risk detection result for the response text indicates that the response text is not risk information, the reward value is a negative reward value or a zero reward value.

[0066] By using the reward value corresponding to the prompt text calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, and with maximizing the reward value as the optimization objective, reinforcement learning training is performed on the first pre-trained language model to guide the first pre-trained language model to generate risk prompt text.

[0067] On the aforementioned first pre-trained language model, a reinforcement learning algorithm for optimizing and updating the model parameters can also be configured. When the first pre-trained language model receives the reward value fed back by the first pre-trained language model, the reinforcement learning algorithm can be run to update the model parameters of the first pre-trained language model based on the reward value.

[0068] The specific type of reinforcement learning algorithm mentioned above is not specifically limited in this specification; for example, it may include the PPO (Proximal Policy Optimization) algorithm, the DQN (Deep Q-Network) algorithm, or the Actor-Critical algorithm. The reinforcement learning algorithm updates the model parameters of the first pre-trained language model, which acts as the agent, making it more inclined to generate prompt text that yields positive rewards, i.e., guiding the first pre-trained language model to generate risk warning text.

[0069] In one embodiment shown, when training the first pre-trained language model using reinforcement learning, the training process can be iteratively performed to maximize the reward value as the optimization objective, thereby training the first pre-trained language model:

[0070] Obtain the prompt text generated by the first pre-trained language model, input the prompt text into the second pre-trained language model, and obtain the response text obtained by the second pre-trained language model inference on the prompt text;

[0071] A risk detection is performed on the response text to determine whether the response text is risky information, and a reward function is run to calculate the reward value corresponding to the prompt text based on the risk detection result.

[0072] The reinforcement learning algorithm is run to update the model parameters of the first pre-trained language model based on the reward value.

[0073] like Figure 3 As shown, the prompt information generated by the first pre-trained language model as an agent can be input into the second pre-trained language model; the second pre-trained language model infers from the input prompt text to generate response text, performs risk detection on the generated response text, runs a reward function based on the risk detection results to calculate the reward value, and then feeds the response text and reward value back to the first pre-trained language model.

[0074] In the reinforcement learning process, the first pre-trained language model is trained in multiple rounds of iteration with the optimization objective of maximizing the cumulative reward value of each response text, thereby guiding the first pre-trained language model to generate risk warning text.

[0075] In one embodiment shown, the model parameters of the second pre-trained language model can remain unchanged during reinforcement learning training of the first pre-trained language model.

[0076] It should be noted that the specific forms of the first pre-trained language model and the second pre-trained language model described above are not particularly limited in this specification. For example, in one embodiment shown, the first pre-trained language model and the second pre-trained language model may include question-and-answer language models. In this scenario, according to the reinforcement learning process described above, the first pre-trained language model, once trained, can generate a risk-free dialogue response text corresponding to the input dialogue initiation text.

[0077] Step 220: During the reinforcement learning training of the first pre-trained language model, obtain the risk warning text generated by the first pre-trained language model, and the expected response text without risk configured for the risk warning text; wherein, the risk warning text is the warning text corresponding to the response text with a calculated reward value of positive reward value.

[0078] Step 230: Using the risk warning text as a training sample and the expected response text configured for the risk warning text as a sample label, further supervised optimization training is performed on the second pre-trained language model to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

[0079] Generally, the training of large models can include a pre-training phase and a fine-tuning phase. Typically, through a combination of pre-training and fine-tuning phases, large models can learn extensive language knowledge and contextual understanding, and produce coherent and logical answers.

[0080] During the pre-training phase, a large-scale text corpus can be used for unsupervised learning of the large model. This text corpus can include text data from various sources such as web pages, articles, books, and forum posts on the internet. A pre-selected task can use masking to cover certain words in the input text, and then have the large model predict these masked words. Such prediction tasks help the large model understand context and language concepts.

[0081] During the fine-tuning phase, supervised learning can be performed on a pre-trained large model using a manually created dialogue dataset. This dataset can contain dialogue pairs, which may include user-input questions and model-response answers. The large model is trained by taking questions as input and generating appropriate answers. During optimization, reinforcement learning techniques can be incorporated to evaluate and adjust the generated answers, thereby improving the performance of the large model's output responses.

[0082] Large, conventional models are ready to use after completing the pre-training and fine-tuning phases. Building upon this, an additional "instruction fine-tuning" phase can be introduced to optimize the model's performance (response speed, output accuracy, etc.). This phase is also known as performance optimization. Performance optimization can be achieved through training or not. It can involve directly adjusting model parameters using a set of instructions; alternatively, it can involve inputting labeled data and adjusting model parameters through supervised training. Specific optimization methods include the two mentioned above.

[0083] In this specification, when using Figure 3 The reinforcement learning method shown involves training the first pre-trained language model. At this point, the first pre-trained language model has been guided to generate risk warning texts containing risk content in advance through reinforcement learning. Based on this, a non-risk expected response text can be configured for each risk warning text.

[0084] Then, the risk warning text is used as training samples, and the expected response text configured for the risk warning text is used as sample labels. These training samples and sample labels are then used to perform supervised optimization training on the second pre-trained language model. The goal of the optimization training is to enable the second pre-trained language model to generate risk-free response text for the input risk warning text.

[0085] In this way, the risk of generating risky response text by the second pre-trained language model can be avoided, thereby achieving risk control over the second pre-trained language model.

[0086] Corresponding to the aforementioned embodiments of the method for constructing a pre-trained model based on reinforcement learning, this specification also provides embodiments of a device for constructing a pre-trained model based on reinforcement learning. These devices can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the device loading the corresponding computer program from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of the device used to construct the pre-trained model based on reinforcement learning, as described in this specification. (Except for...) Figure 4 In addition to the processor, network interface, memory, and non-volatile memory shown, the device in the embodiment may also include other hardware, depending on the actual functionality built based on the reinforcement learning-based pre-trained model, which will not be described in detail here.

[0087] Please see Figure 5 This is a block diagram of a device for constructing a pre-trained model based on reinforcement learning, provided in one embodiment of this specification. The device corresponds to... Figure 2 The illustrated embodiment.

[0088] The pre-trained model includes a first pre-trained language model serving as the agent for reinforcement learning, and a second pre-trained language model serving as the environment for reinforcement learning. The device includes:

[0089] The reinforcement learning unit 410 uses the prompt text generated by the first pre-trained language model as the action corresponding to the reinforcement learning, the response text generated by the second pre-trained language model through reasoning on the input prompt text as the state feedback corresponding to the reinforcement learning, and the reward value corresponding to the prompt text calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, with maximizing the reward value as the optimization objective. It trains the first pre-trained language model through reinforcement learning to guide it to generate risk prompt text; wherein, if the risk detection result of the response text indicates that the response text is risk information, the reward value is a positive reward value.

[0090] The text acquisition unit 420, during the reinforcement learning training of the first pre-trained language model, acquires a risk warning text generated by the first pre-trained language model, and a risk-free expected response text configured for the risk warning text; wherein, the risk warning text is the prompt text corresponding to the response text with a calculated reward value of positive reward value.

[0091] The model optimization unit 430 uses the risk warning text as a training sample and the expected response text configured for the risk warning text as a sample label to further perform supervised optimization training on the second pre-trained language model, so as to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

[0092] Optionally, the reinforcement learning unit 410 includes:

[0093] The first pre-trained language model is trained by iteratively executing the following sub-units with the optimization objective of maximizing the reward value:

[0094] The sub-unit is obtained by obtaining the prompt text generated by the first pre-trained language model, inputting the prompt text into the second pre-trained language model, and obtaining the response text obtained by the second pre-trained language model through reasoning on the prompt text.

[0095] The detection subunit performs risk detection on the response text to determine whether the response text is risk information, and runs a reward function to calculate the reward value corresponding to the prompt text based on the risk detection result.

[0096] Update the sub-unit and run the reinforcement learning algorithm to update the model parameters of the first pre-trained language model based on the reward value.

[0097] Optionally, the reinforcement learning unit 410 performs risk detection on the response text to determine whether the response text is risk information, further including:

[0098] The response text is matched with risk information stored in a preset database; if the response text is successfully matched with any risk information, the response text is determined to be risk information.

[0099] Optionally, if the risk detection result for the response text indicates that the response text is not risk information, the reward value is a negative reward value or a zero reward value.

[0100] Optionally, during the reinforcement learning training of the first pre-trained language model, the model parameters of the second pre-trained language model remain unchanged.

[0101] Optionally, the first pre-trained language model includes a question-and-answer language model, which is used to generate corresponding risk-free dialogue response text in response to the input dialogue initiation text.

[0102] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0103] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0104] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0105] above Figure 5 The internal functional modules and structural diagram of a device for building a pre-trained model based on reinforcement learning are described. Its essential execution entity can be an electronic device, including:

[0106] processor;

[0107] Memory used to store processor-executable instructions;

[0108] The processor is configured to execute an embodiment of any of the above-described methods for constructing a pre-trained model based on reinforcement learning.

[0109] In the embodiments of the above-described electronic device, it should be understood that the processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor, and the aforementioned memory can be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive. The steps of the methods disclosed in the embodiments of this specification can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0110] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0111] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0112] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this specification does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0113] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0114] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0120] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0121] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0123] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0124] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for constructing a pre-trained model based on reinforcement learning, the pre-trained model comprising a first pre-trained language model serving as an agent in reinforcement learning, and a second pre-trained language model serving as an environment for reinforcement learning; the method comprising: The first pre-trained language model generates a prompt text as the action corresponding to the reinforcement learning, the second pre-trained language model infers and generates a response text as the state feedback corresponding to the reinforcement learning, and the reward value calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning. The goal is to maximize the reward value. Reinforcement learning is then performed on the first pre-trained language model to guide it in generating risk prompt text. If the risk detection result of the response text indicates that the response text is risky, the reward value is a positive reward value. During the reinforcement learning training of the first pre-trained language model, a risk warning text generated by the first pre-trained language model and a risk-free expected response text configured for the risk warning text are obtained; wherein, the risk warning text is the warning text corresponding to the response text with a calculated reward value of positive reward value; The risk warning text is used as a training sample, and the expected response text configured for the risk warning text is used as a sample label. The second pre-trained language model is further subjected to supervised optimization training to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

2. The method according to claim 1, wherein the step of using the prompt text generated by the first pre-trained language model as the action corresponding to the reinforcement learning, using the response text generated by the second pre-trained language model through inference in response to the input prompt text as the state feedback corresponding to the reinforcement learning, and using the reward value corresponding to the prompt text calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, with maximizing the reward value as the optimization objective, and performing reinforcement learning training on the first pre-trained language model, comprises: The first pre-trained language model is trained iteratively with the following training process as the optimization objective: maximizing the reward value. Obtain the prompt text generated by the first pre-trained language model, input the prompt text into the second pre-trained language model, and obtain the response text obtained by the second pre-trained language model inference on the prompt text; A risk detection is performed on the response text to determine whether the response text is risky information, and a reward function is run to calculate the reward value corresponding to the prompt text based on the risk detection result. The reinforcement learning algorithm is run to update the model parameters of the first pre-trained language model based on the reward value.

3. The method according to claim 2, wherein performing risk detection on the response text to determine whether the response text is risk information includes: The response text is matched with risk information stored in a preset database; If the response text successfully matches any risk information, the response text is determined to be risk information.

4. The method according to claim 1, wherein if the risk detection result for the response text indicates that the response text is not risk information, the reward value is a negative reward value or a zero reward value.

5. The method according to claim 1, wherein during the reinforcement learning training of the first pre-trained language model, the model parameters of the second pre-trained language model remain unchanged.

6. The method according to claim 1, wherein the first pre-trained language model includes a question-and-answer language model, the question-and-answer language model being used to generate a corresponding risk-free dialogue response text in response to the input dialogue initiation text.

7. An apparatus for constructing a pre-trained model based on reinforcement learning, the pre-trained model comprising a first pre-trained language model as an agent in reinforcement learning and a second pre-trained language model as an environment for reinforcement learning, the apparatus comprising: The reinforcement learning unit uses the prompt text generated by the first pre-trained language model as the action corresponding to the reinforcement learning, the response text generated by the second pre-trained language model through reasoning on the input prompt text as the state feedback corresponding to the reinforcement learning, and the reward value calculated based on the risk detection result of the response text as the reward corresponding to the reinforcement learning, with maximizing the reward value as the optimization objective. Reinforcement learning is then performed on the first pre-trained language model to guide it in generating risk prompt text; wherein, if the risk detection result of the response text indicates that the response text is risky information, the reward value is a positive reward value. The text acquisition unit, during the reinforcement learning training of the first pre-trained language model, acquires a risk warning text generated by the first pre-trained language model, and a risk-free expected response text configured for the risk warning text; wherein, the risk warning text is the prompt text corresponding to the response text with a calculated reward value of positive reward value. The model optimization unit uses the risk warning text as a training sample and the expected response text configured for the risk warning text as a sample label to further perform supervised optimization training on the second pre-trained language model, so as to guide the second pre-trained language model to generate a non-risk response text based on the input risk warning text.

8. The apparatus of claim 7, wherein the reinforcement learning unit comprises: The first pre-trained language model is trained by iteratively executing the following sub-units with the optimization objective of maximizing the reward value: The sub-unit is to obtain the prompt text generated by the first pre-trained language model, input the prompt text into the second pre-trained language model, and obtain the response text obtained by the second pre-trained language model inference on the prompt text; The detection subunit performs risk detection on the response text to determine whether the response text is risk information, and runs a reward function to calculate the reward value corresponding to the prompt text based on the risk detection result. Update the sub-unit and run the reinforcement learning algorithm to update the model parameters of the first pre-trained language model based on the reward value.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method according to any one of claims 1-6.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method as described in any one of claims 1-6.

Images