Method, device and product for generating response

Abstract

A method, an apparatus, and a computer program product for generating a response are presented. The method comprises the following steps: receiving a first prompt word to be input into a first language model; the first language model represents an association relationship between the cue word and the response. The response is detectable by a first detector model corresponding to the first language model. A detection score of the response by the first detector model indicates that the response is generated by the first language model. And generating a first response corresponding to the first cue word by the first language model. The first response is detectable by the first detector model. By using the embodiments of the invention, the LM output marking watermark can be more stable and has better detection accuracy, adaptability and open source feasibility.

Description

Technical Field

[0001] The present disclosure relates generally to computer technology and, more particularly, to methods, apparatus, and computer program products for generating responses. Background Art

[0002] Watermarking language model (LM) outputs (i.e., embedding algorithmically detectable signals into LM-generated text) has recently emerged as a potential solution for tracking LM misuse. Here, it is desirable to find an effective method to watermark LM responses and track whether a response is generated by the model. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for generating a response is provided. In the method, a first prompt word to be input into a first language model is received. The first language model represents an association relationship between the prompt word and the response. The response can be detected by a first detector model corresponding to the first language model. A detection score of the response by the first detector model indicates that the response is generated by the first language model. A first response corresponding to the first prompt word is generated by the first language model. The first response can be detected by the first detector model.

[0004] In a second aspect of the present disclosure, an electronic device is provided, comprising: a computer processor coupled to a computer readable memory unit comprising instructions, which, when executed by the computer processor, implement the method according to the first aspect of the present disclosure.

[0005] In a third aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by an electronic device to enable the electronic device to perform the method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other objects, features and advantages of the present disclosure will become more apparent by describing some embodiments of the present disclosure in more detail in the accompanying drawings, in which like reference numerals generally refer to like parts in the embodiments of the present disclosure.

[0008] Figure 1 An example environment for generating responses based on machine learning techniques is shown;

[0009] Figure 2 An example diagram for generating a response according to an embodiment of the present disclosure is shown;

[0010] Figure 3 An example diagram of a pre-trained detector model according to an embodiment of the present disclosure is shown;

[0011] Figure 4 An example diagram showing iteratively training a language model and a detector model according to an embodiment of the present disclosure is shown;

[0012] Figure 5 An example diagram showing a target for updating a detector model and a language model according to an embodiment of the present disclosure;

[0013] Figure 6 An experimental algorithm of the proposed solution according to an embodiment of the present disclosure is shown;

[0014] Figure 7 An example flow chart showing a method for generating a response according to an embodiment of the present disclosure; and

[0015] Figure 8 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these implementations are only described for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in various ways except for the mode described below.

[0017] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0018] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. In addition, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0019] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more listed terms.

[0020] The specialized terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should be further understood that the terms "comprise", "include", "have", "contain", "include" and / or "comprising" when used herein specify the presence of stated features, elements and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0021] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these implementations are described only for the purpose of illustration and to help those skilled in the art understand and realize the purpose of the present disclosure, without any limitation to the scope of the present disclosure. The disclosure described herein can be implemented in a variety of ways except for the manner described below. In the following specification and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant rules.

[0023] It is understandable that before using the technical solutions disclosed in various embodiments of the present disclosure, the type, scope of use and usage scenarios of the personal information involved in the present disclosure should be notified to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0024] For example, in response to an active request received from a user, a prompt message is sent to the user to explicitly inform the user that the requested operation will require obtaining and using the user's personal information. Therefore, the user can independently choose whether to provide personal information to the software or hardware (such as an electronic device, application, server, or storage medium) that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0025] As an optional but non-limiting implementation, in response to an active request received from the user, the method of sending the prompt information to the user may include, for example, a pop-up window, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above-mentioned process of notifying and obtaining user authorization is merely exemplary and does not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0027] As mentioned above, it is desirable to provide an effective method to watermark the responses of language models (especially large language models) and track language model misuse. Figure 1 The sample environment in is used as an example to illustrate the problems in the process of generating responses. Figure 1 An example environment 100 for generating responses based on machine learning techniques is shown. Figure 1 As shown, the environment 100 may include a language model 120, which may output a response 130 based on a prompt word 110. Here, the prompt word 110 may be a piece of text or instruction provided to the language model 120, which is intended to guide the output or behavior of the language model 120. For example, the prompt word 110 may be a simple question, description, instruction, or scenario designed to elicit a response 130 from the language model 120.

[0028] The environment 100 also includes a module 150 that adds the tag-level watermark information to the response 130 to generate the response 140 (ie, the watermarked response 130). The module 150 may be implemented as a model.

[0029] To date, methods for generating watermarked responses have focused on token-level distortions in the output of language models. This framework has several limitations. For example, the watermarked text still needs to be human-readable. The induced output distortion needs to be minimized. Therefore, watermark accuracy can be suboptimal because the watermark signal injected into the output space is constrained by the readability trade-off. For the same reason, the bounded output distortion leads to vulnerability to paraphrase attacks. In another example, the design space of the watermark is inflexible, and all practitioners can do is post-process the text generated from the fixed language model, which leads to some problems such as lack of adaptability to newly discovered adversarial attacks. In another example, practitioners are prohibited from open-sourcing the watermarked language model. If practitioners want to do this, they must also release the language model without the watermark, because the watermark is added later, thus undermining the original purpose of protecting intellectual property.

[0030] In view of this, the present disclosure proposes a scheme for generating responses. In general, watermarks can be injected through model-level changes, and the resulting language model output carries a signal that can be identified by detection. In other words, in contrast to existing methods that only work with fixed language models, a language model adjustment stage is included in the watermark pipeline, and the design space of the watermark is thus expanded. Unlike existing work where the detector is a simple statistical test, the present disclosure proposes a pair of LMs and detectors, and the detector predicts the likelihood of whether the text is watermarked (i.e., whether the text is generated by the paired LM). Specifically, the language model can be adjusted to inject the watermark signal while training a paired detector model that detects the signal. The key insight is that by adjusting the language model to adapt to the detector, detection is made easier and more accurate.

[0031] In the proposed solution, a first prompt word is received to be input into a first language model. The first language model represents an association relationship between the prompt word and a response. The response can be detected by a first detector model corresponding to the first language model. A detection score of the response by the first detector model indicates that the response is generated by the first language model. A first response corresponding to the first prompt word is generated by the first language model, and the first response can be detected by the first detector model.

[0032] The proposed solution has several advantages over existing work. For example, the language model is adapted to the detector and creates more space for the detector because the language model is explicitly updated to generate text that is easily detectable and it is not intended to rely on low-level (e.g., token-level) output distortions for watermark detection. Watermarks can be more robust to adversarial attacks such as rewriting. Since the proposed framework is data-driven, the language model can be easily iterated to adapt to new attacks by incorporating adversarially generated text into the training data in the form of adversarial training. This is not a feature supported by the traditional fixed model approach. Once the language model is deployed, it does not require any special operations during text generation to embed the watermark. This zero-cost watermark generation makes the present solution attractive when the language model is deployed for service at a very large scale. Since the watermark is embedded in the language model that the detector can detect, no special operations are required in subsequent text generation, and practitioners can publish watermarked language models without being forced to publish versions without watermarks.

[0033] For more details on the solution used to generate the response refer to Figure 2 . Figure 2 An example diagram 200 for generating a response according to an embodiment of the present disclosure is shown. Figure 2As shown, the first language model 220 receives the first prompt word 210. The first language model 220 generates a first response 230 corresponding to the first prompt word 210. The first response 230 can be detected by the first detector model 240. The first language model 220 corresponds to the first detector model 240, for example, the first language model 220 is paired with the first detector model 240 and trained simultaneously.

[0034] In some embodiments, the first detector model 240 may be a reward model. The detection score of the first detector model 240 for the first response 230 indicates whether the first response 230 is generated by the first language model 220. For example, if the first response 230 is generated by the first language model 220, the detection score may be high (e.g., 1), otherwise the detection score may be low (e.g., 0). Alternatively and / or additionally, the detection score may belong to a predefined range (e.g., [0, 1], etc.) that indicates the probability of whether the response is generated by the model.

[0035] In some embodiments, the detector model 240 may be a pre-trained model. Then, both the first language model 220 and the first detector model 240 may be jointly trained iteratively. In each step, the first language model 220 may be adjusted by the instructions to update its weights and thus update its output distribution. Then, the first detector model 240 may be trained to detect signals from the distorted output.

[0036] Reinforcement learning (RL) can be selected as a common training framework. The reward model is suitable as a detector. The strong generalization of the RL algorithm can be used to ensure that the fine-tuned language model can generate text that can be easily detected by the detector. The readability of the text usually needs to be preserved, which can be accomplished by the utility preservation goal of RLHF. Before introducing the present disclosure in detail, RLHF and some symbols will be introduced first.

[0037] make Represents the language model tag space. Used to represent a sequence of tokens (i.e., sentences). A language model is a function that uses a model with parameters θ to predict the probability of the next token given a sequence of tokens. Given a prompt word x, π θ (a t |s t ) is used to represent the probability distribution of the next tag, where s t = x is the current “state” following the notation in the reinforcement learning with human feedback (RLHF) literature. f(x;θ) is used to represent the text y~π generated by θ given the prompt word x in an autoregressive manner. θ (·|x).

[0038] RLHF is currently the standard pipeline for aligning language models with human preferences. In RLHF, we first train a reward model (RM) r: r(x, y) is a measure of whether the completion y given the prompt word x is the reward expected by humans. RM training requires RM dataset Where x is the prompt word, y r is the completion of the rejection, and y c is done based on human preference, and the RM is optimized to minimize

[0039] Second, given the trained reward model θ RM and the original model θ O In the case of , for the language model policy, Proximal Policy Optimization (PPO) is used to maximize the following objectives:

[0040] In the above formula (1), π θ is the learned RL policy for model θ, β is the KL reward coefficient, and γ is the strength of the KL penalty.

[0041] Some embodiments of the present disclosure will be described below with reference to the accompanying drawings. In an embodiment of the present disclosure, the first detector model 240 may be determined by acquiring a reference sample set and updating the parameters of the first detector model 240 based on the reference sample set. The reference sample set may include at least positive samples and negative samples. In this way, the detector model may have better detection performance.

[0042] Reference now Figure 3 , which shows an example process 300 of pre-training a detector model according to an embodiment of the present disclosure. The example process 300 is a pre-training process using a data set 310 (i.e., a reference sample set). The data set 310 may include a plurality of positive samples 312 and a plurality of negative samples 314. The first detector model 240 may receive these samples. For example, with respect to the positive samples 312, the first detector model 240 may be trained in a direction in which the first detector model 240 may output 1. With respect to the negative samples 314, the first detector model 240 may be trained in a direction in which the first detector model 240 may output 0. In this way, the initial first detector model 240 may be pre-trained. In this way, the parameters of the first detector model 240 may be updated for detecting whether a response includes a watermark with an acceptable level of accuracy.

[0043] In an embodiment of the present disclosure, after the first detector model 240 is pre-trained, the first detector model 240 and the language model 220 may be iteratively trained. Specifically, with the parameters of the first language model 220 fixed, the parameters of the first detector model 240 are updated based on the objective function. Then, with the parameters of the first detector model 240 fixed, the parameters of the first language model 220 are updated based on the objective function. The first language model 220 and the first detector model 240 are iteratively trained. In this way, by alternating between fine-tuning the language model to generate text that is easily detectable by the detector model and training the detector model to accurately detect the generated watermarked text, a reinforcement learning-based framework is used to jointly train a paired watermark language model and detector model.

[0044] Figure 4 An example process 400 for iteratively training a language model and a detector model according to an embodiment of the present disclosure is shown. A data set 410 may include a plurality of positive samples 412 and a plurality of negative samples 414. The first language model 220 and the first detector model 240 may be iteratively trained based on these samples. For example, for a first iteration, the first language model 220 is adjusted to fit the first detector model 240. Then, the first detector model 240 is adjusted to fit the first language model 220.

[0045] In an embodiment of the present disclosure, the objective function may include increasing the detection score associated with the positive sample. The positive sample may include a reference prompt word, a reference response for the reference prompt word, and a reference detection score for the reference response. The reference response may be a response corresponding to the reference prompt word generated by the first language model 220. The reference detection score may indicate that the reference response is generated by the first language model 220 (i.e., the reference response includes a watermark). In this way, the output of the detector model for the positive sample tends to be close to 1, and the language model and the detector model can be trained in a more accurate manner toward the direction in which the watermarked response can be detected.

[0046] As an example, given a o The original language model, the parameter θ o is fine-tuned to match the detector D: Paired language model θ w , in addition to the detector outputting a detection score, which quantifies the output y given a hint word x by the first language model 220θ w Generate the likelihood that the detector D: Has the same architecture as RM.

[0047] Let θ d Represents the parameters of the detector model. Given a prompt word x, the language model θ wWhen generating output y, D(x, y; θ d ) indicates that it comes from θ d The prediction score of θ. d The following properties can be satisfied: given the language model θ w Output y generated from prompt word x w : =f(x;θ w ), detection score D(x, y w θ d ) is high.

[0048] In an embodiment of the present disclosure, the objective function may include reducing the detection score associated with the negative sample; the negative sample may include a reference cue word, a reference response for the reference cue word, and a reference detection score for the reference response; the reference response may not be a response corresponding to the reference cue word generated by the first language model 220, and the reference detection score may indicate that the reference response is not generated by the first language model 220. In this way, the output of the detector model for the negative sample tends to be close to 0, and the language model and the detector model can be trained in a more accurate manner toward a direction in which the response without the watermark can be detected.

[0049] Continuing the example above, given that it is not represented by the first language model 220θ w The generated output y nw , e.g., output written by a human or generated by another LM (i.e., a response without a watermark), the detection score D(x, y nw θ d ) is low.

[0050] In an embodiment of the present disclosure, the objective function may include reducing the difference between the response generated by the first language model 220 and the response generated by the updated first language model. In this way, it is ensured that the updated language model is not significantly different from the original language model and can provide a correct response. For example, the output distribution is as small as possible, thereby preserving the utility from the original language model, i.e., f(x; θ w )≈f(x;θ o ).

[0051] The first detector model 240 can be used as a reward model in the RLHF pipeline so that the first language model 220 can be fine-tuned to obtain a high detection score. where y nw is the watermark-free (e.g., human-written) output corresponding to the cue word x. In an embodiment of the present disclosure, the overall objective function may be provided as follows:

[0052] refer to Figure 5 To obtain more details about the overall objective function, as described in the above example, the overall objective function 500 may include a first objective component 510, a second objective component 520, and a third objective component 530. Figure 5 As shown, f(x;θ w ) is from the first language model 220θ w Generated watermarked text, first detector model 240θ d Need to compare it with the text without watermark nw To distinguish, Reg(·,·) is a regularization term that ensures that the reliability of the generated text does not deviate much from the original language model, and λ is the penalty strength. The KL penalty is directly used as regularization in equation (1).

[0053] However, because the generated text f(x;θ w ) involves sampling So the objective function may not be optimized directly. Therefore, it can be proposed that in updating θ w and update θ d RL-based algorithms that iteratively switch between

[0054] In an embodiment of the present disclosure, given a fixed detector θ d In the case of w is adjusted to match the PPO θ d The labeled reward (i.e., detection score) of Given a fixed language model θ w , detector model θ d is trained to w Generated watermarked text y w with text y from any other source (e.g., written by a human) nw Distinguish between:

[0055] Note that unlike traditional RLHF, in PPO, the reward model, i.e., the detector model θ d , also accompanied by the language model θ w Be updated together.

[0056] In an embodiment of the present disclosure, the first language model 220 may be acquired based on a reinforcement learning process, and the first detector model 240 may be updated during the reinforcement learning process together with a reward model corresponding to the first language model 220. RL may be used to jointly train the first language model 220 and the first detector model 240. In this way, it is preferably used together with standard alignment, so that the additional overhead caused by marking watermarks can be significantly reduced.

[0057] As an example, given a normal alignment task, where the reward model is θ RM , we can use θ from PPO RM and the detector model θ d The combined reward of the two, that is, replace the marked reward with:

[0058] In the above equation (4), α is the weight for balancing the reward of the alignment task with the watermarking task. All other steps, such as language model fine-tuning, are the same. Compared with the standard RLHF pipeline, the additional cost introduced is only training an additional reward model (i.e., the first detector model 240) and running inference on it (i.e., marking the detection score). RLHF already tends to use multiple reward models, and the reward model for watermarking can be easily incorporated into the current RLHF pipeline.

[0059] In an embodiment of the present disclosure, a first detection score for a first response may be determined by a first detector model 240. In response to determining that the first detection score satisfies a predetermined condition associated with the first detector model, it may be determined that the first response was generated by a first language model. In other words, the response may be input into a paired detector model to see if the response includes a watermark. In this way, the detector model may determine in an accurate manner whether the response was generated by a paired language model.

[0060] In an embodiment of the present disclosure, a second response generated by a second language model corresponding to the second prompt word may be obtained, and a second detection score of the second response may be determined by the first detector model 240. In response to determining that the second detection score satisfies a predetermined condition, an indication that the second language model is the same as the first language model 220 may be provided. In response to determining that the second detection score does not satisfy the predetermined condition, an indication that the second language model is different from the first language model 220 may be provided.

[0061] In these embodiments, any response (e.g., a response written by a human, a response generated by a third-party LM, or a response generated by an LM paired with the detector model) can be input into the detector model, and the detector can then output a probability of whether the response was generated by the first language model 220. In this way, misuse of the first language model 220 can be detected.

[0062] For the detection process, the detection of the watermark is a simple forward pass through the detector network. Given a cue word x and an output y, the detection score D(x, y; θ d ) will be calculated. A high score indicates that the output y is likely to be represented by the language model θ w The threshold can be chosen based on the criterion that the true positive rate (TPR) reaches a certain value.

[0063] The following is combined with Figure 6 Describe the algorithm in detail. Figure 6 An experimental algorithm 600 of the proposed solution according to an embodiment of the present disclosure is shown. In lines 1 to 8, a process of pre-training a detector model may be performed. In lines 9 to 18, a process of updating a language model and a detector model may be performed.

[0064] The overall pipeline is shown in the experimental algorithm 600. First, the detector is pre-trained to distinguish the unwatermarked text y nw and by the original language model θ o Generated text (lines 1-8). Then, the original language model θ o is fine-tuned to obtain the watermarked language model weights θ w , while training the detector model θ d (Lines 9-18). In particular, at each training step, θ d First, it is frozen and the PPO objective is used to update θ w , with respect to θ w The generated text increases the value of θ from d Then, we generate the watermarked text y w , and train the detector model to w Update θ d to classify between watermarked and non-watermarked text (lines 15-17).

[0065] Multiple existing datasets and network architectures can be used for cue word completion and secure alignment. Watermark detection performance and original task performance (i.e., cue word completion and secure alignment) can be evaluated using AUC and false positive rate. Experiments show that the detection performance of the proposed solution is better than existing solutions. In addition, the responses with watermarks can involve language forms that can be detected without affecting human understanding. Examples are given in Table 1. For example, two pairs of LM and detector models can be trained based on different network architectures and / or datasets. The two LMs can output different responses (as shown in column 3), which can then be detected by corresponding detector models, respectively. Table 1: Examples of responses to different datasets

[0066] In summary, the present disclosure proposes an innovative method designed to generate watermarked and detectable responses. Using these embodiments of the present disclosure, the watermarked LM output can be more robust and have better detection accuracy, adaptability, and open source feasibility.

[0067] The above paragraphs have described the details for generating a response. According to an embodiment of the present disclosure, a method for generating a response is provided. For more details about the method, please refer to Figure 7 ,in Figure 7 An example flow chart of a method 700 for generating a response according to an embodiment of the present disclosure is shown. At box 710, a first prompt word to be input into a first language model is received. The first language model represents an association relationship between the prompt word and the response. The response can be detected by a first detector model corresponding to the first language model. The detection score of the response by the first detector model indicates that the response is generated by the first language model. At box 720, a first response corresponding to the first prompt word is generated by the first language model, and the first response can be detected by the first detector model.

[0068] In an embodiment of the present disclosure, the first detector model is determined by: acquiring a reference sample set, the reference sample set comprising at least positive samples and negative samples; and updating parameters of the first detector model based on the reference sample set.

[0069] In one embodiment of the present disclosure, the first language model and the first detector model are iteratively acquired in the following manner: when the parameters of the first language model are fixed, the parameters of the first detector model are updated based on the objective function; when the parameters of the first detector model are fixed, the parameters of the first language model are updated based on the objective function.

[0070] In an embodiment of the present disclosure, the objective function includes: increasing a detection score associated with a positive sample, the positive sample including a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response, the reference response is a response corresponding to the reference prompt generated by a first language model, and the reference detection score indicates that the reference response is generated by the first language model.

[0071] In an embodiment of the present disclosure, the objective function includes: reducing a detection score associated with a negative sample, the negative sample including a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response, the reference response is not a response corresponding to the reference prompt generated by the first language model, and the reference detection score indicates that the reference response is not generated by the first language model.

[0072] In an embodiment of the present disclosure, the objective function includes reducing the difference between the response generated by the first language model and the response generated by the updated first language model.

[0073] In an embodiment of the present disclosure, the first language model is acquired based on a reinforcement learning process, and updating the first detector model includes: updating the first detector model together with a reward model corresponding to the first language model during the reinforcement learning process.

[0074] In an embodiment of the present disclosure, method 700 further includes: determining, by a first detector model, a first detection score of the first response; and in response to determining that the first detection score satisfies a predetermined condition associated with the first detector model, determining that the first response is generated by the first language model.

[0075] In an embodiment of the present disclosure, method 700 also includes: obtaining a second response generated by a second language model corresponding to a second prompt word; determining a second detection score of the second response by the first detector model; and in response to determining that the second detection score satisfies a predetermined condition, providing an indication that the second language model is the same as the first language model.

[0076] In an embodiment of the present disclosure, method 700 further includes: in response to determining that the second detection score does not satisfy a predetermined condition, providing an indication that the second language model is different from the first language model.

[0077] According to an embodiment of the present disclosure, a device for generating a response is provided. The device includes: a receiving module, configured to receive a first prompt word to be input into a first language model, the first language model represents the association relationship between the prompt word and the response, the response can be detected by a first detector model corresponding to the first language model, and the detection score of the response by the first detector model indicates that the response is generated by the first language model. The device also includes a generating module, configured to be used by the first language model to generate a first response corresponding to the first prompt word, and the first response can be detected by the first detection model. The device also includes other modules, configured to implement other steps in the above method.

[0078] According to an embodiment of the present disclosure, an electronic device for implementing method 700 is provided. The electronic device includes: a computer processor coupled to a computer-readable memory unit, the memory unit including instructions for implementing a method for generating a response when executed by the computer processor. The method includes: receiving a first prompt word to be input into a first language model, the first language model representing an association relationship between the prompt word and the response, and the response can be detected by a first detector model corresponding to the first language model. The detection score of the response by the first detector model indicates that the response is generated by the first language model; and generating a first response corresponding to the first prompt word by the first language model, the first response can be detected by the first detector model.

[0079] In an embodiment of the present disclosure, the first detector model is determined by: acquiring a reference sample set, the reference sample set at least including positive samples and negative samples; and updating parameters of the first detector model based on the reference sample set.

[0080] In an embodiment of the present disclosure, the first language model and the first detector model are iteratively acquired in the following manner: when the parameters of the first language model are fixed, the parameters of the first detector model are updated based on the objective function; and when the parameters of the first detector model are fixed, the parameters of the first language model are updated based on the objective function.

[0081] In an embodiment of the present disclosure, the objective function includes: increasing a detection score associated with a positive sample, the positive sample including a reference prompt, a reference response to the reference prompt word, and a reference detection score for the reference response, the reference response is a response corresponding to the reference prompt word generated by a first language model, and the reference detection score indicates that the reference response is generated by the first language model.

[0082] In an embodiment of the present disclosure, the objective function includes: reducing a detection score associated with a negative sample, the negative sample including a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response, the reference response is not a response corresponding to the reference prompt generated by the first language model, and the reference detection score indicates that the reference response is not generated by the first language model.

[0083] In an embodiment of the present disclosure, the objective function includes reducing the difference between the response generated by the first language model and the response generated by the updated first language model.

[0084] In an embodiment of the present disclosure, the first language model is acquired based on a reinforcement learning process, and updating the first detector model includes: updating the first detector model together with a reward model corresponding to the first language model during the reinforcement learning process.

[0085] In an embodiment of the present disclosure, method 700 further includes: determining, by a first detector model, a first detection score of the first response; and in response to determining that the first detection score satisfies a predetermined condition associated with the first detector model, determining that the first response is generated by the first language model.

[0086] In an embodiment of the present disclosure, method 700 also includes: obtaining a second response generated by a second language model corresponding to a second prompt word; determining a second detection score of the second response by the first detector model; and in response to determining that the second detection score satisfies a predetermined condition, providing an indication that the second language model is the same as the first language model.

[0087] In an embodiment of the present disclosure, method 700 further includes: in response to determining that the second detection score does not satisfy a predetermined condition, providing an indication that the second language model is different from the first language model.

[0088] According to an embodiment of the present disclosure, a computer program product includes a computer-readable storage medium having program instructions embodied therein, and the program instructions can be executed by an electronic device to enable the electronic device to perform method 700.

[0089] Figure 8 800 in which various embodiments of the present disclosure may be implemented. Figure 8 The computing device 800 shown in the figure is for illustration purposes only and does not in any way limit the functionality and scope of the present disclosure. The computing device 800 can be used to implement the above-mentioned methods in the embodiments of the present disclosure. Figure 8 As shown, computing device 800 may be a general computing device and may include at least one or more processors or processing units 810 , a memory 820 , a storage unit 830 , one or more communication units 840 , one or more input devices 850 , and one or more output devices 860 .

[0090] Processing unit 810 may be a physical or virtual processor and may implement various processes based on programs stored in memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 800. Processing unit 810 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0091] The computing device 800 typically includes various computer storage media. Such media can be any media that can be accessed by the computing device 800, including but not limited to volatile and non-volatile media, or removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory), or any combination thereof. The storage unit 830 can be any removable or non-removable medium, and can include a machine-readable medium, such as a memory, a flash drive, a disk, or other medium, which can be used to store information and / or data and can be accessed in the computing device 800.

[0092] The computing device 800 may also include additional removable / non-removable, volatile / non-volatile storage media. Figure 8Not shown in the figure, but a disk drive for reading from and / or writing to a removable non-volatile disk and an optical drive for reading from and / or writing to a removable non-volatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0093] The communication unit 840 communicates with other computing devices via a communication medium. In addition, the functions of the components in the computing device 800 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 800 can operate in a networked environment using logical connections with one or more other servers, networked personal computers (PCs), or other general network nodes.

[0094] The input device 850 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. The output device 860 may be one or more of various output devices, such as a display, a speaker, a printer, etc. With the aid of the communication unit 840, the computing device 800 may also communicate with one or more external devices (not shown), such as a storage device and a display device, in which one or more devices enable a user to interact with the computing device 800, or if necessary, any device (such as a network card, a modem, etc.) enables the computing device 800 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0095] In some embodiments, some or all components of the computing device 800 may also be arranged in a cloud computing architecture, rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using a suitable protocol. For example, a cloud computing provider provides an application via a wide area network, which can be accessed by a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a server at a remote location. The computing resources in a cloud computing environment may be merged or distributed at a location in a remote data center. Cloud computing infrastructure may provide services through a shared data center, although they act as a single access point for users. Therefore, cloud computing architecture may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided from a traditional server or installed directly or otherwise installed on a client device.

[0096] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0097] The program code for executing the method for the subject matter described herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed in whole or in part on the machine, in part on the machine as an independent software package, in part on a remote machine, or in whole on a remote machine or server.

[0098] In the context of the present disclosure, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] In addition, although operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in order, or performing all the operations shown to achieve the desired result. In certain cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the subject matter described herein, but should be interpreted as descriptions of features that may be specific to a particular embodiment. The specific features described in the context of a separate embodiment may also be implemented in a combination of a single embodiment. On the contrary, the various features described in a single embodiment may also be implemented separately in a plurality of embodiments or implemented in a suitable sub-combination.

[0100] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0101] From the foregoing, it should be understood that specific embodiments of the technology of the present disclosure have been described herein for illustrative purposes, but that various modifications may be made without departing from the scope of the present disclosure. Accordingly, the technology of the present disclosure is not to be limited, except as in the appended claims.

[0102] The embodiments of the subject matter and functional operations described in this disclosure can be implemented in various systems, digital electronic circuits or computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a tangible and non-transitory computer-readable medium, for being executed by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing unit" or "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device can also include code to create an execution environment for the computer program in question, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system or a combination of one or more of them.

[0103] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed for execution on one or more computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0104] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0105] The specification, together with the drawings, are intended to be considered exemplary only, where exemplary means example.As used herein, the use of "or" is intended to include "and / or" unless the context clearly dictates otherwise.

[0106] Although this disclosure contains many specific contents, these specific contents should not be interpreted as limitations on the scope of any disclosure or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular disclosure. Specific features described in this disclosure in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in suitable sub-combinations. . In addition, although features may be described above as functioning in a particular combination and even initially claimed as such, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may directly become a sub-combination or a variant of a sub-combination.

[0107] Similarly, although operations are shown in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in order, or that all of the operations shown be performed, to achieve the desired results. In addition, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments. Only some implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and shown in this disclosure.

Claims

1. A method for generating a response, comprising: receiving a first prompt word to be input into a first language model, the first language model representing an association relationship between the prompt word and a response, the response being detectable by a first detector model corresponding to the first language model, and a detection score of the response by the first detector model indicating that the response is generated by the first language model; as well as A first response corresponding to the first prompt word is generated by the first language model, and the first response can be detected by the first detector model.

2. The method of claim 1 , wherein the first detector model is determined by: Acquire a reference sample set, where the reference sample set includes at least positive samples and negative samples; and Based on the reference sample set, parameters of the first detector model are updated.

3. The method according to claim 2, wherein the first language model and the first detector model are iteratively acquired by: When the parameters of the first language model are fixed, updating the parameters of the first detector model based on the objective function; and When the parameters of the first detector model are fixed, the parameters of the first language model are updated based on the objective function.

4. The method according to claim 3, wherein the objective function comprises: Increasing a detection score associated with the positive sample; The positive sample includes a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response; The reference response is a response generated by the first language model corresponding to the reference cue word, and the reference detection score indicates that the reference response is generated by the first language model.

5. The method according to claim 3, wherein the objective function comprises: reducing a detection score associated with the negative sample; The negative sample includes a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response; The reference response is not a response generated by the first language model corresponding to the reference cue word, and the reference detection score indicates that the reference response is not generated by the first language model.

6. The method of claim 3, wherein the objective function comprises reducing a difference between a response generated by the first language model and a response generated by an updated first language model.

7. The method of claim 3, wherein the first language model is acquired based on a reinforcement learning process, and updating the first detector model comprises: During the reinforcement learning process, the first detector model is updated together with a reward model corresponding to the first language model.

8. The method according to claim 1, further comprising: determining, by the first detector model, a first detection score for the first response; as well as In response to determining that the first detection score satisfies a predetermined condition associated with the first detector model, determining that the first response is generated by the first language model.

9. The method according to claim 8, further comprising: Obtaining a second response generated by a second language model corresponding to a second prompt word; determining, by the first detector model, a second detection score for the second response; as well as In response to determining that the second detection score satisfies the predetermined condition, providing an indication that the second language model is the same as the first language model.

10. The method according to claim 9, further comprising: In response to determining that the second detection score does not satisfy the predetermined condition, providing an indication that the second language model is different from the first language model.

11. An electronic device comprising a computer processor coupled to a computer readable memory unit, the memory unit comprising instructions that when executed by the computer processor implement a method for generating a response, the method comprising: receiving a first prompt word to be input into a first language model, the first language model representing an association relationship between the prompt word and a response, the response being detectable by a first detector model corresponding to the first language model, and a detection score of the response by the first detector model indicating that the response is generated by the first language model; as well as A first response corresponding to the first prompt word is generated by the first language model, and the first response can be detected by the first detector model.

12. The apparatus of claim 11, wherein the first detector model is determined by: Acquire a reference sample set, where the reference sample set includes at least positive samples and negative samples; and Based on the reference sample set, parameters of the first detector model are updated.

13. The apparatus according to claim 12, wherein the first language model and the first detector model are iteratively acquired by: When the parameters of the first language model are fixed, updating the parameters of the first detector model based on the objective function; and When the parameters of the first detector model are fixed, the parameters of the first language model are updated based on the objective function.

14. The apparatus of claim 13, wherein the objective function comprises: Increasing a detection score associated with the positive sample; The positive sample includes a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response; The reference response is a response generated by the first language model corresponding to the reference cue word, and the reference detection score indicates that the reference response is generated by the first language model.

15. The apparatus of claim 13, wherein the objective function comprises: reducing a detection score associated with the negative sample; The negative sample includes a reference prompt word, a reference response to the reference prompt word, and a reference detection score for the reference response; The reference response is not a response generated by the first language model corresponding to the reference cue word, and the reference detection score indicates that the reference response is not generated by the first language model.

16. The apparatus of claim 13, wherein the objective function comprises reducing a difference between the first language model and an updated first language model.

17. The apparatus of claim 13, wherein the first language model is acquired based on a reinforcement learning process, and updating the first detector model comprises: During the reinforcement learning process, the first detector model is updated together with a reward model corresponding to the first language model.

18. The apparatus of claim 11, wherein the method further comprises: determining, by the first detector model, a first detection score for the first response; as well as In response to determining that the first detection score satisfies a predetermined condition associated with the first detector model, determining that the first response is generated by the first language model.

19. The apparatus of claim 18, wherein the method further comprises: Obtaining a second response generated by a second language model corresponding to a second prompt word; determining, by the first detector model, a second detection score for the second response; In response to determining that the second detection score satisfies the predetermined condition, providing an indication that the second language model is the same as the first language model; as well as In response to determining that the second detection score does not satisfy the predetermined condition, providing an indication that the second language model is different from the first language model.

20. A non-transitory computer program product, the non-transitory computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method for generating a response, the method comprising: receiving a first prompt word to be input into a first language model, the first language model representing an association relationship between the prompt word and a response, the response being detectable by a first detector model corresponding to the first language model, and a detection score of the response by the first detector model indicating that the response is generated by the first language model; as well as A first response corresponding to the first prompt word is generated by the first language model, and the first response can be detected by the first detector model.

Images