A sentence interaction method based on a generative reward model and a large language model, an electronic device, and a computer-readable storage medium

Through the generative return model and Monte Carlo tree search method, the problem of large language models requiring fine-tuning parameters in new environments is solved, unsupervised self-learning and purposeful text generation is realized, reducing costs and improving the generated text Purpose.

CN118485083BActive Publication Date: 2024-11-15HARBIN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410663865.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2024-11-15
Estimated Expiration
2044-05-27

AI Technical Summary

Technical Problem

Large language models need to supervise fine-tune parameters when applied to new environments, making it difficult to generate text purposefully, and need to be readjusted every time.

Method used

Using the generative return model and Monte Carlo tree search method, candidate sentences are generated through large language models and their action value functions are calculated. The sentence with the highest action value function is selected as the output to achieve unsupervised self-learning and purposeful text generation.

Benefits of technology

Reduces the cost of running the model, allows the language model to learn itself in a new environment, and generates sentences that are more purposeful and suitable for any agent task.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118485083B_ABST
    Figure CN118485083B_ABST
Patent Text Reader

Abstract

The present invention proposes a sentence interaction method, electronic device and computer-readable storage medium based on a generative reward model and a large language model, and belongs to the technical field of sentence interaction. It includes: step one, using the large language model LLM1 to generate k candidate sentences and construct a generative reward model, and the method of constructing the generative reward model is: using the generative model to fit the probability distribution of a certain behavior in a specific environment; taking the logarithm of the probability distribution of the behavior at each moment and then centering it to obtain the reward corresponding to the behavior. Step two, using the large language model LLM2 as a virtual opponent, performing a Monte Carlo tree search on each candidate sentence to obtain the action value function; step three, outputting the candidate sentence with the largest action value function as the result. The present invention allows LLM to self-learn in a new environment without the need for supervised fine-tuning; it can also make the sentences generated by LLM more purposeful.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a sentence interaction method, electronic device and computer-readable storage medium based on a generative reward model and a large language model, and belongs to the technical field of sentence interaction. Background Art

[0002] The Large Language Model (LLM) uses Transformer as a framework, and optimizes model parameters through three steps: massive corpus pre-training, instruction fine-tuning, and human feedback reinforcement learning, so as to achieve text generation tasks under various tasks. The reward model based on the generative model (generative reward model) uses the probability of a certain behavior considered by the generative model to calculate the reward brought by the behavior. Monte Carlo tree search is a method used by AlphaGo to estimate the action value function. It samples the behavior trajectories multiple times in the current environment, and uses the average of the discounted return expectations of different trajectories as the estimated value of the action value function at the current moment.

[0003] Large language models require supervised fine-tuning of parameters each time they are applied to a new environment to adapt to the new environment, and it is difficult to generate text purposefully. Summary of the invention

[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0005] In view of this, in order to solve the technical problems existing in the prior art, the present invention provides a sentence interaction method, an electronic device and a computer-readable storage medium based on a generative reward model and a large language model. The present invention uses a language model as a generative reward model to obtain a reward, and then uses a Monte Carlo tree search to obtain the action value function of each sentence, and then the sentence with the highest action value function can be selected as the output.

[0006] Solution 1: A sentence interaction method based on a generative reward model and a large language model, comprising:

[0007] Step 1: Use the large language model LLM1 to generate k candidate sentences and build a generative reward model;

[0008] Step 2: Using LLM2 as a virtual opponent, perform Monte Carlo tree search on each candidate sentence to obtain the action value function;

[0009] Step 3: Output the candidate sentence with the largest action value function as the result.

[0010] Preferably, the method for constructing a generative reward model is: using a generative model to fit the probability distribution of a certain behavior in a specific environment; after centering the probability distribution of the behavior at each moment, taking the logarithm, and obtaining the reward corresponding to the behavior.

[0011] Taking the logarithm of the probability distribution of each behavior at each moment and centralizing it, the method to obtain the corresponding reward method for the behavior is:

[0012] r=log(π(a))-E π(a) [log(π(a))]

[0013] Among them, π(a) represents the behavior probability generated by the generative model, E π(a) [·] denotes the mathematical expectation operator, and log denotes the logarithmic function.

[0014] Preferably, step 2 is specifically as follows:

[0015] Step 21: Use LLM1 to calculate the probability q(a) of candidate sentence a;

[0016] Step 22: LLM1 calculates the reward r corresponding to the candidate sentence a:

[0017] Step 2: Input the candidate sentence a into the LLM2 model to obtain the answer S corresponding to the candidate sentence;

[0018] Step 24: Based on the answer S corresponding to the candidate sentence, obtain the new candidate sentence a' generated by LLM1 at the next moment, and replace the candidate sentence a with the new candidate sentence a';

[0019] Step 25: Repeat steps 21 to 24, and record the reward r corresponding to the candidate sentence a calculated by step 22 in the i-th cycle as r i , loop N times, get N r i ;

[0020] Step 26: Calculate N r i The discounted cumulative return u is:

[0021] u=∑ i γ i r i

[0022] Among them, r i is the reward corresponding to the candidate sentence obtained in the i-th cycle, γ i is the discount factor γ raised to the power of i, i = 0, 1, ... N-1;

[0023] Step 27: Repeat steps 21 to 26 to obtain multiple discounted cumulative returns, and use the average of the multiple discounted cumulative returns as the action value function of the candidate sentence.

[0024] Solution 2: An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method described in Solution 1 when executing the computer program.

[0025] Solution three: A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in Solution one when executed by a processor.

[0026] The beneficial effects of the present invention are as follows: The present invention uses a generative model as a reward model, that is, the centralized logarithm of the probability of each action of the generative model obtained by maximum likelihood on the target task is used as the reward value of the action. The algorithm is easy to operate and can explicitly give the relationship between the generative model and the reward function in any field; the reward model is obtained in an unsupervised manner to greatly reduce the cost; and it can be applied to any intelligent agent task. The present invention allows LLM to self-learn in a new environment without supervised fine-tuning; it can also make the sentences generated by LLM more purposeful. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 A flowchart of a sentence interaction method based on a generative reward model and a large language model. DETAILED DESCRIPTION

[0029] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0030] Example 1: Reference Figure 1This embodiment describes a sentence interaction method based on a generative reward model and a large language model. In this paper, a large language model LLM1 is used to generate candidate sentences, and LLM1 is used as the generative reward model to obtain rewards. The action value function value of each candidate sentence is obtained based on the Monte Carlo tree search of another large language model LLM2, and then the candidate sentence with the largest action value function is selected as the final output. LLM1 and LLM2 can be any language model that can generate text and text probability, such as GPT, Mistral, etc. LLM1 is a generative reward model constructed by the present invention. The generative reward model is used to generate candidate sentences and the rewards corresponding to the candidate sentences, which can reduce the running cost of the computing model;

[0031] Specifically include:

[0032] Step 1: Use the large language model LLM1 to generate k candidate sentences and build a generative reward model;

[0033] LLM1 outputs N sentences, calculates the probability of each sentence, and selects the k sentences with the highest sentence probability as candidates;

[0034] The method of constructing a generative reward model is: use the generative model to fit the probability distribution of a certain behavior in a specific environment; take the logarithm of the probability distribution of the behavior at each moment and then center it to obtain the reward corresponding to the behavior.

[0035] Taking the logarithm of the probability distribution of each behavior at each moment and centralizing it, the method to obtain the corresponding reward method for the behavior is:

[0036] r=log(π(a))-E π(a) [log(π(a))]

[0037] Among them, π(a) represents the behavior probability generated by the generative model, E π(a) [·] denotes the mathematical expectation operator, and log denotes the logarithmic function.

[0038] Step 2: Using LLM2 as a virtual opponent, perform a Monte Carlo tree search on each candidate sentence to obtain the action value function; specifically:

[0039] Step 21: Use LLM1 to calculate the probability q(a) of candidate sentence a;

[0040] Step 22: LLM1 calculates the reward r corresponding to the candidate sentence a:

[0041] Step 2: Input the candidate sentence a into the LLM2 model to obtain the answer S corresponding to the candidate sentence;

[0042] Step 24: Based on the answer S corresponding to the candidate sentence, obtain the new candidate sentence a' generated by LLM1 at the next moment, and replace the candidate sentence a with the new candidate sentence a';

[0043] Step 25: Repeat steps 21 to 24, and record the reward r corresponding to the candidate sentence a calculated by step 22 in the i-th cycle as r i , loop N times, get N r i ;

[0044] Step 26: Calculate N r i The discounted cumulative return u is:

[0045] u=∑ i γ i r i

[0046] Among them, r i is the reward corresponding to the candidate sentence obtained in the i-th cycle, γ i is the discount factor γ raised to the power of i, i = 0, 1, ... N-1;

[0047] Step 27: Repeat steps 21 to 26 to obtain multiple discounted cumulative returns, and use the average of the multiple discounted cumulative returns as the action value function of the candidate sentence.

[0048] Step 3: Output the candidate sentence with the largest action value function as the result.

[0049] In this embodiment, LLM1 and LLM2 are two large language models, which are distinguished by LLM1 and LLM2.

[0050] Embodiment 2: The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Furthermore, the processor is used to implement the steps of the method described in Embodiment 1 when executing a computer program stored in the memory.

[0051] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0052] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0053] Embodiment 3: Computer readable storage medium embodiment.

[0054] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the method described in Example 1 can be implemented.

[0055] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0056] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A sentence interaction method based on a generative reward model and a large language model, characterized in that: include: Step 1: Use the large language model LLM1 to generate k candidate sentences and build a generative reward model; Step 2: Using the large language model LLM2 as a virtual opponent, perform a Monte Carlo tree search on each candidate sentence to obtain the action value function; Step 3: Output the candidate sentence with the largest action value function as the result; The method of constructing a generative reward model is: use the generative model to fit the probability distribution of a certain behavior in a specific environment; take the logarithm of the probability distribution of the behavior at each moment and then center it to get the reward corresponding to the behavior; Taking the logarithm of the probability distribution of each behavior at each moment and centralizing it, the method to obtain the corresponding reward method for the behavior is: r(log(π(a))-E π(a) [log(π(a))] Among them, π(a) represents the behavior probability generated by the generative model, E π(a) [·] represents the mathematical expectation operator, log represents the logarithmic function; Step 2 is as follows: Step 21: Use LLM1 to calculate the probability q(a) of candidate sentence a; Step 22: LLM1 calculates the reward r corresponding to the candidate sentence a: Step 2: Input the candidate sentence a into the LLM2 model to obtain the answer S corresponding to the candidate sentence; Step 24: Based on the answer S corresponding to the candidate sentence, obtain the new candidate sentence a' generated by LLM1 at the next moment, and replace the candidate sentence a with the new candidate sentence a'; Step 25: Repeat steps 21 to 24, and record the reward r corresponding to the candidate sentence a calculated by step 22 in the i-th cycle as r i , loop N times, get N r i ; Step 26: Calculate N r i The discounted cumulative return u is: u=∑ i c i r i Among them, r i is the reward corresponding to the candidate sentence obtained in the i-th cycle, γ i is the discount factor γ raised to the power of i, i = 0, 1, ... N-1; Step 27: Repeat steps 21 to 26 to obtain multiple discounted cumulative returns, and use the average of the multiple discounted cumulative returns as the action value function of the candidate sentence.

2. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to claim 1 when executing the computer program.

3. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to claim 1 is implemented.