A method for constructing a reward model based on a generative model and its verification method

Abstract

The present invention proposes a method for constructing a reward model based on a generative model and a verification method thereof, which belongs to the technical field of reward model construction. The method includes the following steps: step 1, fitting the probability distribution of behavior in a specific environment; step 2, taking the logarithm of the probability distribution of the behavior at each moment and centralizing it to obtain the reward corresponding to the behavior; step 3, constructing an intelligent agent strategy based on the reward corresponding to the behavior to realize the application of the reward model. The present invention uses a generative model as a reward model, that is, the centralized logarithmic value of the probability of each action of the generative model obtained by maximum likelihood on the target task is used as the reward value of the action. The algorithm is easy to operate and can explicitly give the relationship between the generative model and the reward function in any field; the use of an unsupervised method to obtain a reward model greatly reduces the cost; and it can be applied to any intelligent agent task. It solves the technical problems of cumbersome steps and complex calculations when constructing a reward model.

Description

Technical Field

[0001] The present invention relates to a method for constructing a reward model based on a generative model and a verification method thereof, and belongs to the technical field of reward model construction. Background Art

[0002] Maximum Entropy Inverse Reinforcement Learning: The reward function of inverse reinforcement learning is difficult to specify. The reward function can be easily designed through the design method based on maximum entropy. This algorithm uses the maximum entropy principle to restore the probability distribution that satisfies the constraints, transforms the problem of designing the reward function into a standard optimization problem, and obtains the probability of having the maximum entropy. Finally, the parameter term of the reward function in the probability is solved to obtain the reward function.

[0003] The Direct Preference Optimization (DPO) algorithm derives the implicit relationship between the probability of each sentence of a large language model (LLM) and the corresponding reward. Substituting this relationship into Reinforcement Learning from Human Feedback (RLHF) eliminates the reward model, and then the LLM can be fine-tuned directly based on the preference dataset without training the reward model.

[0004] Traditional reinforcement learning has the problem of difficulty in constructing a reward model; existing technologies include the design method based on maximum entropy, the design method based on relative entropy, the MMP method, and apprenticeship learning. The problems with these methods are that the steps are cumbersome, the calculations are complex, and sometimes unknown partition function terms are encountered, which cannot be solved directly; the DPO algorithm proves the relationship between the LLM and the reward model, but does not give an explicit relationship, and assumes that the discount factor is 0, so it can only be used for fine-tuning the LLM. Summary of the invention

[0005] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0006] In view of this, in order to solve the technical problems of complicated steps and complex calculations in the construction of the reward model in the prior art, the present invention provides a method for constructing a reward model based on a generative model and a verification method thereof.

[0007] Solution 1: A method for constructing a reward model based on a generative model, including

[0008] Fitting probability distributions of behaviors in a particular environment;

[0009] Take the logarithm of the probability distribution of the behavior at each moment and then center it to get the reward corresponding to the behavior.

[0010] Preferably, a generative model is used to fit the probability distribution of a certain behavior in a specific environment.

[0011] Preferably, the probability distribution of the behavior at each moment is logarithmically taken and then centered to obtain the reward corresponding to the behavior;

[0012] r=log(π(a))-E π(a) [log(π(a))]

[0013] Among them, π(a) represents the behavior probability generated by the generative model, E π(a) [·] denotes the mathematical expectation operator, and log denotes the logarithmic function.

[0014] Preferably, the method of constructing an agent strategy based on the reward corresponding to the behavior is to apply the reward corresponding to the behavior to the downstream task of any reward function.

[0015] Solution 2: A verification method for the reward model based on the generative model, which adds the information entropy regularization term to the reinforcement learning strategy objective function to obtain the optimal strategy π * The objective function is:

[0016]

[0017] Among them, π is the strategy, E π is the expectation of the strategy, Q * (a, s) is the optimal action-value function in reinforcement learning, and α is the Lagrange multiplier;

[0018] Apply the variational method and Lagrange multiplier method to solve the objective function and obtain the optimal strategy π * The analytical solution is:

[0019] Construct a Lagrangian function with Lagrangian multiplier μ for the objective function:

[0020] L(π,μ)=∑ a aπ(a)Q * (a, s)-∑ a π(a)log(π(a))+μ(∑ a π(a)-1)

[0021] Let the gradient of the Lagrangian function with respect to π(a) and μ be 0, and we get:

[0022] π * =σ(Q * (a,s)+K s )=σ(A * (a,s)+V * (s)+K s )=σ(A * (a,s)+C s )

[0023] Among them, σ is the softmax function, A * (a, s) is the advantage function in reinforcement learning, K s and C s is a constant, μ is the Lagrange multiplier, and π(a) is the strategy;

[0024] After deducing the advantage function as a reward, it is proved that:

[0025]

[0026] because:

[0027]

[0028] therefore:

[0029]

[0030] Among them, π is the strategy we want to optimize, π′ is an arbitrary strategy, and Q π and Q π′ is the action value function of policy π and policy π′, γ i is the discount factor γ raised to the power of i, where i is time, r i is the return at time i, A π and A π′ is the advantage function of strategy π and strategy π′, a i ,s i It is the action and environment at time i;

[0031] It can be seen that A π' Replace r i The objective functions before and after are equivalent, and we finally get:

[0032] r=log(π(a))-E π(a) [log(π(a))].

[0033] Solution three: An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method described in Solution one when executing the computer program.

[0034] Solution 4: A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in Solution 1 when executed by a processor.

[0035] The beneficial effects of the present invention are as follows: the present invention uses a generative model as a reward model, that is, the centralized logarithm of the probability of each action of the generative model obtained by maximum likelihood on the target task is used as the reward value of the action. The algorithm is easy to operate and can explicitly give the relationship between the generative model and the reward function in any field; the reward model is obtained in an unsupervised manner to greatly reduce the cost; and it can be applied to any intelligent agent task. It solves the technical problems of cumbersome steps and complex calculations in the construction of the reward model in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0037] Figure 1 A flowchart of a method for constructing a reward model based on a generative model. DETAILED DESCRIPTION

[0038] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0039] Example 1: Reference Figure 1 To illustrate this embodiment, a method for constructing a reward model based on a generative model includes the following steps:

[0040] Step 1: Fit the probability distribution of behavior in a specific environment;

[0041] Specifically, a generative model is used to fit the probability distribution of a behavior in a specific environment;

[0042] Monitor and record the behavior trajectory of the observed person; perform maximum likelihood fitting on the trajectory, use the text component in the observed person's trajectory as a corpus to train the GPT, or use the image component in the observed person's trajectory to train the Diffusion model. The specific process is as follows:

[0043] The generative model is GPT: the text from the non-observer in the trajectory is taken as the environment s, and the text from the observer is taken as the behavior a; input s to GPT, and the output of GPT is Calculate a and The cross entropy loss l is used to optimize the parameters of GPT and minimize l using algorithms such as Adam or SGD.

[0044] The generative model is Diffusion: the image or text from the non-observer in the trajectory is taken as the environment s, and the image from the observer is taken as the behavior a; s is input to Diffusion, and the output of Diffusion is Calculate a and L1 loss l; use algorithms such as Adam or SGD to optimize the Diffusion parameters and minimize l;

[0045] Step 2: After centering the probability distribution of the behavior at each moment, take the logarithm to obtain the corresponding reward of the behavior;

[0046] r=log(π(a))-E π(a) [log(π(a))]

[0047] Among them, π(a) is the behavior probability considered by the generative model, E π(a) [·] is the mathematical expectation operator, and log is the logarithmic function.

[0048] This embodiment also includes: Step 3, constructing an intelligent agent strategy based on the rewards corresponding to the behaviors to realize the application of the reward model.

[0049] Specifically, the reward corresponding to the behavior is applied to any downstream task that requires a reward function. The specific method is the reinforcement learning algorithm selected by the downstream task, which may be Q-learning, DQN, AC, PPO or TRPO, etc.

[0050] If DQN is used in the downstream task, the process of the present invention is as follows:

[0051] Use a generative model G to fit the behavior probability π(a) of other agents in the task; instantiate a neural network as DQN; input the current environment s to DQN and output the Q value of each behavior; select the behavior a with the largest Q value as the decision, and record the Q value of a as q1; use G to calculate π(a) and get the reward r of the behavior:

[0052] r=log(π(a))-E π(a) [log(π(a))]

[0053] Observe the environment s' at the next moment, input s' at the next moment into DQN, and output the maximum Q value q2 at the next moment; use r+γ·q2 as the label, calculate the error between the label and q1, and use the SGD algorithm to optimize DQN to minimize the error.

[0054] The downstream task uses the AC algorithm, and the process of using the present invention is:

[0055] The overall process of applying the present invention to Q-learning is to obtain a trained DQN network, and to instantiate a neural network M as a policy model; M is optimized by using the policy gradient algorithm and the DQN obtained in 1;

[0056] Embodiment 2: A verification method of a reward model based on a generative model, comprising:

[0057] The information entropy regularization term is added to the reinforcement learning strategy objective function to obtain the optimal strategy π * The objective function is:

[0058]

[0059] Among them, π is the strategy, E π is the expectation of the strategy, Q * (a, s) is the optimal action-value function in reinforcement learning, and α is the Lagrange multiplier;

[0060] Apply the variational method and Lagrange multiplier method to solve the objective function and obtain the optimal strategy π * The analytical solution is:

[0061] Construct a Lagrangian function with Lagrangian multiplier μ for the objective function:

[0062] L(π,μ)=∑ a π(a)Q * (a, s)-∑ a π(a)log(π(a))+μ(∑ a π(a)-1)

[0063] Let the gradient of the Lagrangian function with respect to π(a) and μ be 0, and we get:

[0064] π * =σ(Q * (a,s)+K s )=σ(A * (a,s)+V * (s)+K s )=σ(A * (a,s)+C s )

[0065] Among them, σ is the softmax function, A * (a, s) is the advantage function in reinforcement learning, K s and C s is a constant, μ is the Lagrange multiplier, and π(a) is the strategy;

[0066] After deducing the advantage function as a reward, it is proved that:

[0067]

[0068] because:

[0069]

[0070] therefore:

[0071]

[0072] Among them, π is the strategy we want to optimize, π′ is an arbitrary strategy, and Q π and Q π′ is the action value function of policy π and policy π′, γ i is the discount factor γ raised to the power of i, where i is time, r i is the return at time i, A π and A π′ is the advantage function of strategy π and strategy π′, a i ,s i It is the action and environment at time i;

[0073] It can be seen that A π′ Replace r i The objective functions before and after are equivalent, and we finally get:

[0074] r=log(π(a))-E π(a) [log(π(a))].

[0075] Embodiment 3: The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Furthermore, the processor is used to implement the steps of the method described in Embodiment 1 when executing the computer program stored in the memory.

[0076] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0077] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0078] Embodiment 4: Computer readable storage medium embodiment.

[0079] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the method described in Example 1 can be implemented.

[0080] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0081] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for constructing a reward model based on a generative model, characterized in that: include: Use generative models to fit the probability distribution of a behavior in a specific environment; Take the logarithm of the probability distribution of the behavior at each moment and then center it to get the reward corresponding to the behavior; Use generative models to fit the probability distribution of a behavior in a specific environment; Monitor and record the behavior trajectory of the observed person; perform maximum likelihood fitting on the trajectory, use the text component in the observed person's trajectory as a corpus to train the GPT, or use the image component in the observed person's trajectory to train the Diffusion model; The generative model is GPT: the text from the non-observer in the trajectory is taken as the environment s, and the text from the observer is taken as the behavior a; input s to GPT, and the output of GPT is Calculate a and The cross entropy loss l is used to optimize the parameters of GPT and minimize l. The generative model is Diffusion: the image or text from the non-observer in the trajectory is taken as the environment s, and the image from the observer is taken as the behavior a; s is input to Diffusion, and the output of Diffusion is Calculate a and L1 loss l; use Adam or SGD algorithm to optimize Diffusion parameters and minimize l; Take the logarithm of the probability distribution of the behavior at each moment and then center it to get the reward corresponding to the behavior; Among them, π(a) represents the behavior probability generated by the generative model, represents the mathematical expectation operator, and log represents the logarithmic function.

2. The method for constructing a reward model based on a generative model according to claim 1, characterized in that: The approach to building an agent's policy based on the rewards corresponding to actions is to apply the rewards corresponding to actions to tasks downstream of any reward function.

3. A verification method for a reward model based on a generative model, implemented by relying on a method for constructing a reward model based on a generative model as described in any one of claims 1-2, characterized in that: include: The information entropy regularization term is added to the reinforcement learning strategy objective function to obtain the optimal strategy π * The objective function is: Among them, π is the strategy, E π is the expected value of the strategy, Q * (a, s) is the optimal action-value function in reinforcement learning, and α is the Lagrange multiplier; Apply the variational method and Lagrange multiplier method to solve the objective function and obtain the optimal strategy π * The analytical solution is: Construct a Lagrangian function with Lagrangian multiplier μ for the objective function: L(π,μ)=∑ a π(a)Q * (a,s)-∑ a π(a)log(π(a))+μ(∑ a π(a)-1) Let the gradient of the Lagrangian function with respect to π(a) and μ be 0, and we get: p * =σ(Q * (a,s)+K s )=σ(A * (a,s)+V * (s)+K s )=σ(A * (a,s)+C s ) Among them, σ is the softmax function, A * (a, s) is the advantage function in reinforcement learning, K s and C s is a constant, μ is the Lagrange multiplier, and π(a) is the strategy; After deducing the advantage function as a reward, it is proved that: because: therefore: Among them, π is the strategy we want to optimize, π' is an arbitrary strategy, and Q π and Q π' is the action value function of policy π and policy π', γ i is the discount factor γ raised to the power of i, where i is time, r i is the return at time i, A π and A π' is the advantage function of strategy π and strategy π', a i ,s i It is the action and environment at time i; It can be seen that A π' Replace r i The objective functions before and after are equivalent, and we finally get: r(log(π(a))-E π(a) [log(π(a))] 4. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 2 when executing the computer program.

5. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images