Individualized large language model Prompt generation method for reinforcement learning supervision

Abstract

The invention provides a reinforcement learning supervision personalized large language model Prompt generation method. The method comprises the following steps: respectively inputting sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model to obtain a first generation Prompt and a second generation Prompt; inputting the first generated Prompt and the corresponding first generated reply into a reward model to obtain a reward value corresponding to the first generated Prompt; and updating parameters of the initial Prompt generation model based on the sample data, the Prompt tag, the first generation Prompt, the second generation Prompt and the reward value corresponding to the first generation Prompt to generate a large language model Prompt. The Prompt generation model is trained based on the first task description sample and the corresponding first user information sample, and personalized replies are provided for the user.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method for generating a personalized large language model Prompt under reinforcement learning supervision. Background Art

[0002] The Large Language Model (LLM) is an artificial intelligence model designed to understand and generate human language. The LLM relies on prompts to guide it to generate content within a specified range.

[0003] However, the existing prompt generation method can only enable the large language model LLM to generate general responses, and cannot provide personalized responses for users, which affects the response effect of the large language model. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a personalized large language model Prompt generation method supervised by reinforcement learning, which can improve the generation effect of Prompt, and then provide users with personalized replies and improve the reply effect of the large language model.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a personalized large language model Prompt with reinforcement learning supervision, the method comprising:

[0006] Acquire sample data and corresponding Prompt tags; the sample data includes a first task description sample and a corresponding first user information sample;

[0007] Input the sample data into the initial Prompt generation model and the pre-trained intermediate Prompt generation model respectively to obtain the first generated Prompt and the second generated Prompt;

[0008] The first generated prompt is input into the large language model to obtain a first generated reply; the first generated prompt and the first generated reply are input into the reward model to obtain a reward value corresponding to the first generated prompt; the reward model is trained based on sample data and reward value labels obtained from the second task description sample and the corresponding second user information sample;

[0009] Based on the sample data, Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt, the parameters of the initial Prompt generation model are updated until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

[0010] In a possible implementation, the objective function for updating the parameters of the initial Prompt generation model to obtain the target Prompt generation model is as follows:

[0011]

[0012] in, is the parameter of the initial Prompt generation model, E is the expected value, x is the sample data, is the prompt label, D is a dataset containing sample data and corresponding prompt labels, model_reward is the reward model, x′ is the first generated prompt, model_reward(x ′ ) is the reward value corresponding to the first generated Prompt, β is a constant, Generate the model for the initial prompt in the model parameters Given the sample data, the probability of getting the first generated prompt is, y is the first generated prompt, model SFT (x|y) is the probability of obtaining the second generated prompt given the sample data by the intermediate prompt generation model, Prompt for the second generation.

[0013] In one possible implementation, the reward model is trained by the following steps:

[0014] Obtain a first prompt sample and a corresponding first reply sample based on the second user information sample and the second task description sample, a second prompt sample and a corresponding second reply sample based on the second task description sample, and a first reward value label corresponding to the first prompt sample and the first reply sample, and a second reward value label corresponding to the second prompt sample and the second reply sample; the value corresponding to the first reward value label is greater than the value corresponding to the second reward value label;

[0015] The reward model is trained through the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first reward value label, and the second reward value label.

[0016] In a possible implementation, training the reward model through the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first reward value label, and the second reward value label includes:

[0017] Input the first prompt sample and the corresponding first reply sample into the reward model to obtain a first predicted reward value corresponding to the first prompt sample;

[0018] Input the second Prompt sample and the corresponding second reply sample into the reward model to obtain a second predicted reward value corresponding to the second Prompt sample;

[0019] Update the parameters of the reward model with the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first predicted reward value, the second predicted reward value, the first reward value label, and the second reward value label until the preset expected value corresponding to the reward model is met.

[0020] In a possible implementation, both the initial Prompt generation model and the intermediate Prompt generation model include a user information feature extraction unit and a Prompt generation unit; the sample data is respectively input into the initial Prompt generation model and the pre-trained intermediate Prompt generation model to obtain a first generated Prompt and a second generated Prompt, including:

[0021] Inputting the first user information sample into the user information feature extraction unit in the initial Prompt generation model and the user information feature extraction unit in the intermediate Prompt generation model to obtain the first user information feature and the second user information feature;

[0022] Inputting the first task description sample and the first user information feature into the Prompt generation unit in the initial Prompt generation model to obtain a first generated Prompt;

[0023] The first task description sample and the second user information feature are input into the Prompt generation unit in the intermediate Prompt generation model to obtain a second generated Prompt.

[0024] In a possible implementation, the intermediate prompt generation model is trained through the following steps:

[0025] Acquire a third task description sample and a corresponding third user information sample, and a Prompt tag corresponding to the third task description sample and the third user information sample;

[0026] Inputting the third task description sample and the third user information sample into the initial Prompt generation model to obtain a third generated Prompt;

[0027] The initial Prompt generation model is trained based on the third task description sample, the third user information sample, the third generated Prompt, and the Prompt label corresponding to the third task description sample and the third user information sample to obtain an intermediate Prompt generation model.

[0028] In a possible implementation, the method further includes:

[0029] Obtain the task description to be generated and the corresponding user information to be generated;

[0030] The task description to be generated and the user information to be generated are input into the target prompt generation model to obtain the target prompt.

[0031] In a second aspect, the embodiment of the present application further provides a personalized large language model Prompt generation device for reinforcement learning supervision, the device comprising:

[0032] An acquisition module is used to acquire sample data and a corresponding Prompt tag; the sample data includes a first task description sample and a corresponding first user information sample;

[0033] An input module, used to input sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model, respectively, to obtain a first generated Prompt and a second generated Prompt;

[0034] The input module is further used to input the first generated prompt into the large language model to obtain a first generated reply; input the first generated prompt and the first generated reply into the reward model to obtain a reward value corresponding to the first generated prompt; the reward model is trained based on sample data and reward value labels obtained from the second task description sample and the corresponding second user information sample;

[0035] An updating module is used to update the parameters of the initial Prompt generation model based on sample data, Prompt labels, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt, until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the personalized large language model Prompt generation method supervised by reinforcement learning as described in any one of the first aspects.

[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the personalized large language model Prompt generation method supervised by reinforcement learning as described in any one of the first aspects are executed.

[0038] An embodiment of the present application provides a personalized large language model Prompt generation method supervised by reinforcement learning, the method comprising: obtaining sample data and a corresponding Prompt label; the sample data comprises a first task description sample and a corresponding first user information sample; the sample data is respectively input into an initial Prompt generation model and a pre-trained intermediate Prompt generation model to obtain a first generated Prompt and a second generated Prompt; the first generated Prompt is input into a large language model to obtain a first generated reply; the first generated Prompt and the first generated reply are input into a reward model to obtain a reward value corresponding to the first generated Prompt; the reward model is trained based on the sample data and reward value label obtained from the second task description sample and the corresponding second user information sample; the parameters of the initial Prompt generation model are updated based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt. This application obtains a Prompt generation model based on the first task description sample and the corresponding first user information sample training, which can improve the generation effect of Prompt, and then provide personalized replies to users, thereby improving the reply effect of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flowchart of a method for generating a personalized large language model Prompt with reinforcement learning supervision provided in an embodiment of the present application is shown;

[0041] Figure 2 A flowchart of another method for generating a personalized large language model Prompt with reinforcement learning supervision provided in an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram of generating a target prompt provided in an embodiment of the present application is shown;

[0043] Figure 4 A schematic diagram of the structure of a personalized large language model Prompt generation device for reinforcement learning supervision provided in an embodiment of the present application is shown;

[0044] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0046] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0047] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario, the "field of large language model technology". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "field of large language model technology", it should be understood that this is only an exemplary embodiment.

[0048] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0049] The following is a detailed description of a method for generating a personalized large language model Prompt with reinforcement learning supervision provided in an embodiment of the present application.

[0050] Reference Figure 1 As shown, it is a flowchart of a method for generating a personalized large language model Prompt with reinforcement learning supervision provided in an embodiment of the present application. The specific execution process of the method is as follows:

[0051] S101, obtaining sample data and a corresponding Prompt tag; the sample data includes a first task description sample and a corresponding first user information sample.

[0052] In the implementation manner of the present application, the first task description sample refers to the description information of the task that the large language model needs to perform. The first user information sample refers to the characteristic information of the user who formulates the first task description sample, which may include age, gender, occupation, personality, etc. (depending on the actual situation). The Prompt tag refers to the prompt information written based on the first task description sample and the first user information sample, which is used to guide the large language model to generate a reply; the specific writing method of the Prompt tag can be manually written.

[0053] S102, inputting the sample data into the initial Prompt generation model and the pre-trained intermediate Prompt generation model respectively, to obtain the first generated Prompt and the second generated Prompt.

[0054] In the implementation manner of the present application, the sample data is input into the initial Prompt generation model to obtain the first generated Prompt; the sample data is input into the intermediate Prompt generation model to obtain the second generated Prompt. The intermediate Prompt generation model is trained based on supervised fine-tuning.

[0055] Among them, both the initial Prompt generation model and the intermediate Prompt generation model include a user information feature extraction unit and a Prompt generation unit; the user information feature extraction unit is used to extract user information features, and the Prompt generation unit is used to generate a Prompt based on the user information features and the task description.

[0056] Among them, the user information feature extraction unit is composed of a feature extraction layer and an output layer. The feature extraction layer can be a common feature extractor of NLP such as RNN, LSTM, GRU and Transformer. The output layer is a Dense layer whose output value is a 1*D vector, and D is the dimension of the hidden layer of the feature extraction layer. The user information feature extraction model represents the input user information as a 1*D vector V1. The prompt generation model first converts the input task description into an L*D matrix V2 through the Embedding layer (L is the input sequence length, and D is the dimension represented by the Embedding vector). After the vector V1 and the matrix V2 are added, prompts are output through multiple layers of transformer-decoder or transformer-encoder-decoder.

[0057] Specifically, the sample data is input into the initial Prompt generation model and the intermediate Prompt generation model to obtain the first generated Prompt and the second generated Prompt, including:

[0058] Step 1: Input the first user information sample into the user information feature extraction unit in the initial Prompt generation model and the user information feature extraction unit in the intermediate Prompt generation model to obtain the first user information feature and the second user information feature.

[0059] In an implementation manner of the present application, the first user information sample is input into the user information feature extraction unit in the initial Prompt generation model to obtain the first user information feature; the first user information sample is input into the user information feature extraction unit in the intermediate Prompt generation model to obtain the second user information feature.

[0060] Step 2: Input the first task description sample and the first user information feature into the Prompt generation unit in the initial Prompt generation model to obtain a first generated Prompt.

[0061] Step 3: Input the first task description sample and the second user information feature into the Prompt generation unit in the intermediate Prompt generation model to obtain a second generated Prompt.

[0062] Furthermore, the intermediate prompt generation model is obtained by training through the following steps:

[0063] Step 1: Obtain a third task description sample and a corresponding third user information sample, as well as a Prompt tag corresponding to the third task description sample and the third user information sample.

[0064] In the implementation manner of the present application, the third task description sample refers to the description information of the task that the large language model needs to perform. The third user information sample refers to the characteristic information of the user who formulates the third task description sample, which may include age, gender, occupation, personality, etc. (depending on the actual situation). The Prompt tag corresponding to the third task description sample and the third user information sample refers to the prompt information written based on the third task description sample and the third user information sample, which is used to guide the large language model to generate a reply; the specific writing method of the Prompt tag can be manually written.

[0065] Step 2: Input the third task description sample and the third user information sample into the initial Prompt generation model to obtain a third generated Prompt.

[0066] In an implementation manner of the present application, the third task description sample is input into the user information feature extraction unit in the initial Prompt generation model to obtain the third user information feature; the third task description sample and the third user information feature are input into the Prompt generation unit in the initial Prompt generation model to obtain the third generated Prompt.

[0067] Step three: update the parameters of the initial Prompt generation model based on the third task description sample, the third user information sample, the third generated Prompt, and the Prompt label corresponding to the third task description sample and the third user information sample until the preset expected value of the intermediate Prompt generation model is met to obtain the intermediate Prompt generation model.

[0068] Specifically, the objective function used to update the parameters of the initial Prompt generation model to obtain the intermediate Prompt generation model is as follows:

[0069]

[0070] in, is the objective function used to update the parameters of the initial Prompt generation model to obtain the intermediate Prompt generation model, is the parameter of the initial Prompt generation model, E is the expected value, x ′ is third sample data consisting of a third task description sample and a third user information sample, is the prompt label corresponding to the third task description sample and the third user information sample, D′ is a data set containing the third sample data and the corresponding prompt label, model(x′|y′) is the initial prompt generation model in the model parameters The probability of obtaining the third generated Prompt given the third sample data is y′, which is the third generated Prompt.

[0071] S103, input the first generated Prompt into the large language model to obtain a first generated reply; input the first generated Prompt and the first generated reply into the reward model to obtain a reward value corresponding to the first generated Prompt.

[0072] In an implementation manner of the present application, the reward model is trained based on sample data and reward value labels obtained based on the second task description sample and the corresponding second user information sample.

[0073] Furthermore, the reward model is trained by the following steps:

[0074] Step 1: Obtain a first Prompt sample and a corresponding first reply sample based on the second user information sample and the second task description sample, a second Prompt sample and a corresponding second reply sample based on the second task description sample, and a first reward value label corresponding to the first Prompt sample and the first reply sample, and a second reward value label corresponding to the second Prompt sample and the second reply sample; the numerical value corresponding to the first reward value label is greater than the numerical value corresponding to the second reward value label.

[0075] In the implementation manner of the present application, the second task description sample refers to the description information of the task that the large language model needs to perform. The second user information sample refers to the characteristic information of the user who formulates the second task description sample, which may include age, gender, occupation, personality, etc. (depending on the actual situation). The first Prompt sample refers to the prompt information written based on the second user information sample and the second task description sample, which is used to guide the large language model to generate a reply; the second Prompt sample refers to the prompt information written based on the second task description sample, which is used to guide the large language model to generate a reply. The specific writing method of the first Prompt sample and the second Prompt sample can be manually written.

[0076] In addition, the first reward value label and the second reward value label can be manually labeled; the reward value is used to characterize the accuracy of the reply (first reply sample, second reply sample) generated by the large language model guided by the Prompt (first Prompt sample, second Prompt sample); the larger the reward value, the higher the quality of the Prompt and the higher the accuracy of the large language model.

[0077] Here, the first Prompt sample is input into the large language model to obtain the first reply sample; the second Prompt sample is input into the large language model to obtain the second reply sample.

[0078] Step 2: Train the reward model using the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first reward value label, and the second reward value label.

[0079] In an implementation manner of the present application, a first Prompt sample and a corresponding first reply sample are input into a reward model to obtain a first predicted reward value corresponding to the first Prompt sample; a second Prompt sample and a corresponding second reply sample are input into a reward model to obtain a second predicted reward value corresponding to the second Prompt sample; the first Prompt sample, the first reply sample, the second Prompt sample, the second reply sample, the first predicted reward value, the second predicted reward value, the first reward value label, and the second reward value label are used to update the parameters of the reward model until the preset expected value corresponding to the reward model is met.

[0080] Here, the objective function of the reward model is as follows:

[0081]

[0082] in, is the objective function of the reward model, E is the expected value, x″ is the second sample data consisting of the first prompt sample, the first reply sample, the second prompt sample, and the second reply sample, y w is the first reward value label, y l is the second reward value label, σ is the sigmoid function, is the first predicted reward value of the sample corresponding to the first reward value label in the second sample data, It is the second predicted reward value of the sample corresponding to the second reward value label in the second sample data.

[0083] Here, based on the objective function of the above reward model, it can be known that the update direction of the reward model is to make the reward value corresponding to the prompt information obtained based on the user information and the task description information greater than the reward value corresponding to the prompt information obtained based on the task description information. The larger the corresponding value, the stronger the scoring ability of the reward model.

[0084] S104. Update the parameters of the initial Prompt generation model based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt, and the reward value corresponding to the first generated Prompt until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

[0085] Here, the objective function used to update the parameters of the initial Prompt generation model to obtain the target Prompt generation model is as follows:

[0086]

[0087] in, is the parameter of the initial Prompt generation model, E is the expected value, x is the sample data, is the prompt label, E is a dataset containing sample data and corresponding prompt labels, model_reward is the reward model, x′ is the first generated prompt, model_reward(x′) is the reward value corresponding to the first generated prompt, β is a constant, Generate the model for the initial prompt in the model parameters Given the sample data, the probability of getting the first generated prompt is, y is the first generated prompt, model SFT (x|y) is the probability of obtaining the second generated prompt given the sample data by the intermediate prompt generation model, Prompt for the second generation.

[0088] An embodiment of the present application provides a personalized large language model Prompt generation method supervised by reinforcement learning, the method comprising: obtaining sample data and a corresponding Prompt label; the sample data comprises a first task description sample and a corresponding first user information sample; the sample data is respectively input into an initial Prompt generation model and a pre-trained intermediate Prompt generation model to obtain a first generated Prompt and a second generated Prompt; the first generated Prompt is input into a large language model to obtain a first generated reply; the first generated Prompt and the first generated reply are input into a reward model to obtain a reward value corresponding to the first generated Prompt; the reward model is trained based on the sample data and reward value label obtained from the second task description sample and the corresponding second user information sample; the parameters of the initial Prompt generation model are updated based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt. This application obtains a Prompt generation model based on the first task description sample and the corresponding first user information sample training, which can improve the generation effect of Prompt, and then provide personalized replies to users, thereby improving the reply effect of the large language model.

[0089] Reference Figure 2 As shown, it is a flow chart of another method for generating a personalized large language model Prompt with reinforcement learning supervision provided in an embodiment of the present application. The specific execution process of the method is as follows:

[0090] S201. Obtain a description of a task to be generated and corresponding user information to be generated.

[0091] In the implementation manner of the present application, the task description to be generated is the task description information input by the user during the actual application process; the user information to be generated refers to the characteristic information of the user who formulates the task description to be generated.

[0092] S202: Input the task description to be generated and the user information to be generated into the target Prompt generation model to obtain the target Prompt.

[0093] In the implementation manner of the present application, the target prompt generation model includes a user information feature extraction unit and a prompt generation unit; Figure 3 As shown, it is a schematic diagram of generating a target Prompt provided in an embodiment of the present application, the user information to be generated is input into the user information feature extraction unit in the target Prompt generation model to obtain the user information features to be generated; the task description to be generated and the user information features to be generated are input into the Prompt generation unit in the target Prompt generation model to obtain the target Prompt.

[0094] Furthermore, the method also includes: inputting the target Prompt into the large language model LLM to obtain a target response.

[0095] Based on the same inventive concept, the embodiments of the present application also provide a personalized large language model Prompt generation device for reinforcement learning supervision corresponding to the personalized large language model Prompt generation method for reinforcement learning supervision. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the personalized large language model Prompt generation method for reinforcement learning supervision mentioned above in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0096] Reference Figure 4 As shown, it is a schematic diagram of a personalized large language model Prompt generation device for reinforcement learning supervision provided in an embodiment of the present application, and the device includes:

[0097] The acquisition module 401 is used to acquire sample data and corresponding Prompt tags; the sample data includes a first task description sample and a corresponding first user information sample;

[0098] An input module 402 is used to input sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model to obtain a first generated Prompt and a second generated Prompt;

[0099] The input module 402 is further used to input the first generated prompt into the large language model to obtain a first generated reply; input the first generated prompt and the first generated reply into the reward model to obtain a reward value corresponding to the first generated prompt; the reward model is trained based on the sample data and reward value label obtained from the second task description sample and the corresponding second user information sample;

[0100] The updating module 403 is used to update the parameters of the initial Prompt generation model based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

[0101] In a possible implementation, the objective function for updating the parameters of the initial Prompt generation model to obtain the target Prompt generation model is as follows:

[0102]

[0103] in, is the parameter of the initial Prompt generation model, E is the expected value, x is the sample data, is the prompt label, D is the data set containing sample data and corresponding prompt labels, model_reward is the reward model, x′ is the first generated prompt, model_reward(x′) is the reward value corresponding to the first generated prompt, β is a constant, Generate the model for the initial prompt in the model parameters Given the sample data, the probability of getting the first generated prompt is, y is the first generated prompt, model SFT (x|y) is the probability of obtaining the second generated prompt given the sample data by the intermediate prompt generation model, Prompt for the second generation.

[0104] In a possible implementation, the input module 402 is further configured to:

[0105] Obtain a first prompt sample and a corresponding first reply sample based on the second user information sample and the second task description sample, a second prompt sample and a corresponding second reply sample based on the second task description sample, and a first reward value label corresponding to the first prompt sample and the first reply sample, and a second reward value label corresponding to the second prompt sample and the second reply sample; the value corresponding to the first reward value label is greater than the value corresponding to the second reward value label;

[0106] The reward model is trained through the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first reward value label, and the second reward value label.

[0107] In one possible implementation, the input module 402 is specifically used to input a first Prompt sample and a corresponding first reply sample into a reward model to obtain a first predicted reward value corresponding to the first Prompt sample; input a second Prompt sample and a corresponding second reply sample into the reward model to obtain a second predicted reward value corresponding to the second Prompt sample; and update the parameters of the reward model using the first Prompt sample, the first reply sample, the second Prompt sample, the second reply sample, the first predicted reward value, the second predicted reward value, the first reward value label, and the second reward value label until a preset expected value corresponding to the reward model is met.

[0108] In a possible implementation, both the initial Prompt generation model and the intermediate Prompt generation model include a user information feature extraction unit and a Prompt generation unit; the input module 402 is specifically used to input the first user information sample into the user information feature extraction unit in the initial Prompt generation model and the user information feature extraction unit in the intermediate Prompt generation model to obtain the first user information feature and the second user information feature; input the first task description sample and the first user information feature into the Prompt generation unit in the initial Prompt generation model to obtain the first generated Prompt; input the first task description sample and the second user information feature into the Prompt generation unit in the intermediate Prompt generation model to obtain the second generated Prompt.

[0109] In a possible implementation, the input module 402 is further configured to:

[0110] Acquire a third task description sample and a corresponding third user information sample, and a Prompt tag corresponding to the third task description sample and the third user information sample;

[0111] Inputting the third task description sample and the third user information sample into the initial Prompt generation model to obtain a third generated Prompt;

[0112] The initial Prompt generation model is trained based on the third task description sample, the third user information sample, the third generated Prompt, and the Prompt label corresponding to the third task description sample and the third user information sample to obtain an intermediate Prompt generation model.

[0113] In a possible implementation, the acquisition module 401 is further used to acquire the task description to be generated and the corresponding user information to be generated;

[0114] The input module 402 is further used to input the task description to be generated and the user information to be generated into the target Prompt generation model to obtain the target Prompt.

[0115] The embodiment of the present application provides a personalized large language model Prompt generation device with reinforcement learning supervision, the method comprising: an acquisition module 401, used to obtain sample data and a corresponding Prompt label; the sample data includes a first task description sample and a corresponding first user information sample; an input module 402, used to input the sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model, respectively, to obtain a first generated Prompt and a second generated Prompt; the input module 402 is also used to input the first generated Prompt into the large language model to obtain a first generated reply; the first generated Prompt is input ... rompt and the first generated reply are input into the reward model to obtain the reward value corresponding to the first generated Prompt; the reward model is trained based on the sample data and reward value label obtained from the second task description sample and the corresponding second user information sample; the update module 403 is used to update the parameters of the initial Prompt generation model based on the sample data, Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt, until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt. The present application obtains the Prompt generation model based on the first task description sample and the corresponding first user information sample training, which can improve the generation effect of the Prompt, thereby providing users with personalized replies and improving the reply effect of the large language model.

[0116] like Figure 5 As shown, an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus, the memory 502 stores machine-readable instructions executable by the processor 501, and when the electronic device is running, the processor 501 communicates with the memory 502 through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the personalized large language model Prompt generation method supervised by reinforcement learning as described above.

[0117] Specifically, the above-mentioned memory 502 and processor 501 can be general-purpose memory and processor, which are not specifically limited here. When the processor 501 runs the computer program stored in the memory 502, it can execute the above-mentioned reinforcement learning supervised personalized large language model Prompt generation method.

[0118] Corresponding to the above-mentioned reinforcement learning supervised personalized large language model Prompt generation method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reinforcement learning supervised personalized large language model Prompt generation method are executed.

[0119] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0120] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0123] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A personalized large language model prompt generation method supervised by reinforcement learning, characterized in that: The method comprises: Acquire sample data and a corresponding Prompt tag; the sample data includes a first task description sample and a corresponding first user information sample; Inputting the sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model respectively to obtain a first generated Prompt and a second generated Prompt; Inputting the first generated prompt into a large language model to obtain a first generated reply; inputting the first generated prompt and the first generated reply into a reward model to obtain a reward value corresponding to the first generated prompt; the reward model is trained based on sample data and reward value labels obtained from a second task description sample and a corresponding second user information sample; Based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt, the parameters of the initial Prompt generation model are updated until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

2. The method for generating a personalized large language model prompt under reinforcement learning supervision according to claim 1, characterized in that: The objective function used to update the parameters of the initial Prompt generation model to obtain the target Prompt generation model is as follows: in, is the parameter of the initial Prompt generation model, E is the expected value, x is the sample data, is the Prompt label, D is a data set containing the sample data and the corresponding Prompt label, model_reward is the reward model, x′ is the first generated Prompt, model_reward(x′) is the reward value corresponding to the first generated Prompt, β is a constant, Generate the model for the initial prompt in the model parameters Given the sample data, the probability of getting the first generated prompt is, y is the first generated prompt, model SFT (x|y) is the probability of obtaining the second generated Prompt given the sample data by the intermediate Prompt generation model, Prompt for the second generation.

3. The method for generating a personalized large language model Prompt with reinforcement learning supervision according to claim 1 or 2, characterized in that: The reward model is trained by the following steps: Obtaining a first Prompt sample and a corresponding first reply sample obtained based on the second user information sample and the second task description sample, a second Prompt sample and a corresponding second reply sample obtained based on the second task description sample, and a first reward value label corresponding to the first Prompt sample and the first reply sample, and a second reward value label corresponding to the second Prompt sample and the second reply sample; the value corresponding to the first reward value label is greater than the value corresponding to the second reward value label; The reward model is trained through the first Prompt sample, the first reply sample, the second Prompt sample, the second reply sample, the first reward value label, and the second reward value label.

4. The method for generating a personalized large language model prompt under reinforcement learning supervision according to claim 3, characterized in that: The training of the reward model by using the first prompt sample, the first reply sample, the second prompt sample, the second reply sample, the first reward value label, and the second reward value label includes: Inputting the first Prompt sample and the corresponding first reply sample into the reward model to obtain a first predicted reward value corresponding to the first Prompt sample; Inputting the second Prompt sample and the corresponding second reply sample into the reward model to obtain a second predicted reward value corresponding to the second Prompt sample; Update the parameters of the reward model with the first Prompt sample, the first reply sample, the second Prompt sample, the second reply sample, the first predicted reward value, the second predicted reward value, the first reward value label, and the second reward value label until a preset expected value corresponding to the reward model is met.

5. The method for generating a personalized large language model Prompt under reinforcement learning supervision according to claim 4, characterized in that: The initial Prompt generation model and the intermediate Prompt generation model both include a user information feature extraction unit and a Prompt generation unit; the sample data is respectively input into the initial Prompt generation model and the pre-trained intermediate Prompt generation model to obtain a first generated Prompt and a second generated Prompt, including: Inputting the first user information sample into the user information feature extraction unit in the initial Prompt generation model and the user information feature extraction unit in the intermediate Prompt generation model to obtain a first user information feature and a second user information feature; Inputting the first task description sample and the first user information feature into the Prompt generation unit in the initial Prompt generation model to obtain the first generated Prompt; The first task description sample and the second user information feature are input into the Prompt generation unit in the intermediate Prompt generation model to obtain the second generated Prompt.

6. The method for generating a personalized large language model Prompt with reinforcement learning supervision according to claim 1, characterized in that: The intermediate Prompt generation model is trained by the following steps: Acquire a third task description sample and a corresponding third user information sample, and a Prompt tag corresponding to the third task description sample and the third user information sample; Inputting the third task description sample and the third user information sample into the initial Prompt generation model to obtain a third generated Prompt; The initial Prompt generation model is trained based on the third task description sample, the third user information sample, the third generated Prompt, and the Prompt label corresponding to the third task description sample and the third user information sample to obtain the intermediate Prompt generation model.

7. The method for generating a personalized large language model prompt under reinforcement learning supervision according to claim 1, characterized in that: The method further comprises: Obtain the task description to be generated and the corresponding user information to be generated; The task description to be generated and the user information to be generated are input into the target Prompt generation model to obtain the target Prompt.

8. A personalized large language model prompt generation device with reinforcement learning supervision, characterized in that: The device comprises: An acquisition module, used to acquire sample data and a corresponding Prompt tag; the sample data includes a first task description sample and a corresponding first user information sample; An input module, used to input the sample data into an initial Prompt generation model and a pre-trained intermediate Prompt generation model respectively, to obtain a first generated Prompt and a second generated Prompt; The input module is further used to input the first generated prompt into the large language model to obtain a first generated reply; input the first generated prompt and the first generated reply into the reward model to obtain a reward value corresponding to the first generated prompt; the reward model is trained based on sample data and reward value labels obtained from the second task description sample and the corresponding second user information sample; An updating module is used to update the parameters of the initial Prompt generation model based on the sample data, the Prompt label, the first generated Prompt, the second generated Prompt and the reward value corresponding to the first generated Prompt, until the preset expected value corresponding to the target Prompt generation model is met to obtain the target Prompt generation model to generate a large language model Prompt.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the personalized large language model Prompt generation method supervised by reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for generating a personalized large language model Prompt under reinforcement learning supervision as described in any one of claims 1 to 7.

Images