Training of models for questions and answers

Abstract

The embodiment of the invention provides a scheme for training a question and answer model. The method comprises the steps of generating a sample response by inputting and providing sample cue words in a training data set into a to-be-trained target model; determining a sample reward score of the sample response using the trained reward model; obtaining a basic reward score of a basic response for the sample cue word input, wherein the basic response is generated by providing the sample cue word input to a basic model corresponding to the target model; and trimming the target model based on an optimization target configured to increase or maximize a difference between the sample reward score input for the sample cue and the base reward score.

Description

Technical Field

[0001] The disclosed example embodiments relate generally to machine learning, and more particularly to methods, apparatus, devices, and computer-readable storage media for training models for question answering. Background Art

[0002] Machine learning models, such as language models (LMs), are able to perform a wide range of natural language processing (NLP) tasks, including but not limited to question answering, text generation, summarization, translation, and sentiment analysis. The success of deploying LMs can be attributed to their ability to follow instructions and learn using human feedback. A key step in achieving the above is LM alignment. Among the different options, the reinforcement learning from human feedback (RLHF) pipeline is a widely recognized method to align LMs from human feedback. Despite its success, the effectiveness of RLHF heavily relies on the reward model (RM) used in the proximal policy optimization (PPO) stage to guide the learning process. However, there are limitations to the RLHF pipeline, and the RLHF pipeline needs further improvement. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for model training is provided. The method includes: generating a sample response by providing a sample prompt word input in a training data set to a target model to be trained; determining a sample reward score of the sample response using a trained reward model; obtaining a basic reward score of a basic response to the sample prompt word input, the basic response being generated by providing the sample prompt word input to a basic model corresponding to the target model; and fine-tuning the target model based on an optimization target, the optimization target being configured to increase or maximize the difference between the sample reward score and the basic reward score for the sample prompt word input.

[0004] In a second aspect of the present disclosure, a device for model training is provided. The device includes: a response generation module, configured to generate a sample response by providing a sample prompt word input in a training data set to a target model to be trained; a reward determination module, configured to determine a sample reward score of the sample response using a trained reward model; a basic reward acquisition module, configured to acquire a basic reward score of a basic response to the sample prompt word input, the basic response being generated by providing the sample prompt word input to a basic model corresponding to the target model; and a model fine-tuning module, configured to fine-tune the target model based on an optimization target, the optimization target being configured to increase or maximize the difference between the sample reward score and the basic reward score for the sample prompt word input.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions executable by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the device performs the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the medium stores a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is contained on a computer-readable medium and comprises computer-executable instructions, which are executed by a processor to perform the method of the first aspect to be implemented.

[0008] It should be understood that the contents described in the summary of the invention are neither intended to identify the key or essential features of the embodiments of the present disclosure, nor intended to limit the scope of the present disclosure. Other features of the present disclosure will be easily conceived through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals refer to the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 A schematic diagram for model training according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of a process for model training according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A block diagram showing an apparatus for model training according to some embodiments of the present disclosure; and

[0014] Figure 5 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms will be understood as open inclusion, i.e., "including but not limited to". The term "based on" will be understood as "based at least in part on". The term "one embodiment" or "the embodiment" will be understood as "at least one embodiment". The term "some embodiments" will be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. As used herein, the term "model" may represent the degree of match between various data. For example, the above-mentioned degree of match can be obtained based on various technical solutions currently available and / or developed in the future.

[0017] It should be understood that the data involved in this technical solution (including but not limited to the data itself, data collection or use) should comply with the requirements of relevant laws, regulations and relevant provisions.

[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of personal information involved in the present disclosure should be notified to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user requires obtaining and using the user's personal information. Therefore, the user can choose whether to provide personal information to software or hardware such as electronic devices, applications, servers, or storage media that perform operations of the technical solution of the present disclosure according to the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending the prompt information to the user may be, for example, a pop-up window presenting the prompt information in text form. In addition, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and user authorization process is merely illustrative and does not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] As used herein, the term "model" can learn the association relationship between the corresponding input and output from the training data, so that the corresponding output can be generated for a given input after the training is completed. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. As used herein, a "model" may also be referred to as a "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably herein.

[0023] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing an input and providing a corresponding output, and typically includes an input and output layer and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected sequentially so that the output of a previous layer is provided as an input to a subsequent layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes an input from a previous layer.

[0024] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage and the application stage (also known as the reasoning stage). During the training stage, a given model can be trained using large-scale training data, and the parameter values ​​can be iteratively updated until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to learn the association between input and output from the training data (also known as input to output mapping). Determine the parameter values ​​of the training model. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input and determine the corresponding output based on the parameter values ​​obtained by training.

[0025] Figure 1 A schematic diagram of an example environment 100 is shown in which embodiments of the present disclosure may be implemented. Figure 1 In the environment 100 of FIG. 1 , three different stages of the model are shown, including a training stage 102, a fine-tuning stage (not shown), and an application stage 106. After the training or fine-tuning stage is completed, there may also be a testing stage, which is Figure 1 Not shown in FIG.

[0026] In the training phase 102, the model training system 110 is configured to perform training of the model 105 using the training data set 112. At the beginning of the training, the model may have initial parameter values. The training process involves updating the parameter values ​​of the model 105 to the desired values ​​based on the training data.

[0027] In the application phase 106, the acquired model 105 has training parameter values ​​that can be provided to the model application system 130 for use. In the application phase 106, the model 105 can be used to process the target input 132 in an actual scenario and provide a corresponding target output 134. In some embodiments where the model 105 is capable of question and answer, for example, the model 105 is a generative model for content generation, the target input 132 can be a prompt word input (which can be considered a question), and the target output 134 can be a response or answer to the prompt word input. In some examples of building the model 105 based on a language model, the prompt word input can include a text sequence, and the response or answer can also include a text sequence of the answer.

[0028] exist Figure 1 In the present invention, the model training system 110 and the model application system 130 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a netbook, a tablet computer, a media computer, a multimedia tablet computer, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include but are not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0029] It should be understood that Figure 1 The components and arrangements in the environment 100 shown are merely examples, and a computing system suitable for implementing the example embodiments described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model application system 130 may be integrated into the same system or device. Implementations of the present disclosure are not limited thereto.

[0030] It should be understood that the structure and function of each element in the environment 100 is described for illustrative purposes only and does not imply any limitation on the scope of the present disclosure.

[0031] In addition to the development of LM, RLHF has become an important method to improve LM. RLHF involves not only training the model on a static dataset, but also incorporating human feedback to guide the learning process. This approach is particularly useful when aligning knowledge learning and mining with human feedback. For example, some existing models have utilized RLHF to tailor responses based on human preferences, resulting in more accurate outputs.

[0032] As mentioned above, there are limitations to the RLHF pipeline. Existing RLHFs rely heavily on accurate and informative reward models that are vulnerable to attacks and sensitive to noise from various sources (e.g., human labeling errors), making the pipeline brittle.

[0033] Designing accurate and informative reward models remains a significant challenge. For example, when deployed in real-world settings, reward models often exhibit limited generalization capabilities. More specifically, the quality of reward models is affected by two sources: the low quality and inherent ambiguity of preference data, and the sensitivity of RM training to training details, which leads to reward hacking. The above observations provide strong motivation for techniques that improve robustness relative to RLHF.

[0034] In order to improve the RLHF pipeline, in an example embodiment of the present disclosure, an improved scheme for model training is provided. Specifically, a sample response is generated by providing a sample prompt word input within a training data set to a target model to be trained. A sample reward score for the sample response is determined using a trained reward model. A base reward score for a base response to the sample prompt word input is obtained, and the base response is generated by providing the sample prompt word input to a base model corresponding to the target model. The target model is fine-tuned based on an optimization objective, and the optimization objective is configured to increase or maximize the difference between the sample reward score and the base reward score for the sample prompt word input.

[0035] Using the solution of the present disclosure, a simple fix to RLHF is proposed, which leads to significant performance improvements when compared to the standard RLHF or PPO utilized in model training. The solution can use the difference between the sample reward scores and the base reward scores to calibrate the RLHF process, thereby improving the robustness of RLHF, reducing the variance of PPO and encouraging improvements over the baseline.

[0036] Before going into the solutions proposed by the embodiments of the present disclosure, the present disclosure begins by providing an overview of the RLHF pipeline.

[0037] RLHF usually involves collecting human feedback, training a reward model, and optimizing a policy using reinforcement learning (RL). This disclosure briefly outlines the last two steps. Regarding reward modeling, taking paired preference data annotation as an example, the supervised fine-tuning (SFT) model πSFT Generate two different outputs (y1, y2) based on user query x~π SFT (y|x). Human annotators were instructed to select their preferred outputs that would result in y w >y l , where y w and l denote the preferred output and rejected output from the output pair (y1, y2), respectively. In order to train the reward model r using human feedback ψ , the parameter ψ is optimized to minimize the following objective with respect to the collected dataset:

[0038] In formula (1), represents a suitable loss function and λ r represents the regularization term. When the feedback consists of pairwise comparisons, a binary ranking loss can be used, where the learning objective of Equation (1) aims to make the selected sample the winner in two cases:

[0039] In formula (2), the data set consists of comparisons, expressed as Reward Model ψ This is usually adjusted by including an additional linear layer at the final Transformer layer, resulting in a representation of r ψ A single scalar prediction for (x,y). This prediction serves as a representation of the reward value associated with the input pair (x,y).

[0040] Regarding strategy optimization using RL, reward model r ψ Can be used to fine-tune the base model through reinforcement learning. The new parameter π RL θ new is trained to maximize the following objective:

[0041] In equation (3), η represents a regularization term, such as a penalty based on the Kullback-Leibler (KL) divergence. In this context, the KL divergence term serves two main purposes. First, it acts as an entropy reward, maintaining generative diversity and preventing mode collapse into a single high-reward answer. Second, it ensures that the output of the RL policy does not deviate significantly from the distribution of the reference model.

[0042] For more details on the model training solution, refer to Figure 2 . Figure 2 A schematic diagram 200 for model training according to some embodiments of the present disclosure is shown.

[0043] exist Figure 2The proposed solution is outlined in . The solution is conducted in two stages. In the first stage, the base (SFT) model is used to generate responses for the cue words that we will use in the PPO stage. These cue words together with the baseline responses will help define the reward and penalty terms.

[0044] In the second phase, the generated baseline response will help define the calibrated and penalized reward that will be used in the PPO phase. The penalty term is lightweight to compute and only requires the original reward for the generated baseline response to be called up by the reward model.

[0045] like Figure 2 As shown, a sample prompt word input 220 (e.g., prompt word A 220-1 or prompt word B 220-2) in a training data set is provided to a target model 230 (sometimes also referred to as a policy model) to be trained or fine-tuned to generate a sample response (e.g., response A 210-1 or response B 210-2). The target model 230 (e.g., LM) to be trained can generate a sample response based on the sample prompt word input 220. The sample prompt word input can be selected from a set of sample prompt word inputs. (also known as the RL dataset) where x i may represent sample prompt word inputs, and M may represent the total number of sample prompt word inputs. can all be used to train the target model, and the processing described in this article can be applied to each sample prompt word input

[0046] After generating a sample response to the sample prompt word input 220, a sample reward score 270 for the sample response is determined using the trained reward model 240. In an example, the sample reward score may be represented as r x,y , where x represents a sample cue word input and y represents a sample response.

[0047] In addition, a base reward score 260 (sometimes also referred to as an offline reward) for the base response to the sample prompt word input 220 can be obtained. The base response is generated by providing the sample prompt word input 220 to a base model 250 (sometimes also referred to as an SFT model) corresponding to the target model 230. In some embodiments, the base model 250 may include a model that has been trained from an original model by supervised fine-tuning (SFT). In an example, the purpose of the base model 250 is to provide a starting point for evaluation and improvement.

[0048] In some embodiments, a base reward score 260 for a base response may be determined using a trained reward model 240 in a manner similar to determining a sample reward score 270 for a sample response.

[0049] In some embodiments, multiple base responses may be generated by providing the sample cue word input 220 multiple times to the base model 250. Given the base model 250 (referred to as the SFT model), k responses may be sampled for each of the M cue word inputs. A set of baseline responses (i.e., base responses) may be obtained, which may be represented as In one example, these responses can be expressed as The original prompt word input combination of , where the j-th baseline response to the unindexed prompt word can be expressed as By adopting this direct sampling approach, synthetic data can be generated. Furthermore, the temperature during sampling can be adjusted to generate a wider range of responses from the same underlying model, effectively improving the diversity of the generated responses.

[0050] In some embodiments, a reward model may be used to determine multiple base reward scores 260() for multiple base responses, respectively. In one example, once the sample output (also referred to as the base response) is obtained from the base model, the reward model 240 may be used to assign a score to each of these combination sequences (i.e., the base response). Thus, a reward list corresponding to each prompt word may be obtained, and the offline reward derived therefrom is represented as

[0051] These offline rewards reflect the implicit ability of the base model with respect to sample cue word inputs in the RL dataset, and they can be referred to as offline contrastive rewards.

[0052] In some embodiments, multiple base reward scores 260 may be aggregated to provide an aggregated base reward score. Can be used to represent the aggregated base reward score, where g(·) can represent an aggregation function and can be used to calculate the average of multiple base reward scores. Using these embodiments, offline sampling can be performed to obtain a set of baseline responses to sampled prompt word inputs that will be used to calculate comparative rewards in the PPO stage, thereby reducing the synchronization time overhead associated with additional sampling during the RL stage.

[0053] After obtaining the base reward score, the target model 230 is fine-tuned based on the optimization target. The optimization target is configured to increase or maximize the difference between the sample reward score 270 and the base reward score 260 for the sample prompt word input. Figure 2, the difference between the sample reward score 270-2 and the base reward score 260-2 corresponding to the pair of prompt word B 220-2 and response B 210-2 (i.e., 1.5) is greater than the difference between the sample reward score 270-1 and the base reward score 260-1 corresponding to the pair of prompt word A 220-1 and response B 210-1 (i.e., 0.5), so the target model can be fine-tuned towards prompt word B 220-2 and response B210-2.

[0054] In some embodiments, the target model 230 to be trained may include a model that has been fine-tuned from the base model 250 after at least one round of reinforcement learning. In an example, the target model 230 may be fine-tuned from the base model by reinforcement learning by the reward model. With these embodiments, the target model may be improved for several rounds and may output more accurate results.

[0055] In some embodiments, the target model 230 can be fine-tuned by reinforcement learning based on the optimization goal. The reward model 240 can be used to fine-tune the base model by reinforcement learning. In the RL stage, the main goal is to learn the θ The strategy of (·|x) maximizes the following contrast reward: Where g(·) may represent an aggregation function. In some examples, g(·) may be selected as an average function. Based on formula (4), the optimization objective may be expressed as follows:

[0056] During the RL stage, by following the PPO training setting, the optimization objective can also be expressed as follows: With these embodiments, a comparative reward is calculated using a sampled baseline response. The rewards obtained during RL training are compared to their corresponding comparative rewards, and an implicit comparative reward framework is established during the RL phase. This "penalty" reward information enables the RL policy to improve itself based on the observed differences.

[0057] In some embodiments, the target model 230 and the base model 250 can be built based on the content generation model. Using these embodiments, after several rounds of fine-tuning, the target model can generate content that better captures human preferences.

[0058] The following will provide rewards for contrast penalties The performance of the solution proposed by the embodiments of the present disclosure is analyzed based on an intuitive understanding of how it works. The analysis is simplified by assuming that using an aggregated baseline answer is equivalent to drawing a single baseline answer from a specific distribution, thereby inducing a specific reward:

[0059] To simplify the analysis, only binary rewards r∈{0,1} are considered. The following two variables are introduced to capture the “(in)consistency” of the reward function on (x,y): in corresponds to a perfect reward function that accurately evaluates the quality of y for cue word x. High c x,0 、c x,1 Indicates high inconsistency / variance in the reward function over samples x, thus capturing the uncertainty of the reward model.

[0060] The following theorem can be proved. Assume r x,y , In a given In the case of conditional independence, then The above theorem reveals the following functionality in the proposed contrastive penalty reward.

[0061] The first function is to punish uncertainty. The magnitude is relative to (1-c x,0 -c x,1 ) shows a linear decreasing relationship, where high uncertainty (small c x,0 ,c x,1 ) is heavily penalized by a constant. In other words, when the reward function is highly inaccurate at a particular x, the influence of x during PPO decreases linearly with respect to the uncertainty term.

[0062] The second function is to improve robustness. If the reward noise is assumed to be x,0 ≡c0,c x,1 ≡c1 (i.e., the reward function suffers from similar amounts of error for different (x,y) pairs), then the first constant linear term (i.e., (1-c0-c1)) becomes irrelevant to the reward maximization problem and thus improves the training’s resistance to this noise.

[0063] The third function is to encourage improvement. It also reveals that by using comparative rewards, The new answer y is encouraged to be consistent with the baseline answer y base Significantly different.

[0064] The fourth function is about the calibration of task difficulty. The last item, namely Downweight tasks with higher difficulty, i.e., where high true rewards are observed This helps the PPO step to focus less on instances where there may be inherent ambiguity in obtaining high-quality answers, either due to poor cue words or the nature of the question.

[0065] The fifth function is variance reduction. Baseline rewards can help reduce variance. This is also obvious from the above theorem. Linear terms, such as (1-c x,0 -c x,1 ), appropriately reducing the reward magnitude and thus reducing its variance.

[0066] Figure 3 FIG. 3 is a flowchart of a process 300 for model training according to some embodiments of the present disclosure. The process 300 may be performed in Figure 1 The model training system is implemented at 110.

[0067] At box 310, the model training system 110 generates sample responses by providing sample prompt word inputs within the training data set to the target model to be trained.

[0068] At block 320 , the model training system 110 determines a sample reward score for the sample response using the trained reward model.

[0069] At block 330, the model training system 110 obtains a base reward score for a base response to the sample prompt word input. The base response is generated by providing the sample prompt word input to a base model corresponding to the target model.

[0070] At block 340 , the model training system 110 fine-tunes the target model based on the optimization objective. The optimization objective is configured to increase or maximize the difference between the sample reward score and the base reward score for the sample prompt word input.

[0071] In some embodiments, obtaining the base reward score includes determining the base reward score for the base response using a trained reward model.

[0072] In some embodiments, obtaining a base reward score includes: generating a plurality of base responses by providing a sample prompt word input to a base model multiple times; determining a plurality of base reward scores for the plurality of base responses respectively using the reward model; and aggregating the plurality of base reward scores to provide an aggregated base reward score.

[0073] In some embodiments, fine-tuning the target model based on the optimization objective includes: fine-tuning the target model through reinforcement learning based on the optimization objective.

[0074] In some embodiments, the base model includes a model that has been trained from the original model through supervised fine-tuning (SFT), and the target model to be trained includes a model fine-tuned from the base model after at least one round of reinforcement learning.

[0075] In some embodiments, the target model and the base model are built based on the content generation model.

[0076] Figure 4 4 shows a block diagram of an apparatus 400 for model training according to some embodiments of the present disclosure. The apparatus 400 may be, for example, Figure 1 The model training system 110 is implemented or included in the device 400. The various modules / components in the device 400 can be implemented by hardware, software, firmware or any combination thereof.

[0077] As shown in the figure, the device 400 includes: a response generation module 410, which is configured to generate a sample response by providing a sample prompt word input in a training data set to a target model to be trained; a reward determination module 420, which is configured to determine a sample reward score of the sample response using the trained reward model; a basic reward acquisition module 430, which is configured to obtain a basic reward score for a basic response to the sample prompt word input, where the basic response is generated by providing the sample prompt word input to a basic model corresponding to the target model; and a model fine-tuning module 440, which is configured to fine-tune the target model based on an optimization target, where the optimization target is configured to increase or maximize the difference between the sample reward score and the basic reward score for the sample prompt word input.

[0078] In some embodiments, the base reward obtaining module 430 includes a first sub-base reward obtaining module configured to determine a base reward score of a base response using a trained reward model.

[0079] In some embodiments, the basic reward acquisition module 430 includes a second sub-basic reward acquisition module, which is configured to generate multiple basic responses by providing the sample prompt word input to the basic model multiple times; use the reward model to determine multiple basic reward scores for the multiple basic responses respectively; and aggregate the multiple basic reward scores to provide an aggregated basic reward score.

[0080] In some embodiments, the model fine-tuning module 440 includes a reinforcement learning module, which is configured to fine-tune the target model through reinforcement learning based on the optimization goal.

[0081] In some embodiments, the base model includes a model that has been trained from an original model by supervised fine-tuning (SFT), and the target model to be trained includes a model fine-tuned from the base model after at least one round of reinforcement learning.

[0082] In some embodiments, the target model and the base model are built based on the content generation model.

[0083] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is only an example and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 500 can be used to implement, for example Figure 1 The electronic device 500 can also be used to implement Figure 4 Device 400.

[0084] like Figure 5 As shown, the electronic device 500 is in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and may perform various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 500.

[0085] The electronic device 500 typically includes various computer storage media. Such media can be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or any combination thereof. The storage device 530 can be any removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data (such as training data for training) and can be accessed within the electronic device 500.

[0086] The electronic device 500 may also include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0087] The communication unit 540 communicates with another computing device via a communication medium. Additionally, the functions of the components in the electronic device 500 can be implemented by a single computing cluster or multiple computing machines, which can communicate via a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0088] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed. The external devices (such as storage devices, display devices, etc.) communicate with one or more devices that enable a user to interact with the electronic device 500, or communicate with any device (e.g., a network card, a modem, etc.) that enables the electronic device 500 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0089] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which a computer-executable instruction or a computer program is stored, and the computer-executable instruction or the computer program is executed by a processor to implement the above method. According to an exemplary implementation of the present disclosure, a computer program product is also provided. The computer program product is physically stored on a non-transitory computer-readable medium and includes a computer-executable instruction, and the computer-executable instruction is executed by a processor to implement the above method.

[0090] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of each frame in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to the processing unit of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and this machine generation device is to realize the function / action specified in one or more boxes in the flow chart and / or block diagram when these instructions are executed by the processing unit of a computer or other programmable data processing device. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions enable a computer, a programmable data processing device and / or other equipment to work in an ad hoc manner. Therefore, the computer-readable medium comprising instructions includes a product, and the product includes instructions for realizing the various aspects of the function / action specified in one or more boxes in the flow chart and / or block diagram.

[0092] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps can be executed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0093] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product implemented according to the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, which contains one or more executable instructions for realizing a specified logical function. In some alternative embodiments, the function marked in the frame can also occur in an order different from the order marked in the accompanying drawings. For example, two continuous frames can actually be executed in parallel, and sometimes can also be executed in reverse order, depending on the function involved. It should also be noted that each frame in the block diagram and / or the flow chart and the combination of the frames in the block diagram and / or the flow chart can be realized by a special hardware-based system or by a combination of special hardware and computer instructions that performs a specified function or action.

[0094] Each embodiment of the present disclosure has been described above. The above description is exemplary rather than exhaustive and is not limited to the disclosed implementations. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the market for each embodiment, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method for model training, comprising: Generate sample responses by providing sample prompt word inputs in the training data set to the target model to be trained; determining a sample reward score for the sample response using the trained reward model; Obtaining a basic reward score for a basic response to the sample prompt word input, wherein the basic response is generated by providing the sample prompt word input to a basic model corresponding to the target model; as well as The target model is fine-tuned based on an optimization objective, wherein the optimization objective is configured to increase or maximize a difference between the sample reward score and the base reward score for the sample prompt word input.

2. The method according to claim 1, wherein obtaining the basic reward score comprises: The base reward score for the base response is determined using the trained reward model.

3. The method according to claim 1 or 2, wherein obtaining the basic reward score comprises: generating a plurality of base responses by providing the sample cue word input to the base model a plurality of times; Determine a plurality of base reward scores for the plurality of base responses respectively using the reward model; as well as The plurality of base reward scores are aggregated to provide an aggregated base reward score.

4. The method according to any one of claims 1 to 3, wherein fine-tuning the target model based on the optimization target comprises: Based on the optimization objective, the target model is fine-tuned through reinforcement learning.

5. The method according to any one of claims 1 to 4, wherein the base model comprises a model that has been trained from an original model by supervised fine-tuning (SFT), and The target model to be trained includes a model obtained by fine-tuning the base model after at least one round of reinforcement learning.

6. The method according to any one of claims 1 to 5, wherein the target model and the base model are constructed based on a content generation model.

7. A device for model training, comprising: A response generation module configured to generate sample responses by providing sample prompt word inputs in the training data set to a target model to be trained; a reward determination module configured to determine a sample reward score for the sample response using a trained reward model; A basic reward acquisition module is configured to acquire a basic reward score for a basic response to the sample prompt word input, wherein the basic response is generated by providing the sample prompt word input to a basic model corresponding to the target model; as well as A model fine-tuning module is configured to fine-tune the target model based on an optimization target, wherein the optimization target is configured to increase or maximize the difference between the sample reward score and the base reward score for the sample prompt word input.

8. An electronic device comprising: at least one processing unit; as well as at least one memory coupled to the at least one processing unit and storing instructions executable by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the apparatus to perform: Generate sample responses by providing sample prompt word inputs in the training data set to the target model to be trained; determining a sample reward score for the sample response using the trained reward model; Obtaining a basic reward score for a basic response to the sample prompt word input, where the basic response is generated by providing the sample prompt word input to a basic model corresponding to the target model; as well as The target model is fine-tuned based on an optimization objective, wherein the optimization objective is configured to increase or maximize a difference between the sample reward score and the base reward score for the sample prompt word input.

9. The electronic device according to claim 8, wherein obtaining the basic reward score comprises: The base reward score for the base response is determined using the trained reward model.

10. The electronic device according to claim 8 or 9, wherein obtaining the basic reward score comprises: generating a plurality of base responses by providing the sample cue word input to the base model a plurality of times; Determine a plurality of base reward scores for the plurality of base responses respectively using the reward model; as well as The plurality of base reward scores are aggregated to provide an aggregated base reward score.

11. The electronic device according to any one of claims 8 to 10, wherein fine-tuning the target model based on the optimization target comprises: Based on the optimization objective, the target model is fine-tuned through reinforcement learning.

12. The electronic device according to any one of claims 8 to 11, wherein the base model comprises a model that has been trained from an original model by supervised fine-tuning (SFT), and The target model to be trained includes a model obtained by fine-tuning the base model after at least one round of reinforcement learning. 13 . The electronic device according to claim 8 , wherein the target model and the base model are constructed based on a content generation model.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by an electronic device, the device is caused to perform the method according to any one of claims 1 to 6.

15. A computer program product embodied on a computer readable medium and comprising computer executable instructions executable by a processor to perform the method according to any one of claims 1 to 6.

Images