Reinforcement learning model training methods, devices, electronic equipment, and storage media

Abstract

This application relates to the field of artificial intelligence technology and provides a method, apparatus, electronic device, and storage medium for training a reinforcement learning model. The method first acquires N sample data, including questions, candidate answers, evaluations of candidate answers by review groups, and review group identifiers. Then, based on the N sample data, it generates a preference context vector for each review group. At least based on this preference context vector, it constructs a reward model for a pre-trained reinforcement learning model. Next, based on the reward model, it constructs a loss function that includes at least an inter-group alignment loss term, so that the loss function can characterize the evaluation differences of different review groups on the same sample data. Finally, it uses the constructed loss function to train the pre-trained reinforcement learning model, obtaining a trained reinforcement learning model. This method effectively solves the reward learning bias caused by inter-group differences in the reward model and improves the stability of reinforcement learning model training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a reinforcement learning model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the combination of reinforcement learning and RLHF (Reinforcement Learning from Human Feedback), reward models typically use human subjective annotation or evaluation to assign preference scores to multiple candidate outputs and then train them based on these human subjective evaluation scores or comparisons. Current reward model training suffers from two problems affecting training stability:

[0003] 1) Due to cognitive biases, cultural differences, and inconsistent evaluation standards among different annotators regarding non-deterministic tasks, inconsistencies in scoring arise between samples, introducing high variance into the aligned data. This inconsistency directly affects the preference function learned by the reward model, causing it to perform unstablely or become biased in reinforcement learning tasks.

[0004] 2) During the training of the reward model, there is a deviation in the scoring standards between different sampling groups. That is, the reward model lacks a stable standard in different rounds of sampling. This situation will also lead to an imbalance in the reward scores given by the reward model after training.

[0005] Therefore, a method is needed to introduce review panel scoring information and multi-round scoring context information of the model during the training process, in order to structurally model and reduce the differences between scoring groups. Summary of the Invention

[0006] In view of this, embodiments of this application provide a reinforcement learning model training method, apparatus, electronic device, and storage medium to solve the problem of poor training stability of reinforcement learning models in the prior art due to unreasonable construction of reward models and loss functions.

[0007] A first aspect of this application provides a reinforcement learning model training method, including:

[0008] Obtain N sample data, each sample data includes at least a question, a candidate answer to the question from a reinforcement learning pre-trained model, the evaluation of each candidate answer by the target review group, and the target review group identifier; the target review group is any one of the M review groups, where M and N are both positive integers;

[0009] Based on N sample data, generate the preference context vector for each review group;

[0010] At least a reward model for the reinforcement learning pre-trained model should be constructed based on the preference context vectors of each review panel;

[0011] A loss function is constructed based on the reward model. The loss function includes at least an inter-group alignment loss term, which is used to characterize the difference in evaluation of the same sample data by different review groups.

[0012] The pre-trained reinforcement learning model is trained based on the loss function to obtain the trained reinforcement learning model.

[0013] A second aspect of this application provides a reinforcement learning model training apparatus, comprising:

[0014] The acquisition module is configured to acquire N sample data, each sample data including at least a question, candidate answers to the question from the reinforcement learning pre-trained model, evaluations of each candidate answer by the target review group, and the target review group identifier; the target review group is any one of M review groups, where M and N are both positive integers.

[0015] The generation module is configured to generate a preference context vector for each review group based on N sample data.

[0016] The building module is configured to construct a reward model for the reinforcement learning pre-trained model based at least on the preference context vectors of each review panel;

[0017] The building module is also configured to construct a loss function based on the reward model. The loss function includes at least an inter-group alignment loss term, which is used to characterize the difference in evaluation of the same sample data by different review groups.

[0018] The training module is configured to train a pre-trained reinforcement learning model based on a loss function, resulting in a trained reinforcement learning model.

[0019] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0021] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment first obtains N sample data, which includes questions, candidate answers, evaluations of candidate answers by review groups, and review group identifiers. Then, based on the N sample data, a preference context vector for each review group is generated. At least based on the preference context vector, a reward model for a reinforcement learning pre-trained model is constructed. Then, based on the reward model, a loss function including at least an inter-group alignment loss term is constructed so that the loss function can characterize the evaluation differences of different review groups on the same sample data. Finally, the constructed loss function is used to train the reinforcement learning pre-trained model to obtain the trained reinforcement learning model. This can effectively solve the reward learning bias caused by inter-group differences in the reward model and improve the stability of reinforcement learning model training. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a reinforcement learning model training method provided in an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating the method for generating a preference context vector for each review group based on N sample data, as provided in this application embodiment.

[0025] Figure 3 This is a flowchart illustrating the method for determining the preference context vector of a target review group based on a token-encoded vector, as provided in an embodiment of this application.

[0026] Figure 4 This is a flowchart illustrating the method for constructing a reward model provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating the method for determining the context representation of target sample data provided in an embodiment of this application.

[0028] Figure 6 This is a flowchart illustrating the method for determining the inter-group alignment loss term provided in an embodiment of this application.

[0029] Figure 7 This is a schematic diagram of a reinforcement learning model training device provided in an embodiment of this application.

[0030] Figure 8 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] The following will describe in detail, with reference to the accompanying drawings, a reinforcement learning model training method and apparatus according to embodiments of this application.

[0033] As mentioned above, current reward model training suffers from issues affecting the stability of reinforcement learning model training. Related techniques have attempted to address this problem using methods such as normalization, reviewer weighting, and Bayesian modeling. While these methods can alleviate the problem to some extent, they still exhibit poor generalization and significant inference ambiguity in non-deterministic tasks, such as subjective judgment of the merits of answers.

[0034] In view of this, embodiments of this application provide a reinforcement learning model training method. First, N sample data are acquired, including questions, candidate answers, evaluations of candidate answers by review groups, and review group identifiers. Then, a preference context vector for each review group is generated based on the N sample data. At least based on this preference context vector, a reward model for a pre-trained reinforcement learning model is constructed. Next, a loss function, including at least an inter-group alignment loss term, is constructed based on the reward model so that the loss function can characterize the evaluation differences of different review groups on the same sample data. Finally, the constructed loss function is used to train the pre-trained reinforcement learning model, resulting in a trained reinforcement learning model. This method effectively solves the reward learning bias caused by inter-group differences in the reward model and improves the stability of reinforcement learning model training.

[0035] Figure 1 This is a flowchart illustrating a reinforcement learning model training method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0036] In step S101, N sample data are obtained. Each sample data includes at least a question, a candidate answer to the question from the reinforcement learning pre-trained model, the evaluation of each candidate answer by the target review group, and the target review group identifier.

[0037] The target review group is any one of the M review groups, where M and N are both positive integers.

[0038] In step S102, a preference context vector for each review group is generated based on N sample data.

[0039] In step S103, a reward model for the reinforcement learning pre-trained model is constructed based at least on the preference context vectors of each review group.

[0040] In step S104, a loss function is constructed based on the reward model, and the loss function includes at least an inter-group alignment loss term.

[0041] Among them, the inter-group alignment loss term is used to characterize the difference in evaluation of the same sample data by different review groups.

[0042] In step S105, a pre-trained reinforcement learning model is trained based on the loss function to obtain the trained reinforcement learning model.

[0043] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain computing capabilities.

[0044] In some embodiments of this application, N sample data can be obtained first. Each sample data includes at least a question, candidate answers to the question from a reinforcement learning pre-trained model, evaluations of each candidate answer by a target review group, and a target review group identifier. Here, the review group refers to an annotation group or individuals who score or select preferences for each candidate answer. Different review groups have different style preferences.

[0045] In some embodiments of this application, a preference context vector for each review group can be generated based on the N sample data. This preference context vector can be used to characterize the style preferences of each review group. Furthermore, the generated preference context vector can be associated with the review group identifier of each review group.

[0046] In some embodiments of this application, the reward model of the reinforcement learning pre-trained model can be constructed at least based on the preference context vectors of each review group. In one example, a reinforcement learning pre-trained model can be prepared, and the reward model of this pre-trained model can be constructed based on the preference context vectors of each review group.

[0047] In some embodiments, a loss function can also be constructed based on the reward model. The constructed loss function includes at least an inter-group alignment loss term, which characterizes the differences in evaluations of the same sample data by different review groups. Here, the same sample data refers to the same question and the candidate answers of the reinforcement learning pre-trained model to that question.

[0048] After constructing the loss function, a reinforcement learning pre-trained model can be trained based on the loss function to obtain the trained reinforcement learning model.

[0049] According to the technical solution provided in the embodiments of this application, N sample data are first obtained, including questions, candidate answers, evaluations of candidate answers by review groups, and review group identifiers. Then, a preference context vector for each review group is generated based on the N sample data. At least based on the preference context vector, a reward model for a reinforcement learning pre-trained model is constructed. Then, a loss function including at least an inter-group alignment loss term is constructed based on the reward model so that the loss function can characterize the evaluation differences of different review groups on the same sample data. Finally, the constructed loss function is used to train the reinforcement learning pre-trained model to obtain the trained reinforcement learning model. This can effectively solve the reward learning bias caused by inter-group differences in the reward model and improve the stability of reinforcement learning model training.

[0050] Figure 2 This is a flowchart illustrating the method for generating a preference context vector for each review group based on N sample data, as provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0051] In step S201, all sample data of the target review group are obtained from N sample data.

[0052] In step S202, the sample data of the target review group is lexical encoded to obtain the lexical encoding vector of the target review group.

[0053] In step S203, the preference context vector of the target review group is determined based on the word encoding vector.

[0054] In some embodiments of this application, when generating the preference context vector for each review group based on sample data, all sample data of the target review group can be obtained first. Then, the sample data of the target review group is tokenized to obtain the token-encoded vector of the target review group. Finally, the preference context vector of the target review group is determined based on the token-encoded vector.

[0055] By iterating through the M review groups, the preference context vector for each review group can be obtained.

[0056] Figure 3 This is a flowchart illustrating the method for determining the preference context vector of a target review group based on a token-encoded vector, as provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0057] In step S301, the word encoding vector is embedded to obtain the preference context vector of the target review group.

[0058] In step S302, the historical word encoding vector of the target review group is trained to obtain the preference context vector of the target review group.

[0059] In some embodiments of this application, the preference context vector of the target review group can be obtained by embedding the token encoding vector. Alternatively, the preference context vector of the target review group can be obtained by training the historical token encoding vector of the target review group.

[0060] In one example, a method of dynamically adjusting the embedding vector, such as a learnable embedding table, can be used to embed the token-encoded vector to obtain the target review group's preference context vector. Alternatively, historical sample data corresponding to the target review group can be collected. This historical sample data can be part or all of the N sample data. Then, the sample data in the historical sample data that has the identifier of the target review group is tokenized, and the preference context vector of the target review group is trained on each token-encoded vector.

[0061] Figure 4 This is a flowchart illustrating the method for constructing a reward model provided in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0062] In step S401, candidate answers from the target sample data and the contextual representation of the target sample data are obtained.

[0063] The target sample data can be any single sample data.

[0064] In step S402, the candidate answer is concatenated with the context representation to obtain the word encoding of the concatenated text.

[0065] In step S403, the word encoding of the concatenated text is concatenated with the preference context vector of the review group corresponding to the target sample data to obtain the concatenated features.

[0066] In step S404, the reward model for the target sample data is obtained by using the splicing features as the independent variable of the preset network prediction function.

[0067] In some embodiments of this application, when constructing the reward model, candidate answers in the target sample data and the contextual representation of the target sample data can be obtained first. In one example, candidate answers in the target sample data can be represented by r. i The context representation of the target sample data is indicated by p. t The expression is represented as follows: where i and t are both positive integers, i is less than or equal to the number of candidate answers in the target sample data, and t is determined according to actual needs, representing the data rounds preceding the target sample data used for reference.

[0068] In some embodiments, the candidate answer can be concatenated with the context representation of the target sample data, and the concatenated text can be tokenized. In one example, this token encoding can be represented as token(r) i *p t ), where "*" is a text concatenation character.

[0069] In some embodiments of this application, the preference context vector of the review group corresponding to the target sample data can also be obtained. For example, the review group identifier in the target sample data can be obtained, and its corresponding preference context vector can be determined based on the review group identifier. In one example, the preference context vector of the review group corresponding to the target sample data can be represented by g. i express.

[0070] The token encoding of the concatenated text can be combined with the preference context vector of the review group corresponding to the target sample data to obtain the concatenated feature. In one example, this concatenated feature can be represented as [token(r i *p t ); g i The symbol ] indicates that the semicolon is a feature concatenation character.

[0071] In some implementations, the concatenated features are used as independent variables in a preset network prediction function to obtain a reward model for the target sample data. In one example, if the preset network prediction function is f... θ (), then the reward model for the target sample data is S(r) i p t g i =f θ ([token(r)) i *p t ); g i ]).

[0072] Figure 5 This is a flowchart illustrating the method for determining the contextual representation of target sample data provided in an embodiment of this application. For example... Figure 5 As shown, the method includes the following steps:

[0073] In step S501, the evaluation window for the previous t rounds is obtained before the target sample data is acquired.

[0074] The evaluation window includes questions for each round, candidate answers from the reinforcement learning pre-trained model to the questions, and evaluations of each candidate answer by the target review panel.

[0075] In step S502, all items in the evaluation window of round t are concatenated with text to obtain the contextual representation of the target sample data.

[0076] In some embodiments of this application, when determining the contextual representation of the target sample data, the evaluation window of the target sample data in the previous t rounds can be obtained first. The evaluation window includes the questions in each round, the candidate answers of the reinforcement learning pre-trained model to the questions, and the evaluation of each candidate answer by the target review group.

[0077] Then, by concatenating the text of all items in the evaluation window of round t, the contextual representation of the target sample data can be obtained.

[0078] Figure 6 This is a flowchart illustrating the method for determining the inter-group alignment loss term provided in an embodiment of this application. Figure 6 As shown, the method includes the following steps:

[0079] In step S601, the first reward value and the second reward value of the target sample data are calculated respectively.

[0080] The first reward value and the second reward value correspond to the same candidate answer and target sample data context representation, and correspond to the preference context vectors of different review groups.

[0081] In step S602, the difference between the first reward value and the second reward value is calculated using the second normal form to obtain the alignment loss term between candidate groups.

[0082] In step S603, the reward values ​​of all candidate answers and context representations of the target sample data that correspond to the same candidate answer and target sample data, and the preference context vectors of different review groups are traversed in the target sample data, and the sum of all candidate group alignment loss terms is determined as the group alignment loss term.

[0083] In some embodiments of this application, when determining the inter-group alignment loss term, a first reward value and a second reward value of the target sample data can be used respectively. The first reward value and the second reward value correspond to the same candidate answer and the context representation of the target sample data, but to the preference context vectors of different review groups. For example, the first reward value can be denoted as S(r i p t g j The second reward value can be denoted as S(r). i p t g k ), j and k are both positive integers less than or equal to M, and j is not equal to k.

[0084] The difference between the first and second reward values ​​can be calculated using the second normal form to obtain the alignment loss term between candidate groups. In one example, g j and g k The candidate group alignment loss term is || S(r) i p t gj )- S(r i p t g k )|| 2 .

[0085] Finally, we can iterate through all the reward values ​​in the target sample data that correspond to the same candidate answer and the context representation of the target sample data, and to the preference context vectors of different review groups, and determine the sum of all candidate group alignment loss terms as the group alignment loss term.

[0086] In one example, let L be the inter-group alignment loss term. align ,but .

[0087] In some embodiments of this application, the loss function may further include an intra-group consistency loss term, which characterizes the consistency between different candidate answers to the same question. This intra-group consistency loss term can be determined as follows:

[0088] Determined using the within-group normalized relative loss method; or

[0089] Determined using a metric learning algorithm; or

[0090] The intra-group relative strategy optimization algorithm is used to determine this.

[0091] In other words, this application's embodiments effectively address the reward learning bias caused by differences in standards between reward model participants by introducing reviewer group scoring information and a multi-round scoring context-aware modeling method into the reward model training. This method guides the reward model to learn a preference function that is consistent within the standard and compatible between groups by jointly modeling the outputs of different reviewer groups and the scoring context information.

[0092] During the data preparation phase, scoring data from multiple review groups on the same task sample can be collected. Each sample data may include the question, the candidate answers of the reinforcement learning pre-trained model to the question, the evaluation of each candidate answer by the target review group, and the target review group identifier.

[0093] Next, we can embed the review group and generate the preference context vector g corresponding to the review group identifier. i The embedding can be generated based on a learnable embedding table (dynamically adjusted embedding) method or trained from historical scores from the review panel.

[0094] Then, a reward model is constructed. The input to the reward model can include candidate answers r from the target sample data. i Contextual representation of target sample data p tThe preference context vector g of the review group corresponding to the target sample data i The reward function is modeled as S(r) i p t g i =f θ ([token(r)) i *p t ); g i ]).

[0095] Next, we can design the loss function. The loss function can be designed to simultaneously optimize two objectives: the intra-group consistency loss L... intra Inter-group alignment loss L align L intra The relative loss can be determined using the within-group normalized relative loss algorithm, or using the conventional pairwise ranking loss algorithm. align To characterize the evaluation differences between different review groups for the same sample, regularization terms can be added to encourage the model outputs to remain semantically similar. For example, the margin value in the model can be set to prevent it from being too large.

[0096] In one real-time mode, it is possible to set .

[0097] Ultimately, the joint loss function can be obtained as L=L intra +L align Using this joint loss function to train the reinforcement learning pre-trained model yields the trained reinforcement learning model.

[0098] The technical solution provided in this application enhances the adaptability of the reward model in the reinforcement learning model to the scoring context by explicitly introducing the review group context; at the same time, it models the intra-group accuracy and inter-group alignment goals, effectively reducing scoring conflicts in the training data. This method is more robust in non-deterministic and subjective tasks (such as content quality and value ranking), does not rely heavily on the accuracy of the reviewer's identity, and is effective even if the group identification is coarse.

[0099] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0100] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0101] Figure 7 This is a schematic diagram of a reinforcement learning model training device provided in an embodiment of this application. Figure 7 As shown, the device includes:

[0102] The acquisition module 701 is configured to acquire N sample data, each sample data including at least a question, a candidate answer to the question from a reinforcement learning pre-trained model, the evaluation of each candidate answer by the target review group, and the target review group identifier; the target review group is any one of the M review groups, where M and N are both positive integers.

[0103] The generation module 702 is configured to generate a preference context vector for each review group based on N sample data.

[0104] Module 703 is configured to build a reward model for a reinforcement learning pre-trained model based at least on the preference context vectors of each review panel.

[0105] Module 703 is also configured to construct a loss function based on a reward model. The loss function includes at least a between-group alignment loss term, which is used to characterize the differences in evaluations of the same sample data by different review groups.

[0106] Training module 704 is configured to train a pre-trained reinforcement learning model based on a loss function, resulting in a trained reinforcement learning model.

[0107] According to the technical solution provided in the embodiments of this application, N sample data are first obtained, including questions, candidate answers, evaluations of candidate answers by review groups, and review group identifiers. Then, a preference context vector for each review group is generated based on the N sample data. At least based on the preference context vector, a reward model for a reinforcement learning pre-trained model is constructed. Then, a loss function including at least an inter-group alignment loss term is constructed based on the reward model so that the loss function can characterize the evaluation differences of different review groups on the same sample data. Finally, the constructed loss function is used to train the reinforcement learning pre-trained model to obtain the trained reinforcement learning model. This can effectively solve the reward learning bias caused by inter-group differences in the reward model and improve the stability of reinforcement learning model training.

[0108] In some implementations, generating a preference context vector for each review group based on N sample data includes: obtaining all sample data of the target review group from the N sample data; performing token encoding on the sample data of the target review group to obtain the token encoding vector of the target review group; and determining the preference context vector of the target review group based on the token encoding vector.

[0109] In some implementations, the preference context vector of the target review group is determined based on the token encoding vector, including: embedding the token encoding vector to obtain the preference context vector of the target review group; or training the historical token encoding vector of the target review group to obtain the preference context vector of the target review group.

[0110] In some implementations, the reward model is constructed as follows: Candidate answers from the target sample data and the context representation of the target sample data are obtained; the target sample data can be any sample data; the candidate answers and the context representation are concatenated to obtain the token encoding of the concatenated text; the token encoding of the concatenated text is concatenated with the preference context vector of the review group corresponding to the target sample data to obtain the concatenated features; the concatenated features are used as the independent variable of a preset network prediction function to obtain the reward model for the target sample data.

[0111] In some implementations, the contextual representation of the target sample data is determined as follows: the evaluation windows for the previous t rounds of the target sample data are obtained, and the evaluation windows include the questions for each round, the candidate answers of the reinforcement learning pre-trained model to the questions, and the evaluations of the target review panel for each candidate answer; all items in the evaluation windows for the previous t rounds are concatenated to obtain the contextual representation of the target sample data.

[0112] In some implementations, the inter-group alignment loss term is determined as follows: a first reward value and a second reward value for the target sample data are calculated respectively; wherein the first reward value and the second reward value correspond to the same candidate answer and the context representation of the target sample data, and correspond to the preference context vectors of different review groups; the difference between the first reward value and the second reward value is calculated using the second normal form to obtain the candidate inter-group alignment loss term; all reward values ​​in the target sample data that correspond to the same candidate answer and the context representation of the target sample data, and correspond to the preference context vectors of different review groups are traversed, and the sum of all candidate inter-group alignment loss terms is determined as the inter-group alignment loss term.

[0113] In some implementations, the loss function further includes a intra-group consistency loss term, which is used to characterize the consistency between different candidate answers to the same question. The intra-group consistency loss term is determined in the following ways: by using the intra-group normalized relative loss method; or by using a metric learning algorithm; or by using an intra-group relative policy optimization algorithm.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0115] Figure 8 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 8As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module / unit in the various device embodiments described above.

[0116] Electronic device 8 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.

[0117] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0118] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0121] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A reinforcement learning model training method, characterized in that, include: Obtain N sample data, each sample data includes at least a question, a candidate answer to the question from a reinforcement learning pre-trained model, the evaluation of each candidate answer by a target review group, and the target review group identifier; the target review group is any one of M review groups, where M and N are both positive integers; Based on the N sample data, generate a preference context vector for each review group; The reward model of the reinforcement learning pre-trained model is constructed based at least on the preference context vectors of each review group; A loss function is constructed based on the reward model. The loss function includes at least an inter-group alignment loss term, which is used to characterize the difference in evaluation of the same sample data by different review groups. The reinforcement learning pre-trained model is trained based on the loss function to obtain the trained reinforcement learning model. The reward model is constructed as follows: Candidate answers from the target sample data and the context representation of the target sample data are obtained; the target sample data can be any sample data; the candidate answers and the context representation are concatenated to obtain a token encoding of the concatenated text; the token encoding of the concatenated text is concatenated with the preference context vector of the review group corresponding to the target sample data to obtain concatenated features; the concatenated features are used as the independent variable of a preset network prediction function to obtain the reward model for the target sample data. The context representation of the target sample data is determined as follows: the evaluation windows of the target sample data in the previous t rounds are obtained, the evaluation windows include the questions in each round, the candidate answers of the reinforcement learning pre-trained model to the questions, and the evaluations of the target review group to each candidate answer; all items in the evaluation windows of the t rounds are concatenated to obtain the context representation of the target sample data.

2. The method according to claim 1, characterized in that, Based on the N sample data, a preference context vector is generated for each review group, including: From the N sample data, obtain all sample data of the target review group; The sample data of the target review group is tokenized to obtain the token encoding vector of the target review group. The preference context vector of the target review group is determined based on the token encoding vector.

3. The method according to claim 2, characterized in that, The step of determining the preference context vector of the target review group based on the token encoding vector includes: The token encoding vector is embedded to obtain the preference context vector of the target review group; or The historical token encoding vector of the target review group is trained to obtain the preference context vector of the target review group.

4. The method according to claim 1, characterized in that, The inter-group alignment loss term is determined in the following manner: Calculate the first reward value and the second reward value for the target sample data respectively; wherein the first reward value and the second reward value correspond to the same candidate answer and the context representation of the target sample data, and correspond to the preference context vectors of different review groups; The difference between the first reward value and the second reward value is calculated using the second normal form to obtain the alignment loss term between candidate groups; Traverse the target sample data to find all reward values ​​for the same candidate answer and the context representation of the target sample data, and for the preference context vectors of different review groups, and determine the sum of all candidate group alignment loss terms as the group alignment loss term.

5. The method according to claim 1, characterized in that, The loss function also includes an intra-group consistency loss term, which is used to characterize the consistency between different candidate answers to the same question. The intra-group consistency loss term is determined in the following manner: Determined using the within-group normalized relative loss method; or Determined using a metric learning algorithm; or The intra-group relative strategy optimization algorithm is used to determine this.

6. A reinforcement learning model training device, characterized in that, include: The acquisition module is configured to acquire N sample data, each sample data including at least a question, a candidate answer to the question from a reinforcement learning pre-trained model, the evaluation of each candidate answer by a target review group, and the target review group identifier; the target review group is any one of M review groups, where M and N are both positive integers. The generation module is configured to generate a preference context vector for each review group based on the N sample data; The building module is configured to construct the reward model of the reinforcement learning pre-trained model based at least on the preference context vectors of each review group; The building module is also configured to construct a loss function based on the reward model, the loss function including at least an inter-group alignment loss term, which is used to characterize the difference in evaluation of the same sample data by different review groups. The training module is configured to train the reinforcement learning pre-trained model based on the loss function to obtain the trained reinforcement learning model. The reward model is constructed as follows: Candidate answers from the target sample data and the context representation of the target sample data are obtained; the target sample data can be any sample data; the candidate answers and the context representation are concatenated to obtain a token encoding of the concatenated text; the token encoding of the concatenated text is concatenated with the preference context vector of the review group corresponding to the target sample data to obtain concatenated features; the concatenated features are used as the independent variable of a preset network prediction function to obtain the reward model for the target sample data. The context representation of the target sample data is determined as follows: the evaluation windows of the target sample data in the previous t rounds are obtained, the evaluation windows include the questions in each round, the candidate answers of the reinforcement learning pre-trained model to the questions, and the evaluations of the target review group to each candidate answer; all items in the evaluation windows of the t rounds are concatenated to obtain the context representation of the target sample data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images