Generative model training method and system based on reward sorting
Through the generative model training method of reward sorting, triple data sets are constructed and iteratively trained, which solves the problem of large and unstable calculation overhead during the generative model adjustment process. The generation results are in line with human preferences and are suitable for any black box generation model.
Patent Information
- Application Number
- CN202510460982.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-08-15
AI Technical Summary
In the prior art, the calculation overhead of the generative model is large and unstable during the adjustment process, and the image generation task requires multiple adaptations, resulting in increased complexity and poor results, and there are biased problems in the model generation results.
Through a reward sorting method, a triple data set of prompt samples is constructed, high-quality samples are selected using reward information, iterative training and supervised fine-tuning, separate sample generation and model training, and reduce the amount of calculation.
It realizes stable and efficient generative model adjustment, the generation results are in line with human preferences, reduces computing overhead and memory requirements, and is suitable for any black box generation model.
Smart Images
Figure CN120495464A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a generative model training method and system based on reward ranking. Background Art
[0002] Basic generative models, such as large language models and divergence models, have demonstrated powerful generative capabilities in natural language processing and computer vision. However, the knowledge base of generative models is pre-trained on large amounts of text or other modal data from the internet, and their knowledge may contain biases, stereotypes, and inequalities inherent in human society and culture.
[0003] However, the reinforcement learning-based method in related technologies is less efficient in adjusting the generative model and requires a large number of samples for repeated attempts, resulting in high computational overhead and instability in the adjustment process. Image generation methods are used in related technologies, but when currently processing image generation tasks, many adaptations of the algorithm are required to achieve human-computer interaction, which increases the complexity of the method and the effect does not reach the best level.
[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention
[0005] The main purpose of the embodiments of the present application is to propose a generative model training method and system based on reward ranking, which can separate sample generation and model training. Sample generation does not require gradient calculation and reduces the computational complexity of model training.
[0006] To achieve the above objectives, one aspect of an embodiment of the present application proposes a generative model training method based on reward ranking, the method comprising:
[0007] Generate responses to the target dataset based on the current image-text generation model and construct a triplet data set of prompt samples;
[0008] Based on the reward information of the prompt text, the triple data set of the prompt sample is selected to obtain the triple data set of the target prompt sample;
[0009] The current image-text generation model is iteratively trained and supervised fine-tuned according to the triple data set of the target prompt sample until a preset iteration termination condition is met, thereby obtaining an image-text generation model that meets human preferences.
[0010] In some embodiments, generating a response to the target dataset based on the current image-text generation model to construct a triplet data set of prompt samples includes:
[0011] Acquire the target data set and sample it according to a preset batch to obtain a prompt text data set;
[0012] Based on a preset temperature parameter, performing a generation response calculation on the prompt text dataset using the current image-text generation model to obtain a responsiveness value of the prompt text;
[0013] performing an associated reward calculation on the prompt text dataset and the responsiveness value of the prompt text according to a preset reward function to obtain reward information of the prompt text;
[0014] The prompt text data set, the responsiveness value of the prompt text and the reward information of the prompt text are combined to construct a triple data set of the prompt sample.
[0015] In some embodiments, the expression for generating the response calculation is specifically as follows:
[0016]
[0017] In the above formula, g represents the current image-text generation model, w t-1 Represents the parameters of the current image-text generation model, α represents a hyperparameter, i.e., the preset temperature parameter, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response, represents the result of probability distribution p after adjustment by temperature parameter α, x i Indicates the prompt text.
[0018] In some embodiments, the expression of the preset reward function is specifically as follows:
[0019] r i (x i ,y i )=MLP(Encoder([x i ;y i ]))
[0020] In the above formula, Encoder(·) represents the transformer encoder, MLP(·) represents the multi-layer perceptron model, and r i (·) represents the reward function, x i Indicates the prompt text, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response.
[0021] In some embodiments, the reward information based on the prompt text selects the triple data set of the prompt sample to obtain the triple data set of the target prompt sample, including:
[0022] Sorting the triplet data set of the prompt samples in descending order according to the reward information of the prompt text to obtain a sorted triplet data set of the prompt samples;
[0023] Based on the sorted triple data set of prompt samples, triple data sets of the first several prompt samples are selected according to a preset proportion coefficient to obtain the triple data set of the target prompt sample.
[0024] In some embodiments, the expression for selecting the triple data set of the prompt sample is specifically as follows:
[0025]
[0026] In the above formula, |B| represents the number of triplet data sets of target prompt samples, T represents the number of triplet data sets of prompt samples, and k represents an adjustable parameter, that is, a preset proportional coefficient.
[0027] In some embodiments, the iterative training and supervised fine-tuning of the current image-text generation model based on the triplet data set of the target prompt sample until a preset iteration termination condition is met to obtain an image-text generation model that meets human preferences includes:
[0028] According to a preset objective function, the current image-text generation model is trained using the triplet data set of the target prompt sample to obtain a target loss value;
[0029] Performing supervised fine-tuning on the current image-text generation model according to the target loss value to obtain a fine-tuned image-text generation model;
[0030] The steps of selecting the triplet data set of the prompt sample, training the model, and fine-tuning the supervised model are repeated until a preset number of iterations is met or the reward information of the prompt text becomes stable, and a picture-text generation model that meets human preferences is output.
[0031] In some embodiments, the expression of the preset objective function is specifically as follows:
[0032]
[0033] In the above formula, L(·) represents the objective function, Indicates that in the image-text generation model g(w t-1 ), the probability of generating output y given input x, β represents the weight of the regularization term, Represents the measurement of the current image-text generation model distribution g(w t-1) and the difference between the initial image-text generation model distribution g(w0), B represents the triple data set of the target prompt sample.
[0034] In some embodiments, the expression for supervised fine-tuning of the current image-text generation model is specifically as follows:
[0035]
[0036]
[0037]
[0038] In the above formula, v t and s t denote the estimation of the first-order moment and the second-order moment respectively, β1 and β2 denote the decay rate, ∈ denotes a small constant to avoid the denominator being zero, α denotes the learning rate, L(·) denotes the objective function, and w t-1 Represents the parameters of the current image-text generation model, w t Represents the parameters of the fine-tuned image-text generation model.
[0039] To achieve the above objectives, another aspect of the present application provides a generative model training system based on reward ranking, the system comprising:
[0040] The first module is used to generate responses to the target dataset based on the current image-text generation model and construct a triplet data set of prompt samples;
[0041] The second module is used to select the triple data set of the prompt sample based on the reward information of the prompt text to obtain the triple data set of the target prompt sample;
[0042] The third module is used to iteratively train the current image-text generation model based on the triple data set of the target prompt sample and perform supervised fine-tuning until the preset iteration termination condition is met to obtain an image-text generation model that meets human preferences.
[0043] The embodiments of the present application include at least the following beneficial effects: The present application provides a generative model training method and system based on reward sorting, which generates responses to the target data set based on the current image-text generation model, constructs a triple data set of prompt samples, further selects the triple data set of prompt samples based on the reward information of the prompt text, and obtains the triple data set of the target prompt sample. Through reward-driven sample quality evaluation, stable supervised training is provided. Finally, the current image-text generation model is iteratively trained and supervised fine-tuned based on the triple data set of the target prompt sample, separating sample generation and model training. Sample generation does not require gradient calculation, reducing the amount of calculation. Training only loads the model, reducing memory requirements, making it scalable to any black box generation model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a flowchart of a generative model training method based on reward ranking provided in an embodiment of the present application;
[0045] Figure 2 Schematic diagram of a reward ranking-based generative model training system provided in an embodiment of the present application;
[0046] Figure 3 It is a flowchart of the generative model training provided in the embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.
[0048] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0049] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.
[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0051] First, it's important to note that during pre-training, the model absorbs and learns patterns and associations within this data. To ensure that the model's generation behavior aligns with human ethics and preferences, it's crucial to adjust the model's values. Currently, reinforcement learning methods are not stable or efficient enough for language model tuning; image generation also requires various adaptations.
[0052] In related technologies, for example, a reinforcement learning method based on human feedback is used to train language models to follow a wide range of written instructions. The method uses a pre-trained language model (such as GPT-3) and a group of trained human annotators to train the model in three steps. The first step is to collect demonstration data and train a supervision strategy, where the annotators provide demonstrations of the desired behavior on the input prompt distribution. The second step is to use reinforcement learning to fine-tune the model from human feedback, and the annotators provide comparisons of the model outputs. The third step is to use unsupervised learning to improve the generalization ability of the model by clustering the model output to determine whether it conforms to the expected behavior.
[0053] However, there are some drawbacks. For example, the reinforcement learning-based approach is inefficient in adjusting the generative model and requires a large number of samples for repeated trial and error, resulting in high computational overhead and instability in the adjustment process. This problem arises because the approach also uses reinforcement learning.
[0054] Existing methods use fine-tuning CLIP models to better predict human choices in generated images, thereby improving the aesthetic quality and consistency of generated images with human intent. Stable Diffusion is used as a generative model, which treats the generation process as the inverse of the diffusion process. This allows the diffusion model to generate images with better aesthetic quality and more consistent with human intent. This utilizes image generation technology, but current approaches to image generation require extensive algorithm adaptation to achieve human-computer interaction, increasing the complexity of the method and resulting in suboptimal results.
[0055] In summary, basic generative models, such as large language models and divergence models, demonstrate powerful generative capabilities in natural language processing and computer vision tasks. These models benefit from pre-training on massive amounts of unlabeled data and can generate high-quality text and images. However, due to the potential biases in the training data, the models can also learn these biases, leading to inaccurate and unfair generation results. Therefore, it is crucial to adjust the value of the model to ensure that the generated results conform to human ethics and preferences. Currently, reinforcement learning is mainly used to adjust language models, but the algorithm is unstable and inefficient. Image generation also requires numerous adaptations to achieve human-computer interaction. Therefore, a robust and efficient solution is needed to adjust different types of generative models.
[0056] In view of this, the embodiments of the present application provide a generative model training method based on reward ranking. By designing an adjustment algorithm based on sample quality, a reward function is used to select high-quality samples and supervise the adjustment of the model. Compared with reinforcement learning methods, this model of supervised training is more stable and has lower computational overhead. Since sample generation and model adjustment are separated, the algorithm can also be extended to any black-box generative model. Through this new idea, a universal generative model adjustment scheme is proposed, so that its generated results are more in line with human preferences.
[0057] Reference Figure 1 , Figure 1 A flowchart of a generative model training method based on reward ranking provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:
[0058] S100, generating a response to the target dataset based on the current image-text generation model, and constructing a triplet data set of prompt samples;
[0059] It should be noted that, in some embodiments, step S100 may include:
[0060] S110, obtaining a target data set and sampling it according to a preset batch to obtain a prompt text data set;
[0061] In this embodiment, the training prompt set, ie, the target data set X = {x1, x2, ..., x n} to sample a batch of data sets D t , the dataset size is b.
[0062] S120: Based on the preset temperature parameter, a generation response calculation is performed on the prompt text dataset using the current image-text generation model to obtain a responsiveness value of the prompt text;
[0063] In this embodiment, D t Each hint x in i, using the current generative model g to generate each prompt x with temperature parameter α i Corresponding response y i , whose expression is:
[0064]
[0065] In the above formula, g represents the current image-text generation model, w t-1 Represents the parameters of the current image-text generation model, α represents a hyperparameter, i.e., the preset temperature parameter, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response, represents the result of probability distribution p after adjustment by temperature parameter ɑ, x i Indicates the prompt text.
[0066] S130, performing an associated reward calculation on the prompt text dataset and the responsiveness value of the prompt text according to a preset reward function to obtain reward information for the prompt text;
[0067] In this embodiment, calculate each pair (x i ,y i ) reward r i (x i ,y i ), the calculation formula is as follows:
[0068] r i (x i ,y i )=MLP(Encoder([x i ;y i ]))
[0069] In the above formula, Encoder(·) represents the transformer encoder, MLP(·) represents the multi-layer perceptron model, and r i (·) represents the reward function, x i Indicates the prompt text, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response.
[0070] S140 , combining the prompt text dataset, the responsiveness value of the prompt text, and the reward information of the prompt text to construct a triplet data set of prompt samples.
[0071] In this embodiment, the data set D is obtained t The set of all prompt, response and reward triplets R(x i ,y i ,r i ).
[0072] S200, based on the reward information of the prompt text, selecting the triple data set of the prompt sample to obtain the triple data set of the target prompt sample;
[0073] It should be noted that, in some embodiments, step S200 may include:
[0074] S210, sorting the triplet data set of the prompt samples in descending order according to the reward information of the prompt text to obtain a sorted triplet data set of the prompt samples;
[0075] In this embodiment, the collected triples (x i ,y i ,r i ) According to the reward r i Sort from high to low to obtain a sorted triplet data set of prompt samples.
[0076] S220 , based on the sorted triple data set of prompt samples, select triple data sets of the first several prompt samples according to a preset ratio coefficient to obtain a triple data set of the target prompt sample.
[0077] In this embodiment, the top 1 / k samples with the highest reward are selected, denoted as B, and the number of high-reward samples retained is:
[0078]
[0079] In the above formula, |B| represents the number of triplet data sets of target prompt samples, T represents the number of triplet data sets of prompt samples, and k represents an adjustable parameter, that is, a preset proportional coefficient.
[0080] S300, iteratively training the current image-text generation model based on the triplet data set of the target prompt sample and performing supervised fine-tuning until a preset iteration termination condition is met, thereby obtaining an image-text generation model that meets human preferences;
[0081] It should be noted that, in some embodiments, step S300 may include:
[0082] S310: According to a preset objective function, the current image-text generation model is trained using a triplet data set of target prompt samples to obtain a target loss value;
[0083] In this embodiment, the obtained high reward sample set B(x,y) is used to generate the current model g(w t-1 ) to perform supervised fine-tuning so that the output of the generated model is more and more in line with human expectations, then the loss function L is defined as follows:
[0084]
[0085] In the above formula, L(·) represents the objective function, Indicates that in the image-text generation model g(w t-1 ), the probability of generating output y given input x, β represents the weight of the regularization term, Represents the measurement of the current image-text generation model distribution g(w t-1 ) and the difference between the initial image-text generation model distribution g(w0), B represents the triple data set of the target prompt sample.
[0086] S320, performing supervised fine-tuning on the current image-text generation model according to the target loss value to obtain a fine-tuned image-text generation model;
[0087] In this embodiment, the loss value L(w t-1 ), Adam is used as the optimization algorithm to iteratively update the model parameters to obtain a new generative model g(w t ), the process expression is:
[0088]
[0089]
[0090]
[0091] In the above formula, v t and s t denote the estimation of the first-order moment and the second-order moment respectively, β1 and β2 denote the decay rate, ∈ denotes a small constant to avoid the denominator being zero, α denotes the learning rate, L(·) denotes the objective function, and w t-1 Represents the parameters of the current image-text generation model, w t Represents the parameters of the fine-tuned image-text generation model.
[0092] S330, looping through the steps of selecting the triplet data set of prompt samples, training the model, and fine-tuning the supervised steps until the preset number of iterations is met or the reward information of the prompt text becomes stable, and outputting a picture-text generation model that meets human preferences.
[0093] In this embodiment, from the initial model g(w0,x i ) starts. Perform steps S100 to S300 to obtain a new generation model g(w i ,x), g(w i ,x) as the current model and perform the first step of the next round of adjustment. Repeat the above steps until the predetermined number of rounds T is reached or the reward function no longer significantly improves on the validation set. Finally, the generative model g(w i,x) can generate text and images that are consistent with human ethics and preferences, alleviating the biased knowledge problem of the original generation model.
[0094] In summary, if Figure 3 As shown, this embodiment of the present invention samples text prompts from a dataset, generates responses for each prompt using a generative model, and calculates the reward for each response. The generated prompt-response pairs are ranked according to their reward values, retaining the top k samples with the highest rewards. The high-reward samples are then used to supervise and fine-tune the current generative model to obtain a new generative model. This new model is then used as the current model, and the above steps are repeated, forming a closed-loop adjustment loop. To alleviate the biased knowledge problem associated with generative models as the knowledge foundation, this embodiment of the present invention provides a stable and efficient generative model adjustment scheme that gradually approximates the model generation distribution to the high-reward response distribution, ensuring that model generation behavior conforms to human ethics and preferences. Generated samples are evaluated for quality using a predefined reward function and filtered based on reward value to obtain a subset of high-quality samples. This mechanism achieves the goal of obtaining high-quality training samples without manual labeling, separating sample generation and model training into two independent steps. Sample generation does not require gradient calculation, reducing computational effort. Training only requires loading the model, reducing memory requirements. This separation of concerns improves the versatility of the framework, making it extensible to any black-box generative model.
[0095] See also Figure 2 The present application also provides a generative model training system based on reward ranking, which can implement the above-mentioned generative model training method based on reward ranking. The system includes:
[0096] The first module 201 is used to generate a response to the target dataset based on the current image-text generation model, and construct a triplet data set of prompt samples;
[0097] The second module 202 is used to select the triple data set of the prompt sample based on the reward information of the prompt text to obtain the triple data set of the target prompt sample;
[0098] The third module 203 is used to iteratively train the current image-text generation model based on the triplet data set of the target prompt sample and perform supervised fine-tuning until the preset iteration termination condition is met to obtain an image-text generation model that meets human preferences.
[0099] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0100] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.
Claims
1. A generative model training method based on reward ranking, characterized in that: The method comprises the following steps: Generate responses to the target dataset based on the current image-text generation model and construct a triplet data set of prompt samples; Based on the reward information of the prompt text, the triple data set of the prompt sample is selected to obtain the triple data set of the target prompt sample; The current image-text generation model is iteratively trained and supervised fine-tuned according to the triple data set of the target prompt sample until a preset iteration termination condition is met, thereby obtaining an image-text generation model that meets human preferences.
2. The method according to claim 1, characterized in that The generating response to the target dataset based on the current image-text generation model to construct a triplet data set of prompt samples includes: Acquire the target data set and sample it according to a preset batch to obtain a prompt text data set; Based on a preset temperature parameter, performing a generation response calculation on the prompt text dataset using the current image-text generation model to obtain a responsiveness value of the prompt text; performing an associated reward calculation on the prompt text dataset and the responsiveness value of the prompt text according to a preset reward function to obtain reward information of the prompt text; The prompt text data set, the responsiveness value of the prompt text and the reward information of the prompt text are combined to construct a triple data set of the prompt sample.
3. The method according to claim 2, characterized in that The expression for generating the response calculation is specifically as follows: In the above formula, g represents the current image-text generation model, w t-1 Represents the parameters of the current image-text generation model, α represents a hyperparameter, i.e., the preset temperature parameter, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response, represents the result of probability distribution p after adjustment by temperature parameter α, x i Indicates the prompt text.
4. The method according to claim 2, characterized in that The expression of the preset reward function is specifically as follows: r i (x i ,y i )=MLP(Encoder([x i ;y i ])) In the above formula, Encoder(·) represents the transformer encoder, MLP(·) represents the multi-layer perceptron model, and r i (·) represents the reward function, x i Indicates the prompt text, y i Indicates that the current image-text generation model is for the input prompt text x i The generated response.
5. The method according to claim 1, wherein The reward information based on the prompt text selects the triple data set of the prompt sample to obtain the triple data set of the target prompt sample, including: Sorting the triplet data set of the prompt samples in descending order according to the reward information of the prompt text to obtain a sorted triplet data set of the prompt samples; Based on the sorted triple data set of prompt samples, triple data sets of the first several prompt samples are selected according to a preset proportion coefficient to obtain the triple data set of the target prompt sample.
6. The method according to claim 5, characterized in that The expression for selecting the triple data set of the prompt sample is specifically as follows: In the above formula, |B| represents the number of triplet data sets of target prompt samples, T represents the number of triplet data sets of prompt samples, and k represents an adjustable parameter, that is, a preset proportional coefficient.
7. The method according to claim 1, characterized in that The iterative training and supervised fine-tuning of the current image-text generation model based on the triplet data set of the target prompt sample until a preset iteration termination condition is met to obtain an image-text generation model that meets human preferences includes: According to a preset objective function, the current image-text generation model is trained using the triplet data set of the target prompt sample to obtain a target loss value; Performing supervised fine-tuning on the current image-text generation model according to the target loss value to obtain a fine-tuned image-text generation model; The steps of selecting the triplet data set of the prompt sample, training the model, and fine-tuning the supervised model are repeated until a preset number of iterations is met or the reward information of the prompt text becomes stable, and a picture-text generation model that meets human preferences is output.
8. The method according to claim 7, characterized in that The expression of the preset objective function is specifically as follows: In the above formula, L(·) represents the objective function, Indicates that in the image-text generation model g(w t-1 ), the probability of generating output y given input x, β represents the weight of the regularization term, Represents the measurement of the current image-text generation model distribution g(w t-1 ) and the difference between the initial image-text generation model distribution g(w0), B represents the triple data set of the target prompt sample.
9. The method according to claim 7, characterized in that The expression for supervised fine-tuning of the current image-text generation model is specifically as follows: In the above formula, v t and s t denote the estimation of the first-order moment and the second-order moment respectively, β1 and β2 denote the decay rate, ∈ denotes a small constant to avoid the denominator being zero, α denotes the learning rate, L(·) denotes the objective function, and w t-1 Represents the parameters of the current image-text generation model, w t Represents the parameters of the fine-tuned image-text generation model.
10. A generative model training system based on reward ranking, characterized in that: The system comprises: The first module is used to generate responses to the target dataset based on the current image-text generation model and construct a triplet data set of prompt samples; The second module is used to select the triple data set of the prompt sample based on the reward information of the prompt text to obtain the triple data set of the target prompt sample; The third module is used to iteratively train the current image-text generation model based on the triple data set of the target prompt sample and perform supervised fine-tuning until the preset iteration termination condition is met to obtain an image-text generation model that meets human preferences.