Model training method and device, computer device, and storage medium

Abstract

The application discloses a model training method and device, computer equipment and a storage medium. The model training method comprises the following steps: obtaining answer content output by an initial generation model for a question in a question set; outputting a plurality of subjective score values of a plurality of subjective score dimensions of the answer content corresponding to each question by a preference score model; determining a preference data pair based on the each question, the answer content corresponding to the each question, and the subjective score values of the plurality of subjective score dimensions corresponding to the answer content; and training the initial generation model based on the preference data pair and by using a direct preference optimization algorithm to obtain a target generation model. The method can improve the accuracy of the generated content output by the trained generation model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a model training method, apparatus, computer device, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence (AI) technology, AI-based applications are becoming increasingly common. Currently, AI technology is frequently used for content generation. However, in scenarios where user subjectivity exists, such as text creation, the accuracy of AI-generated content is relatively low. Summary of the Invention

[0003] This application proposes a model training method, apparatus, computer equipment, and storage medium that can improve the accuracy of the generated content output by the trained generative model.

[0004] In a first aspect, embodiments of this application provide a model training method, the method comprising: obtaining the answer content output by an initial generation model for questions in a question set; using a preference scoring model to output multiple subjective scoring dimensions of the answer content corresponding to each question; determining preference data pairs based on each question, the answer content corresponding to each question, and the preference scores of the multiple subjective scoring dimensions corresponding to the answer content; and training the initial generation model based on the preference data pairs and using a direct preference optimization algorithm to obtain a target generation model.

[0005] Secondly, embodiments of this application provide a content acquisition module, a content scoring module, a data construction module, and a preference optimization module. The content acquisition module acquires the answer content output by an initial generation model for questions in a question set. The content scoring module uses a preference scoring model to output preference scores for multiple subjective scoring dimensions for the answer content corresponding to each question. The data construction module determines preference data pairs based on each question, the answer content corresponding to each question, and the preference scores for the multiple subjective scoring dimensions corresponding to the answer content. The preference optimization module trains the initial generation model based on the preference data pairs using a direct preference optimization algorithm to obtain a target generation model.

[0006] Thirdly, embodiments of this application provide a computer device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the model training method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be called by a processor to execute the model training method provided in the first aspect above.

[0008] The solution provided in this application obtains the answer content output by an initial generative model for questions in a question set; it then uses a preference scoring model to output preference scores for multiple subjective scoring dimensions for the answer content corresponding to each question; based on each question, the answer content corresponding to each question, and the preference scores for the multiple subjective scoring dimensions corresponding to the answer content, preference data pairs are determined; and based on the preference data pairs, the initial generative model is trained using a direct preference optimization algorithm to obtain a target generative model. Therefore, when determining the preference data pairs used to train the generative model, the answer content of the generative model for questions can be scored on multiple subjective scoring dimensions, and the preference data pairs are determined based on the scores of multiple subjective scoring dimensions. This ensures the accuracy of the preference data pairs, improves the accuracy of training the generative model using the determined preference data, and ultimately improves the accuracy of the generated content output by the trained generative model. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic flowchart of a model training method according to an embodiment of this application is shown.

[0011] Figure 2 A schematic flowchart of a model training method according to another embodiment of this application is shown.

[0012] Figure 3 A schematic flowchart of a model training method according to yet another embodiment of this application is shown.

[0013] Figure 4 A schematic flowchart of a model training method according to another embodiment of this application is shown.

[0014] Figure 5 A block diagram of a model training apparatus according to an embodiment of this application is shown.

[0015] Figure 6 This is a block diagram of a computer device for performing a model training method according to an embodiment of this application.

[0016] Figure 7 This is a storage unit in this application embodiment for storing or carrying program code that implements the model training method according to this application embodiment. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0018] With the rapid development of AI technology, Large Language Models (LLMs) have gradually moved from theoretical research to large-scale applications, attracting widespread attention, especially as deep learning-based language models have demonstrated outstanding capabilities in tasks such as text generation, language understanding, and conversion. The application of large models is particularly prominent in dialogue scenarios. Trained on large datasets, large models can generate highly coherent and natural dialogue content, significantly improving the user experience.

[0019] In dialogue scenarios, large language models are often used to implement Artificial Intelligence Generated Content (AIGC), thereby meeting users' needs in daily life and work. AIGC is a technology that automatically generates content through artificial intelligence techniques such as deep learning algorithms. It can understand natural language instructions and quickly generate high-quality, diverse, and creative content based on user needs.

[0020] In related technologies, Online Direct Preference Optimization (ODPO) algorithms are commonly used to train models for content generation. When training a model using the DPO algorithm, preference data pairs need to be constructed, and then these pairs are used to train the model. These preference data pairs are typically constructed through manual annotation or by using a scoring model to score the model's responses. However, manually annotating preference data pairs increases the model's training cost; and when constructing preference data pairs using a scoring model, in some use cases, such as creative writing, poetry writing, or generating holiday greetings, users have subjective feelings and aesthetic judgments about the content generated by the model. Therefore, the scoring model's scores are not accurate, resulting in insufficient accuracy of the constructed preference data pairs. Consequently, the generative model trained using these preference data pairs struggles to accurately capture this subjectivity and diversity, leading to insufficient accuracy in the generated content.

[0021] To address the aforementioned problems, the inventors have proposed a model training method, apparatus, computer device, and storage medium as provided in the embodiments of this application. These methods enable the determination of preference data pairs for training the generative model. Specifically, they allow for scoring the generative model's responses to questions across multiple subjective scoring dimensions, and the preference data pairs are determined based on these scores. This ensures the accuracy of the preference data pairs, improves the accuracy of training the generative model using the determined preference data, and ultimately enhances the accuracy of the generated content output by the trained model. The specific model training method will be described in detail in subsequent embodiments.

[0022] The model training method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0023] Please see Figure 1 , Figure 1 A schematic flowchart of a model training method provided in one embodiment of this application is shown. The following will focus on... Figure 1 The process shown will be explained in detail. The model training method may specifically include the following steps:

[0024] Step S110: Obtain the answer content output by the initial generation model for the questions in the question set.

[0025] In this embodiment of the application, when training the model for content generation, an initial generation model can output answers to questions in a query set. The initial generation model can be a large language model; the content generation scenario can be one where users have subjective feelings, evaluations, or aesthetic judgments about the content output by the generation model, such as a content creation scenario.

[0026] In some implementations, the above set of questions may include multiple questions, so that each question can be answered by an initial generative model, thereby enabling the construction of preference data pairs corresponding to multiple questions, and then using multiple preference data pairs to train a generative model.

[0027] In some implementations, supervised fine-tuning (SFT) can be used to train an initial model based on a sample dataset, resulting in an initial generative model capable of outputting content according to the input question. SFT refers to fine-tuning a pre-trained model using high-quality labeled data to adapt it to a specific task or domain. This process involves creating a new target model, copying all parameters of the source model (pre-trained model) except for the output layer, and adding a new output layer to the target model whose parameters are trained using labeled data.

[0028] In some implementations, the above-mentioned question set can be obtained for the use case targeted by the model to be trained. This can be achieved by acquiring questions related to the above use case from multiple different data sources and constructing the question set; alternatively, it can be based on user input to obtain questions related to the above use case and form the question set, meaning the question set can also be manually created for the above use case; or it can be generated using a large language model for the above use case. Of course, the specific method for obtaining the above question set is not limited.

[0029] Step S120: Output the preference scores for multiple subjective rating dimensions for the answer content corresponding to each question using the preference rating model.

[0030] In this embodiment, after obtaining the initial generation model's output of the answers to the above question set, a preference rating model can be used to score the answer content for each question across multiple subjective rating dimensions, thereby obtaining preference scores for each answer content across multiple subjective rating dimensions. These subjective rating dimensions can be dimensions reflecting the user's subjective preferences for the model's answer content. For example, in the scenario of generating holiday greetings, subjective rating dimensions could include: relevance, accuracy, avoidance of templates, timeliness, practicality, and reading experience.

[0031] For example, the above subjective rating dimensions include dimension 1, dimension 2, dimension 3 and dimension 4. Then, for answer content A of question 1, the preference score of answer content A may include: the score of dimension 1, the score of dimension 2, the score of dimension 3 and the score of dimension 4.

[0032] In some implementations, the preference scoring model described above can be a pre-trained model used to score the preferences of the responses output by the generated model. For example, it could be a model obtained by fine-tuning a large language model for scenarios involving evaluating preference scores of responses. Of course, the specific type of the preference scoring model described above is not limited.

[0033] Step S130: Determine preference data pairs based on each question, the corresponding answer content for each question, and the preference scores for multiple subjective rating dimensions corresponding to the answer content.

[0034] In this embodiment of the application, after obtaining the preference scores of multiple subjective rating dimensions for each answer, preference data pairs can be constructed based on each question in the above question set, the answer content corresponding to each question, and the preference scores of multiple subjective rating dimensions corresponding to the answer content. These pairs are used to train the initial generative model.

[0035] In some implementations, when constructing preference data pairs, the preferred and unpreferred answers to the questions can be determined by referring to the preference scores of multiple subjective rating dimensions corresponding to the answers, thereby identifying the preference data pairs. Understandably, since the above question set contains multiple questions, multiple preference data pairs can be obtained corresponding to multiple questions.

[0036] Step S140: Based on the preference data pair, the initial generative model is trained using the direct preference optimization algorithm to obtain the target generative model.

[0037] In this embodiment, after obtaining the above preference data pairs, the initial generative model can be trained using the Direct Preference Optimization (DPO) algorithm based on the preference data pairs to obtain the target generative model. The core idea of ​​the DPO algorithm is to directly optimize the model parameters using human preference data without explicitly training the reward model. It can directly correlate the probability of human preferences with the output probability of the language model. Specifically, the DPO algorithm optimizes the model by maximizing the difference between the log probability of the preference output and minimizing the log probability of the non-preference output.

[0038] In some implementations, when training the initial generative model using preference data pairs, the DPO algorithm can be used to update the model parameters of the initial generative model, and this process of updating the model parameters can be repeated. The DPO algorithm's parameter updating process primarily relies on the provided preference data pairs to optimize the objective function so that the model's output better aligns with human preferences. During the repeated parameter updates, the initial generative model can undergo automated subjective evaluation for various problems to identify its weaknesses and facilitate DPO preference-aligned training.

[0039] The model training method provided in this application embodiment can score the generator model's answers to questions on multiple subjective rating dimensions when determining the preference data pairs used to train the generator model, and determine the preference data pairs based on the scores of multiple subjective rating dimensions, thereby ensuring the accuracy of the preference data pairs, improving the accuracy of training the generator model using the determined preference data, and thus improving the accuracy of the generated content output by the trained generator model.

[0040] Please see Figure 2 , Figure 2 A schematic flowchart of a model training method according to another embodiment of this application is shown. This model training method is applied to the aforementioned computer device, and will be discussed below. Figure 2 The process shown will be explained in detail. The model training method may specifically include the following steps:

[0041] Step S210: Obtain the answer content output by the initial generation model for the questions in the question set.

[0042] In this embodiment, step S210 can be referred to the content of other embodiments, and will not be repeated here.

[0043] Step S220: Input the first prompt information and the answer content into the pre-trained preference rating model to obtain the preference scores of multiple subjective rating dimensions output by the preference rating model. The first prompt information is used to prompt the preference rating model to score the answer content from the multiple subjective rating dimensions.

[0044] In this embodiment of the application, when determining the preference scores of multiple subjective rating dimensions corresponding to each answer content, the preference rating model pre-trained with the first prompt information and the above answer content input values ​​can be used to obtain the preference scores of multiple subjective rating dimensions output by the preference rating model.

[0045] In this context, the preference rating model can be a large language model, and the first prompt is a prompt used to guide the large language model. The prompt can be a piece of text or data specifically designed to guide the large language model in the rating task, and can be provided within the context of the large model (such as a deep learning model, natural language processing model, etc.). The prompt aims to activate the relevant knowledge and capabilities in the model, enabling it to respond to or complete the task in the expected way. Prompts are widely used in large language models, especially in the field of Natural Language Processing (NLP).

[0046] In some implementations, the computer device can obtain the first prompt information above to prompt the preference rating model to perform preference ratings on multiple subjective rating dimensions for each of the above answers.

[0047] In one possible implementation, subjective rating dimensions may include relevance, accuracy, timeliness, usability, and reading experience. The definitions of each subjective rating dimension can be as follows:

[0048] Relevance: Ensure that the content of the greetings is closely related to the specific festival, accurately convey the sentiments of the greetings that match the festive atmosphere, and enable the recipient to deeply feel the unique charm of the festival.

[0049] Accuracy: The cultural knowledge related to the festival should be expressed accurately and without error, avoiding misinterpretations or misleading information, ensuring the rigor and correctness of the blessings at the cultural level, and also meeting any special requests from users.

[0050] Avoid templates: Show unique ingenuity in the conception, expression, and language of blessings, break through the fixed pattern of traditional festival blessings, and cleverly integrate fun and creativity to bring a refreshing experience to the transmission of blessings.

[0051] Timeliness: Closely related to current festivals, accurately reflecting their core characteristics, while ensuring that the blessings not only contain the essence of traditional festivals, but also fit people's current lifestyles and communication styles, highlighting the spirit of the times.

[0052] Practicality: Through warm and sincere words of blessing, it creates a pleasant emotional experience for the recipient, effectively enhances mutual feelings, conveys warmth and care in the festive atmosphere, and plays a good role in emotional communication and maintenance.

[0053] Reading experience: Ensure that the answers are concise, clear, and sincere, both universally applicable and in line with the festive atmosphere, positive and uplifting, and contain personalized elements that fit the festival, so that readers can feel warmth and beauty as soon as they read them.

[0054] Thus, in the process of scoring the preferences of the answers, the preference frequency model can be used to comprehensively and meticulously evaluate the answers to each question from key creative dimensions such as Relevance, Accuracy, Template Utilization, Timeliness, Helpfulness, and Reading Experience. This multi-dimensional scoring mechanism reflects the quality of the answers in a comprehensive and three-dimensional way, providing more accurate and richer training signals for the subsequent DPO algorithm training, and helping the model achieve a qualitative leap in content generation.

[0055] Of course, the above subjective rating dimensions are merely illustrative. In addition to these, subjective rating dimensions can be expanded to include Creativity, Consistency, Interactivity, Sentiment Analysis, and Cultural Sensitivity. Specifically, Creativity is defined as assessing the novelty of the answer in terms of content and expression; Consistency is defined as judging whether the answer aligns with the user's historical behavior and preferences; Interactivity is defined as measuring whether the answer effectively elicits further user interaction and feedback; Sentiment Analysis is defined as analyzing the emotional tone of the answer, such as positive, negative, or neutral, to ensure the appropriateness of emotional expression; and Cultural Sensitivity is defined as examining whether the answer conforms to the norms and customs of different cultural backgrounds to avoid cultural conflicts.

[0056] In one possible implementation, considering that users' subjective feelings, subjective evaluations, and subjective aesthetic judgments of the model's answers differ in different usage scenarios, the computer device can determine multiple subjective rating dimensions corresponding to the current business scenario in the content creation scenario; and generate the first prompt information based on the multiple subjective rating dimensions.

[0057] In one possible implementation, when inputting the first prompt information and the answer content into a pre-trained preference rating model, the answer content can be added to the first prompt information so that the first prompt information carries the answer content. Thus, the preference rating model can perform preference rating on the answer content based on the relevant prompt words in the first prompt information and the carried answer content, according to multiple subjective rating dimensions.

[0058] Optionally, the first prompt information above may include the definition of each subjective rating dimension. This definition can be regarded as the rating standard of the subjective rating dimension, so that the preference rating model can better understand each subjective rating dimension and thus accurately rate the above answer from multiple subjective rating dimensions.

[0059] Optionally, the definition of the same subjective rating dimension can differ across different use cases. For example, regarding the rating dimension of "relevance," in the scenario of generating a presentation (PowerPoint, PPT) outline, relevance can be defined as follows: the outline closely adheres to the user-specified theme, comprehensively and meticulously covering all key points under that theme; its logical structure is highly consistent with the user's expected PPT presentation style; from the opening introduction and content development to the concluding summary, each part precisely matches the user's needs without any deviation. In the scenario of generating holiday greetings, relevance can be defined as follows: the greeting content closely revolves around the user-specified greeting content, accurately conveying the corresponding emotional sentiment, allowing the recipient to deeply feel the unique charm of the greeting. The definitions of the above subjective rating dimensions can be generated by obtaining the content defined by the user for these subjective rating dimensions, or they can be generated through a large language model. The specific method for obtaining the definitions of the above subjective rating dimensions is not limited.

[0060] Step S230: Determine preference data pairs based on each question, the corresponding answer content for each question, and the preference scores for multiple subjective rating dimensions corresponding to the answer content.

[0061] Step S240: Based on the preference data pair, the initial generative model is trained using the direct preference optimization algorithm to obtain the target generative model.

[0062] In the embodiments of this application, steps S230 and S240 can be referred to the contents of other embodiments, and will not be repeated here.

[0063] The model training method provided in this application embodiment can, when determining the preference data pairs used to train the generative model, input the prompt information used to prompt the model to score the answer content from multiple subjective scoring dimensions and the answer content into the preference scoring model. This enables the generative model to score the answer content of the question from multiple subjective scoring dimensions, and determines the preference data pairs based on the scores from multiple subjective scoring dimensions. This ensures the accuracy of the preference data pairs, improves the accuracy of training the generative model using the determined preference data, and further improves the accuracy of the generated content output by the trained generative model.

[0064] Please see Figure 3 , Figure 3 A schematic flowchart of a model training method according to another embodiment of this application is shown. This model training method is applied to the aforementioned computer device, and will be discussed below. Figure 3 The process shown will be explained in detail. The model training method may specifically include the following steps:

[0065] Step S310: Obtain the answer content output by the initial generation model for the questions in the question set.

[0066] Step S320: Output the preference scores for multiple subjective rating dimensions for the answer content corresponding to each question using the preference rating model.

[0067] In the embodiments of this application, steps S310 and S320 can be referred to the contents of other embodiments, and will not be repeated here.

[0068] Step S330: For each question, based on the preference scores of multiple subjective rating dimensions corresponding to the answer content, determine the preferred content and non-preferred answer content for each question.

[0069] In this embodiment of the application, when constructing preference data pairs based on each question in the above question set, the corresponding answer content for each question, and the preference scores of multiple subjective rating dimensions corresponding to the answer content, the preferred and non-preferred answer content for each question can be determined based on the preference scores of the multiple subjective rating dimensions corresponding to the answer content. Here, preferred answer content can be understood as answer content that is preferred by the user, and non-preferred answer content, the opposite of preferred answer content, can be understood as answer content that is not preferred by the user.

[0070] In some implementations, when the initial generation model outputs answers for each of the above questions, it can output multiple candidate answers, meaning that the answer content for each question includes multiple candidate answers. When determining the preferred and unpreferred answers for each question, a total preference score for each candidate answer can be determined based on the preference scores of multiple subjective rating dimensions corresponding to the answer content. Candidate answers with a total preference score greater than a first score are selected as preferred answers for each question; candidate answers with a total preference score less than a second score are selected as unpreferred answers for each question, where the second score is less than the first score. The specific values ​​of the first and second scores can be determined based on the range of the total preference score. For example, if the maximum total preference score is 30, then the first score could be 24 and the second score could be 20.

[0071] In one possible implementation, when determining the total preference score for each candidate answer content corresponding to each question based on the preference scores of multiple subjective rating dimensions corresponding to the answer content, the sum of the preference scores of multiple subjective rating dimensions corresponding to each candidate answer content can be obtained as the total preference score for each candidate answer content.

[0072] For example, subjective rating dimensions may include: relevance, accuracy, avoidance of templates, timeliness, usability, and reading experience. The preference score for each subjective rating dimension ranges from 0 to 5. For candidate answer content B of question 1, the obtained preference scores are: "relevance" 4, "accuracy" 4, "avoidance of templates" 3, "timeliness" 2, "usability" 3, and "reading experience" 3. Therefore, the total preference score can be: 4 + 4 + 3 + 2 + 3 + 3 = 19.

[0073] In one possible implementation, when determining the total preference score of each candidate answer for each question based on the preference scores of multiple subjective rating dimensions corresponding to the answer for each question, the total preference score of each candidate answer can also be obtained by weighted summation of the preference scores of multiple subjective rating dimensions corresponding to each candidate answer based on the weights of each subjective rating dimension.

[0074] For example, subjective rating dimensions may include: relevance, accuracy, avoidance of templates, timeliness, usability, and reading experience. The preference score for each subjective rating dimension ranges from 0 to 5. For candidate answer content B of question 1, the obtained preference score for "relevance" is 4, with a weight of 0.15; the preference score for "accuracy" is 4, with a weight of 0.25; the preference score for "avoidance of templates" is 3, with a weight of 0.1; the preference score for "timeliness" is 2, with a weight of 0.2; the preference score for "usability" is 3, with a weight of 0.2; and the preference score for "reading experience" is 3, with a weight of 0.1. Therefore, the total preference score can be: 4*0.15 + 4*0.25 + 3*0.1 + 2*0.2 + 3*0.2 + 3*0.1 = 3.2.

[0075] Optionally, the weights of each of the above subjective rating dimensions can be determined based on the degree of influence each subjective rating dimension has on the overall quality of the answer. The weight of a subjective rating dimension can be positively correlated with its influence on the overall quality of the answer; that is, the greater the influence of a subjective rating dimension, the greater its weight, and vice versa. For example, if the subjective rating dimensions include relevance, accuracy, avoidance of templates, timeliness, usability, and reading experience, relevance is the most important because the answer must closely address the core of the question to truly meet the user's needs. While timeliness is also crucial, its importance may be relatively lower in some cases, especially when answering non-timely questions.

[0076] Step S340: Based on each question and the corresponding preferred and non-preferred answers, determine the preference data pairs.

[0077] In this embodiment of the application, after obtaining the preferred and non-preferred answers for each question, a preference data pair can be determined based on each question and the preferred and non-preferred answers for each question. The determined preference data pair may include the question, the preferred answer, and the non-preferred answer.

[0078] In some implementations, preference data pairs can be provided in the form of triplets, i.e. (query, chosen, rejected). This data format can clearly express human preferences for different outputs, where query is the input question, chosen is the preferred answer, and rejected is the non-preferred answer.

[0079] Step S350: Based on the preference data pair, the initial generative model is trained using the direct preference optimization algorithm to obtain the target generative model.

[0080] In some implementations, when training the initial generative model based on preference data pairs and using the direct preference optimization algorithm, the question input values ​​from the preference data pairs can be used to initialize the generative model, obtaining the output of the initial generative model. Based on the output and the preferred and non-preferred answers from the preference data pairs, a loss value is determined. Then, the model parameters of the initial generative model are adjusted according to the determined loss value. The process is repeated until the training termination condition is met, resulting in the trained target generative model. The loss value can be calculated based on the binary cross-entropy function, reflecting the degree of consistency between the model's generated output and human preferences.

[0081] The process involves inputting the questions from each preference data pair into the initial generative model, obtaining the output for each question, determining the loss value based on the output for each question and the preferred and non-preferred responses for each preference data pair, and then adjusting the model parameters of the initial generative model based on the determined loss value. This completes one epoch. The process then repeats, inputting the questions from the preference data pair into the initial generative model again and obtaining the output for the next epoch. This process is repeated to complete multiple epochs. An epoch refers to the number of times all preference data pairs are used; one epoch is equivalent to training once using all preference data pairs.

[0082] In some implementations, the Adam optimizer can be used to iteratively update the initial generative model based on the loss value, reducing the loss value with each iteration until the total loss value converges. The model at this point is then saved, resulting in the trained target generative model. The Adam optimizer combines the advantages of AdaGra (Adaptive Gradient) and RMSProp optimization algorithms, comprehensively considering both the first-moment estimation (mean of the gradient) and the second-moment estimation (uncentered variance of the gradient) to calculate the update step size. The training termination conditions for iterative training can include: reaching the target number of iterations; or the loss value meeting a set condition.

[0083] Optionally, the convergence condition is to minimize the loss value, using an initial learning rate of 1e-3, with the learning rate cosine decaying with the number of steps, and a batch size of 512. After training for several epochs, convergence is considered complete. Here, batch size can be understood as a batch processing parameter, and its limit is the total number of preferred data pairs.

[0084] Optionally, satisfying the set condition for the total loss value may include: the total loss value being less than a set threshold. Of course, the specific set condition is not a limitation.

[0085] The model training method provided in this application embodiment can, when determining the preference data pairs used to train the generative model, score the generative model's answers to questions across multiple subjective rating dimensions, determine the preferred and non-preferred answer content for each question based on the scores across multiple subjective rating dimensions, and then construct preference data pairs based on the determined preferred and non-preferred answer content. This ensures the accuracy of the preference data pairs, improves the accuracy of training the generative model using the determined preference data, and ultimately improves the accuracy of the generated content output by the trained generative model.

[0086] Please see Figure 4 , Figure 4 A schematic flowchart of a model training method provided in another embodiment of this application is shown. This model training method is applied to the aforementioned computer device, and will be discussed below. Figure 4 The process shown will be explained in detail. The model training method may specifically include the following steps:

[0087] Step S410: Expand the initial problem set to obtain the problem set.

[0088] In this embodiment, considering that obtaining the question set typically relies on a predefined set of prompts, which cannot cover all possible scenarios and task types, and due to the fixed and finite nature of queries, the data and scenarios encountered by the model during training are relatively limited. This may lead to insufficient generalization ability of the model when faced with new or unseen prompts or scenarios. Furthermore, since the model is mainly optimized for these fixed questions during training, it may rely too heavily on these specific prompt patterns when generating answers, lacking sufficient flexibility and adaptability, thus affecting its performance and applicability in practical applications. Therefore, when obtaining the question set, the initial question set can be expanded to obtain a new question set. The initial question set can be obtained based on a predefined set of prompts.

[0089] In some implementations, when expanding the initial question set, a sentence generation model can be used to generate new questions based on the user characteristics corresponding to the questions in the initial question set. These new questions are then merged into the initial question set to obtain the final question set. The user characteristics corresponding to the questions can be derived by analyzing key information such as user preferences and writing styles. Based on this user characteristic analysis, the sentence generation model can intelligently synthesize a batch of new and more challenging questions. These new questions effectively complement and expand upon the original questions in terms of content and form, dynamically refreshing the question synthesis seed and injecting continuous innovation and personalized elements into the model training. This allows the model to have stronger adaptability and accuracy when facing diverse user needs. The sentence generation model described above can be obtained by fine-tuning a large language model for the scenario of generating the above questions.

[0090] In one possible implementation, when generating a new question through the above statement generation model, a second prompt can be generated based on the questions in the initial question set. The second prompt is used to prompt the statement generation model to generate a new question according to the user characteristics corresponding to the question. The second prompt is then input into the statement generation model to obtain the new question output by the statement generation model.

[0091] In one possible implementation, the above statement generation model can be the same as the above preference rating model. That is, when constructing preference data pairs, this model can be used to expand the questions in the question set, and preference ratings can be performed on the answer content output by the model in multiple subjective rating dimensions.

[0092] In one possible implementation, when generating new questions through a statement generation model, in addition to considering the user characteristics corresponding to the question, environmental information such as time (morning, noon, evening, etc.), scenario (work report, friend gathering, holiday celebration, etc.), and personal role attributes (such as student, teacher, professional, parent, etc.) can be incorporated into the question synthesis process. By integrating these diverse factors into the question synthesis process, questions that are more relevant to the user's actual usage scenarios, more targeted, and richer can be generated, providing broader and more realistic training materials for model training, thereby improving the model's adaptability and application effectiveness in complex and ever-changing real-world environments.

[0093] Optionally, technologies such as natural language processing and data mining can be used to conduct in-depth analysis of the collected data, identifying or inferring the user's environmental factors and personal role attributes. Based on user characteristics, environmental factors, and personal role attributes, flexible and diverse question generation rules can be formulated. According to the synthesis rules, natural language generation technology can be used to generate a rich variety of questions. Furthermore, in the process of generating new questions, the expression and content of the questions can be continuously optimized to ensure they are natural, accurate, and targeted, effectively guiding the model to generate answers that meet user needs and scenario requirements.

[0094] Step S420: Obtain the answer content output by the initial generation model for the questions in the question set.

[0095] Step S430: Output the preference scores for multiple subjective rating dimensions for the answer content corresponding to each question using the preference rating model.

[0096] Step S440: Determine preference data pairs based on each question, the corresponding answer content for each question, and the preference scores for multiple subjective rating dimensions corresponding to the answer content.

[0097] Step S450: Based on the preference data pair, the initial generative model is trained using the direct preference optimization algorithm to obtain the target generative model.

[0098] In this embodiment, steps S420 to S450 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0099] The model training method provided in this application embodiment can score the generator model's answers to questions across multiple subjective rating dimensions when determining the preference data pairs used to train the generator model. Furthermore, the preference data pairs are determined based on these scores, ensuring their accuracy and improving the accuracy of training the generator model using the determined preference data. This, in turn, enhances the accuracy of the generated content output by the trained generator model. Additionally, by expanding the question set, the method effectively complements and expands upon the original questions, enabling dynamic question updates. This injects continuous innovation and personalized elements into the model training, making the model more adaptable and accurate in the face of diverse user needs.

[0100] Please see Figure 5This document illustrates a structural block diagram of a model training apparatus 500 provided in an embodiment of this application. The model training apparatus 500 utilizes the aforementioned computer equipment and includes: a content acquisition module 510, a content scoring module 520, a data construction module 530, and a preference optimization module 540. The content acquisition module 510 acquires the answer content output by an initial generation model for questions in a question set. The content scoring module 520 uses the preference scoring model to output preference scores for multiple subjective scoring dimensions for the answer content corresponding to each question. The data construction module 530 determines preference data pairs based on each question, the answer content corresponding to each question, and the preference scores for the multiple subjective scoring dimensions corresponding to the answer content. The preference optimization module 540 trains the initial generation model based on the preference data pairs and using a direct preference optimization algorithm to obtain a target generation model.

[0101] In some implementations, the content scoring module 520 may be specifically used to: input the first prompt information and the answer content into a pre-trained preference scoring model to obtain preference scores for multiple subjective scoring dimensions output by the preference scoring model, wherein the first prompt information is used to prompt the preference scoring model to score the answer content from the multiple subjective scoring dimensions.

[0102] In one possible implementation, the content scoring module 520 may also be used to determine multiple subjective scoring dimensions corresponding to the current business scenario based on the current business scenario in the content creation scenario before inputting the first prompt information and the answer content into the pre-trained preference scoring model to obtain the preference scores of multiple subjective scoring dimensions output by the preference scoring model; and generate the first prompt information based on the multiple subjective scoring dimensions.

[0103] In some implementations, the data construction module 530 may be specifically used to: for each question, determine the preferred content and non-preferred answer content corresponding to each question based on the preference scores of multiple subjective rating dimensions corresponding to the answer content; and determine preference data pairs based on each question and the preferred and non-preferred answer content corresponding to each question.

[0104] In one possible implementation, the answer content corresponding to each question includes multiple candidate answer contents. The data construction module 530 can be specifically used to: for each answer content corresponding to each question, determine the total preference score of each candidate answer content corresponding to each question based on the preference scores of multiple subjective rating dimensions corresponding to the answer content; obtain candidate answer content whose total preference score is greater than a first score, and use it as the preferred answer content corresponding to each question; obtain candidate answer content whose total preference score is less than a second score, and use it as the non-preferred answer content corresponding to each question, where the second score is less than the first score.

[0105] Optionally, the data construction module 530 can also be used to obtain the sum of preference scores for multiple subjective rating dimensions corresponding to each candidate answer content for each question, as the total preference score for each candidate answer content; or

[0106] Based on the weights corresponding to each of the multiple subjective rating dimensions, the preference scores of the multiple subjective rating dimensions corresponding to each candidate answer are weighted and summed to obtain the total preference score of each candidate answer.

[0107] In some embodiments, the model training apparatus 500 may further include a question expansion module. The question expansion module can be used to expand the questions in the initial question set before obtaining the answer content output by the initial generation model for the questions in the question set, thus obtaining the question set.

[0108] In one possible implementation, the question expansion module can be specifically used to: generate new questions by using a statement generation model and based on the user characteristics corresponding to the questions in the initial question set; and merge the new questions into the initial question set to obtain the question set.

[0109] Optionally, the question expansion module can be specifically used to: generate a second prompt based on the questions in the initial question set, the second prompt being used to prompt the statement generation model to generate a new question according to the user characteristics corresponding to the question; and input the second prompt into the statement generation model to obtain a new question output by the statement generation model.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0112] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0113] In summary, the solution provided in this application obtains the answer content output by the initial generation model for questions in a question set; it then uses a preference scoring model to output preference scores for multiple subjective scoring dimensions for the answer content corresponding to each question; based on each question, the answer content corresponding to each question, and the preference scores for the multiple subjective scoring dimensions corresponding to the answer content, preference data pairs are determined; and based on the preference data pairs, the initial generation model is trained using a direct preference optimization algorithm to obtain the target generation model. Therefore, when determining the preference data pairs used to train the generation model, the answer content of the generation model for questions can be scored on multiple subjective scoring dimensions, and the preference data pairs are determined based on the scores of multiple subjective scoring dimensions. This ensures the accuracy of the preference data pairs, improves the accuracy of training the generation model using the determined preference data, and ultimately improves the accuracy of the generated content output by the trained generation model.

[0114] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer device according to an embodiment of this application. The computer device 100 can be a server, PC, laptop, smartphone, tablet, smartwatch, e-reader, or other device capable of running applications. The computer device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0115] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0116] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0117] Please refer to Figure 7 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0118] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain the output of the initial generated model for the questions in the question set; The preference scoring model outputs preference scores across multiple subjective rating dimensions for the answers to each question. Based on each question, the corresponding answer to each question, and the preference scores of multiple subjective rating dimensions corresponding to the answer, preference data pairs are determined; Based on the preference data pairs, and by training the initial generative model using the direct preference optimization algorithm, the target generative model is obtained.

2. The method according to claim 1, characterized in that, The preference scores for multiple subjective rating dimensions output by the preference rating model for the answer content corresponding to each question include: The first prompt information and the answer content are input into a pre-trained preference rating model to obtain preference scores for multiple subjective rating dimensions output by the preference rating model. The first prompt information is used to prompt the preference rating model to score the answer content from the multiple subjective rating dimensions.

3. The method according to claim 2, characterized in that, Before inputting the first prompt information and the answer content into the pre-trained preference rating model to obtain the preference scores of multiple subjective rating dimensions output by the preference rating model, the method further includes: Based on the current business scenario in the content creation scenario, determine multiple subjective scoring dimensions corresponding to the current business scenario; The first prompt message is generated based on the multiple subjective rating dimensions.

4. The method according to claim 1, characterized in that, The determination of preference data pairs based on each question, the corresponding answer content for each question, and the preference scores for multiple subjective rating dimensions corresponding to the answer content includes: For each question, based on the preference scores of multiple subjective rating dimensions corresponding to the answer content, the preferred content and non-preferred answer content for each question are determined. Based on each question, and the corresponding preferred and non-preferred answers to each question, preference data pairs are determined.

5. The method according to claim 4, characterized in that, The answer content corresponding to each question includes multiple candidate answer contents. For each answer content corresponding to a question, based on the preference scores of multiple subjective rating dimensions corresponding to the answer content, the preferred and non-preferred answer contents corresponding to each question are determined, including: For each question, the total preference score for each candidate answer is determined based on the preference scores of multiple subjective rating dimensions corresponding to the answer. The candidate answers with a total preference score greater than a first score are obtained and used as the preference answers for each question. Candidate answers whose total preference score is less than the second score are obtained and used as non-preference answers for each question, wherein the second score is less than the first score.

6. The method according to claim 5, characterized in that, For each question, based on the preference scores of multiple subjective rating dimensions corresponding to the answer content, the total preference score for each candidate answer content is determined, including: For each question, the sum of preference scores across multiple subjective rating dimensions for each candidate answer is obtained, and this sum is used as the total preference score for each candidate answer; or Based on the weights corresponding to each of the multiple subjective rating dimensions, the preference scores of the multiple subjective rating dimensions corresponding to each candidate answer are weighted and summed to obtain the total preference score of each candidate answer.

7. The method according to any one of claims 1-6, characterized in that, Before obtaining the initial generation model's output of answers to questions in the question set, the method further includes: Expand the initial set of problems to obtain the problem set.

8. The method according to claim 7, characterized in that, The problem set is obtained by expanding the initial problem set, including: The model is generated by using statements, and new questions are generated based on the user characteristics corresponding to the questions in the initial question set; The new problem is merged into the initial problem set to obtain the problem set.

9. The method according to claim 8, characterized in that, The step of generating a model through statements and generating new questions based on user characteristics corresponding to questions in the initial question set includes: Based on the questions in the initial question set, a second prompt message is generated. The second prompt message is used to prompt the statement generation model to generate a new question according to the user characteristics corresponding to the question. The second prompt information is input into the statement generation model to obtain a new question output by the statement generation model.

10. A model training device, characterized in that, The device includes: a content acquisition module, a content rating module, a data construction module, and a preference optimization module, wherein... The content acquisition module is used to acquire the answer content output by the initial generation model for the questions in the question set; The content scoring module is used to output preference scores for multiple subjective scoring dimensions for the answer content corresponding to each question through a preference scoring model. The data construction module is used to determine preference data pairs based on each question, the corresponding answer content for each question, and the preference scores for multiple subjective rating dimensions corresponding to the answer content; The preference optimization module is used to train the initial generative model based on the preference data pair and using the direct preference optimization algorithm to obtain the target generative model.

11. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-9.

Images